PNNL

Abstract

Active Storage provides an opportunity for reducing the band- width requirements between the storage and compute elements of cur- rent supercomputing systems, and leveraging the processing power of the storage nodes used by some modern file systems. To achieve both objec- tives, Active Storage allows certain processing tasks to be performed directly on the storage nodes, near the data they manage. However, Active Storage must also support key requirements of scientific applications. In particular, Active Storage must be able to support striped files and files with complex formats (e.g., netCDF). In this paper, we describe how these important requirements can be addressed. The experimental results on a Lustre file system not only show that our proposal can re- duce the network traffic to near zero and scale the performance with the number of storage nodes, but also that it provides an efficient treatment of striped files and can manage files with complex data structures.