ZFS: Improving Lustre Efficiency

Sequoia, LLNL’s newest and largest supercomputer, is paired with a 55-petabyte file system with a Lustre foundation, known as Grove, which stores vast quantities of simulation data. Grove must transfer information to and from Sequoia at a minimum of 500 gigabytes per second. As computing platforms and data sizes grow, Lustre bandwidths must be increased to keep computation resources working efficiently. To support Grove’s high storage capacity and bandwidth requirements, LC software developers have engaged in a multi-year project to replace much of Grove’s Lustre foundation with the Zettabyte File System (ZFS). LC led the work of porting ZFS to Linux, while a team at Sun Microsystems (the company that created ZFS) focused on making Lustre compatible with ZFS.

LC scientists demonstrated the integrated product on a prototype system in 2010. Once built out to Sequoia scale and optimized, the retooled file system would attain Sequoia’s stretch bandwidth goal of one terabyte per second at less than half the cost of a file system built with standard Lustre components. LC also released a Linux port of ZFS to the public, which generated significant interest within the Linux community. Community members volunteered to complete the porting work and test ZFS on various types of Linux hardware. These efforts produced a more robust product while allowing LC to focus on building a viable file system for Sequoia. Work on Grove’s back end wrapped up in late 2012, though testing and tuning continue.

For LC computer scientists, who have championed the idea of combining ZFS and Lustre for a number of years, successfully completing the Grove project has been gratifying. Implementing ZFS has resulted in a cheaper, less complex, and higher performance file system for Sequoia. ZFS effectively converts the concurrent, random writes arriving at an object storage target to a stream of faster, less resource-intensive sequential writes—especially important on large systems with high data storage rates. Lustre-supported ZFS also uses advanced caching technology and solid-state disks to improve read performance and incorporates robust data integrity-checking features for increased system availability and reliability. For instance, ZFS prevents accidental overwrites of existing data and not only detects data corruption, but also automatically corrects the bad data. As an added benefit, this software infrastructure will allow for a wide range of alternate file systems to exist under Lustre.

For more information, contact Brian Behlendorf.