A Flexible Package Manager for HPC Software
High performance computing (HPC) software is becoming increasingly complex, quickly outpacing the capabilities of existing software management tools. To support scientific applications, system administrators and developers frequently build, install, and support different configurations of math and physics libraries and other software on the same HPC system. Those applications are later rebuilt to fix bugs and to support new operating system versions, compilers (special programs that turn programming language into code that can be used by a computer’s processor), message passing interface (MPI) versions, and other dependency libraries. Forcing all application teams to use a single, standard software stack—a set of software subsystems or components needed to create a complete platform—is infeasible, but managing many software configurations and versions for all users on a single system is a time-consuming task for supercomputing staff.
Existing tools can automate portions of this process, but they either cannot manage installation of multiple versions and configurations, or they require numerous configuration files for each software version, leading to organizational and maintenance issues. Enter Spack, a flexible, Python-based package management tool designed by LLNL computer scientists. Spack operates on a wide variety of HPC platforms and environments, and it allows any number of builds to peacefully coexist on the same system. Software installed by Spack runs correctly regardless of environment, and file management is streamlined.
Spack maintains a single file for many different builds of the same package, and its simple syntax lets users specify versions and configuration options concisely. Each software’s dependency graph—showing the packages it relies on to function—is a unique configuration, and Spack installs each of those configurations in a unique directory, enabling configurations of the same package to run on the same machine. Spack uses RPATH linking so that each package knows where to find its dependencies.
Various Livermore code teams have begun employing Spack. For instance, Spack was used to successfully automate software builds for ARES, a large Livermore radiation hydrodynamics code. ARES runs on Livermore’s commodity Linux clusters and Blue Gene/Q machines. With Spack, the team was able to test more configurations more efficiently than ever before. The tool has enabled them to detect and fix compiler incompatibilities and to complete more testing, which has helped both ARES and LLNL library developers to build more robust software. Because other LLNL code teams use many of the same libraries as ARES, LLNL code teams also have begun creating an internal repository of Spack build recipes, which will make packaging the next code much easier.
Spack is open-source software, and a rapidly growing community of Spack users at Livermore, other national laboratories, and universities is helping to grow its popularity.