Catching bugs with the Automated Testing System
Software developers use regression testing to help uncover new bugs in an application after enhancements, patches, or other changes have been made. It is an essential but repetitive activity, as it entails rerunning a representative set of test cases each time the code is updated to ensure that the code continues to run the same way and produce the same answers as before. For a large code that is updated many times a day by a sizeable team of computer scientists, such as some at Lawrence Livermore, regression testing would be impossible to keep up with if done in a traditional fashion.
Shawn Dawson, a computational scientist working with the Weapons and Complex Integration (WCI) Directorate, notes, “Many of our programmatic codes at Livermore are too massive, dynamic, and complex to develop with manual testing. We simply had to automate the process.” Dawson is in charge of the maintenance and enhancement of one such solution—the Automated Testing System (ATS), a Livermore-developed, scalable, Python-based tool for automating the running of tests for an application. ATS currently operates on Livermore’s Linux clusters and IBM Blue Gene machines and can be used to execute thousands of tests with a single command.
By helping projects to meet their milestones and maintain code integrity, ATS serves as a fundamental enabler for important projects in WCI and the Physical and Life Sciences Directorate. Tens of thousands of tests a day are run at Livermore with the help of ATS. Moreover, these project teams are able to make a more efficient use of resources by adopting the common testing framework, rather than each project developing their own ad hoc solutions.
It was dissatisfaction with commercial testing products that spurred the development of ATS a decade ago. Dawson, who is both a developer for and a customer of the tool, notes, “Commercial codes didn’t suit the needs of the codes we develop. We run on unique systems, not simply desktop PCs or Macs. Also, those codes don’t have the flexibility to extend testing or compare data.” With ATS, Dawson and his colleagues have been able to introduce various extensions into the tool, such as—most recently—a capability for testing and comparing image files.
Not only do Livermore developers test the code before committing to each proposed change, they also run multiple larger nightly test suites on several computer platforms to test the correctness and performance of the code with different configurations and compilers. ATS’s customer codes average about 4,000 tests each night, and each test produces multiple plots, resulting in a substantial amount of data. ATS organizes the output of these nightly runs into a text-based list and emails it to the project team for review the next morning. It also creates a graphical web-based report for the team to compare the curves side by side and quickly spot bugs introduced to their code. Scientists can easily post-process the testing results for deeper analysis.
Dawson’s team strives to enhance the tool on an ongoing basis, for instance by incorporating user feedback. “Testing is a complex issue. We can’t ever test 100% of the code, as the lines depend on the type of data input, but we are testing the large majority of our use cases. When a user finds an area we missed, we incorporate it into the test suite.” Customer needs also drive the development of new features for the tool. In 2015, the ATS team will begin archiving metrics related to each test run, a much-requested capability. This requires connecting ATS to a Laboratory-developed database called Tracker. With this enhancement, users will be able to run queries and generate reports on the data.
While ATS was designed for regression testing, it also can be used to look at code performance over time and across platforms. “The tool is really useful for comparing one machine to another. I use it frequently to compare compiler releases,” says Dawson. He anticipates employing ATS for acceptance testing for the new compilers and libraries during the integration of Livermore’s next advanced technology system, Sierra. “ATS was designed to be easily extendable to new machines,” he adds. “As we move to the next advanced technology system, ATS will be ported to the machine, and we will provide support for it.”
ATS will also support code preparation for Sierra and other future machines. ATS user Patrick Brantley notes, “ATS is a very flexible framework that allows us to deliver a more reliable simulation code to our users. As we contemplate potentially significant changes to our simulation code to enable its use on advanced computing architectures, we will continue to rely on ATS to help us maintain the integrity of our code.”