StarSapphire is a collection of projects in the area of scientific data mining focusing on the analysis of data from scientific simulations, observations, and experiments. StarSapphire is a follow-on to the Sapphire scientific data mining project, where we conducted research in algorithms, incorporated this research into software, and applied the software to real-world problems, which, in turn, motivated our research. Our experiences showed that we could use techniques from data mining, machine learning, image and video processing, statistics, and pattern recognition to improve the way in which scientists extract useful information from data.
In the StarSapphire projects, we are leveraging these earlier experiences to address the recent challenges in data-driven modeling and analysis. These challenges are the result of newer types of data, such as data streams, larger volumes of data, such as those resulting from three-dimensional simulations of complex phenomena, and new constraints on the analysis, such as the need for in-situ analysis in exascale systems or real-time analysis for anomaly detection.
Despite these new challenges, our approach to analysis remains the same as the one we developed and used in Sapphire (shown in Figure 1). This approach worked very well in the analysis of data from a variety of problems in many different domains. We found that it was important to consider scientific data mining as an iterative and interactive process, involving data pre-processing, search for patterns, knowledge evaluation, and possible refinement of the process based on input from domain experts or feedback from one of the steps. As the pre-processing of the data is a time-consuming, but critical, first step in data mining, we include it as an integral part of the process. The pre-processing is often domain and problem dependent; however, several techniques developed in the context of one problem or domain can be applied to other problems and domains as well. The pattern recognition step is usually independent of the domain or problem.
As part of StarSapphire, we are involved in the following projects:
- Poincaré plots: Classification and characterization of orbits
- Blob tracking: Analysis of coherent structures in NSTX images
- GSEP Analysis: Analysis of fluid and particle data from GSEP simulations
- WindSENSE: Managing the integration of wind energy on the power grid
- SensorStreams: Real-time analysis of streaming data from sensors
- MINDES: Data mining for inverse design
- Exa-DM: Enabling scientific discovery in exascale simulations
We gratefully acknowledge the following funding sources (in alphabetical order) for supporting our work:
- DOE Office of Advanced Scientific Computing Research
- DOE Office of Energy Efficiency and Renewable Energy
- DOE SciDAC program