By Eduardo D’Azevedo, Esmond G. Ng, and Stefan M. Wild
The Office of Science at the U.S. Department of Energy created the SciDAC (Scientific Discovery through Advanced Computing) program to address challenging large-scale scientific problems that are central to the mission of DOE. Given the scale of these scientific problems, progress requires high-performance computing platforms, such as those at the National Energy Research Scientific Computing Center, the Argonne Leadership Computing Facility, and the Oak Ridge Leadership Computing Facility. Established in 2001 as a five-year program, SciDAC is now in its third five-year cycle; its longevity can be attributed to both the increasing scale of the scientific problems to be solved and the growing complexity of computer architectures.
At the time of the June 2001 launch of SciDAC, the fastest machine in the world (at Lawrence Livermore National Laboratory) attained 7.2 teraflops per second across 8192 cores. Twelve years later, the fastest machine (the Tianhe-2 in China) has achieved 33,800 teraflops per second across 3.12 million cores, representing an almost 500-fold increase in computing power. At the same time, computer architectures have become much more elaborate, with increasingly heterogeneous design and deepening memory hierarchies. In this setting, a multidisciplinary approach is needed for solving large-scale scientific problems.
A defining aspect of the SciDAC program has been its strong emphasis on partnerships among domain scientists, applied mathematicians, and computer scientists; a reorganization of the program at the beginning of its current cycle reinforces the importance of partnerships. The goal is to leverage expertise in computer science and mathematics, as well as state-of-the-art mathematical algorithms and software libraries for advanced computing, in solving large-scale scientific problems and thus making possible and accelerating scientific discoveries. Understanding the current SciDAC program begins with a look at the structure of DOE’s Office of Science.
Connections between the four SciDAC institutes (left) and the science partnerships (right). A program of the Office of Science at the Department of Energy, SciDAC was reorganized in 2011, at the beginning of its third five-year cycle.
SciDAC as Nexus
The DOE Office of Science comprises six program offices: Along with Advanced Scientific Computing Research (known as ASCR, and the most familiar to the SIAM community) are Basic Energy Sciences (BES), Biological and Environmental Sciences (BER), Fusion Energy Sciences (FES), High Energy Physics (HEP), and Nuclear Physics (NP). Most projects of the Office of Science are funded by individual program offices; the SciDAC projects have a different structure.
The current SciDAC program has two components: institutes and science partnerships. The four institutes, which are focused on applied mathematics and computer science, are funded entirely by ASCR. Each science partnership, focused on a distinct set of scientific problems, is funded jointly by ASCR and one of the five other program offices.
Almost all SciDAC projects are large, multi-institutional collaborative efforts. Overall, 13 DOE laboratories, 44 universities, and three companies receive funding under the current SciDAC program.
The SciDAC Institutes
FASTMath—the Frameworks, Algorithms, and Scalable Technologies for Mathematics Institute—develops scalable mathematical algorithms and software for reliable simulation of complex phenomena. Its focus is on structured and unstructured meshing, linear and nonlinear equation solvers, time integrators, variational inequality solvers, and eigensolvers.
QUEST—the Quantification of Uncertainty in Extreme Scale Computations Institute—is focused on uncertainty quantification (UQ) in large-scale scientific computations. The overarching goal of QUEST is to provide modeling, algorithmic, and general UQ expertise, together with software tools, to other SciDAC institutes, SciDAC science partnerships, and Office of Science projects in general, thereby enabling and guiding a broad range of UQ activities in their respective contexts.
SUPER—the Institute for Sustained Performance, Energy, and Resilience—seeks to ensure that computational scientists are able to exploit the emerging generation of leadership-class computing systems. To this end, SUPER addresses performance portability for new systems, management of energy consumption, resilient computation, and end-to-end optimization.
SDAV—the Institute of Scalable Data Management, Analysis, and Visualization—addresses the explosive growth of data in scientific computing. SDAV provides solutions in three areas: data management, including the infrastructure for efficient indexing, compression, and organization of datasets; data analysis, with a focus on in situ analysis, filtering, and data reduction; and data visualization, for identifying features in multiscale and multiphysics datasets.
The main goals of the institutes are the development of scalable algorithms and tools for core components of scientific simulation, and the distribution of these capabilities through portable high-performance libraries. At the same time, the institutes work to deploy general mathematics and computer science expertise and algorithms, tools, and libraries in the science partnership projects.
Science Partnerships
Scientific discovery is the focus of the 18 science partnership projects, each of which is supported jointly by ASCR and one of the five other science program offices.
The BES program office supports seven projects. The application areas include energy-related advanced materials for photovoltaic and photocatalysis, nanoscale materials, superconductors, and lithium cell batteries. Many of these projects require algorithmic advances in tensor decomposition, sparse linear equation solvers, eigensolvers, and quantum Monte Carlo methods.
The BER program office supports three projects aimed toward a predictive understanding of climate and environmental systems. Activities include simulating the dynamics of the atmosphere, ocean, and ice sheets, as well as biogeochemical responses and feedback.
Three projects span the energy, intensity, and cosmic frontiers of the HEP program office. The projects focus on accelerator modeling and design, lattice quantum chromodynamics for moving beyond the standard model, and cosmological simulations of dark energy/matter.
Through large-scale simulations, the three NP program office projects study the strong interaction of low-energy physics, lattice QCD calculations for heavy-ion/medium-energy physics, and connections between the two.
Two projects are funded by the FES program office. One project is developing simulation tools capable of predicting the performance of tungsten-based plasma-facing components and divertor components in extreme conditions in a burning plasma environment. The other is working on first-principles physical models that can provide insight into edge plasma physics.
As indicated earlier, the partnerships seek to advance understanding of basic science. Each partnership includes a team made up of domain scientists, applied mathematicians, and computer scientists who work closely together. In some cases, the collaborations leverage and build on what the applied mathematicians and computer scientists have developed. In other cases, the collaborations require the development of new techniques so that specific computational challenges can be tackled in the solution of the science problems.
The domain scientists in all the science partnership projects are funded by their respective science program offices. Some of the applied mathematicians and computer scientists in these partnership projects are also part of SciDAC institutes and serve to bridge the institutes and partnership projects.
Multidisciplinary Meetings
A hallmark of the SciDAC program is its annual meeting, at which principal investigators interact, communicate their partnership experiences, exchange ideas, and highlight successes. The third incarnation of SciDAC having begun in 2011, the 2012 meeting had the goal of introducing the projects to the entire SciDAC community. The second annual meeting of the current SciDAC program, held in Rockville, Maryland, July 24–26, 2013, focused on connections between the domain scientists and applied mathematicians and computer scientists, as well as the links between science partnerships and institutes. Approximately 150 researchers participated in each of those meetings.
SciDAC and the Mystery of Carbon-14’s Role in Carbon Dating
One of the scientific discoveries made possible by the SciDAC program concerns carbon-14 (14C), the isotope used in carbon dating. The effectiveness of carbon-14 in determining the age of organic materials lies in its half-life of more than 5000 years, as compared with a maximum of 21 minutes for all other unstable carbon isotopes. Why this one particular form of carbon has such a long half-life was long a mystery.
As part of a SciDAC-2 project supporting nuclear physics research, a team of nuclear physicists, applied mathematicians, and computer scientists collaborated to produce a scalable and efficient code that, used in large-scale simulations, unraveled this mystery. Participating scientists determined simultaneous interactions of three nucleons (i.e., protons and neutrons in the atomic nucleus) to be an important factor. Their result appeared in a paper titled “Origin of the Anomalous Long Lifetime of 14C” (Physical Review Letters, May 2011).
To achieve their result, the team performed 25 simulations, each running for about six hours on 215,000 processing cores of a Cray XT5 supercomputer (named Jaguar) at Oak Ridge National Laboratory. The heart of each simulation was the solution of a sparse eigenvalue problem with a dimension of about 1 billion. The matrix has approximately 40 trillion nonzero elements, and eight eigenpairs were needed. Although the original simulation code was developed prior to the SciDAC collaboration, the multidisciplinary collaboration under SciDAC resulted in new algorithms, and some computer science issues were resolved. This led to performance improvements to the code, making it run several orders of magnitude faster. Without these improvements, our understanding of carbon-14 would not be as advanced as it is today. Indeed, in the words of Jeremy Holt of the Technical University of Munich, in an article in New Scientist, “One can now say confidently that the problem is solved.”
Further information on the SciDAC program can be found here.
Eduardo D'Azevedo is a member of the Mathematics Group and the Computer Science and Mathematics Division at Oak Ridge National Laboratory. Esmond Ng is a member of the National Energy Research Scientific Computing Division at Lawrence Berkeley National Laboratory. Stefan Wild is a computational mathematician in the Laboratory for Applied Mathematics and Numerical Software group at Argonne National Laboratory.