u.s. department of energy’s office of science
DESCRIPTION
U.S. Department of Energy’s Office of Science. Office of Advanced Scientific Computing Research - Status - Petascale Simulations SciDAC. Walter M. Polansky [email protected]. American Competitiveness Initiative. - PowerPoint PPT PresentationTRANSCRIPT
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
Walter M. [email protected]
Office of Advanced ScientificComputing Research
- Status -
Petascale SimulationsSciDAC
U.S. Department of Energy’s Office of Science
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 2
American Competitiveness Initiative
President Bush in the President’s State of the Union Address on January 31, 2006
“I propose to double the federal commitment to the most critical basic research program in the physical sciences over the next ten years. This funding will support the work of America’s most creative minds as they explore promising areas such as nanotechnology, supercomputing, and alternative energy sources.”
Secretary Bodman commenting on President Bush’s statement
“Developing revolutionary, science-driven technology is at the heart of the Department of Energy’s mission. To ensure that America remains at the forefront in an increasingly competitive world, our Department is pursuing transformational new technologies in the cutting-edge scientific fields of the 21st century—areas like nanotechnology, material science, biotechnology, and high-speed computing.”
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 3
Office of Science: Mission
“The Office of Science plays a critical role in ensuring America’s scientific leadership and economic dynamism…” The mission of the Science program is to deliver the discoveries and scientific tools that transform our understanding of energy and matter and advance the national, economic, and energy security of the United States.
Secretary Samuel Bodman, House Committee on Science Hearing, February 15, 2006
In support of it’s mission, the Science program has responsibilities in three main areas: selection and management of research; operation of world-class, state-of-the-art scientific facilities; and design and construction of new facilities.
“Investment in these facilities is much more than bricks and mortar: it is an investment in discovery, and in the future of our nation.”
Facilities for the Future of Science: a 20-Year Outlook
U.S. Department of Energy
Office of Science
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
Office of Science -- FY2007 Budget Request- $4.1B --
FY2006 Appropriation- $3.6B
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 6
ASCR Vision
First in Computational Science “Best in class in advancing science and technology through modeling and simulation”
Facilities
Enabling Technologies
Computational Partnerships
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 7
ASCR Research Strategy
• Applied Mathematics and Computer Science deliver the operating systems, programming models, software tools, and mathematical algorithms and libraries needed for scientists to make effective use of petascale computing.
• Scientific Discovery through Advanced Computing (SciDAC) program, will strengthen activities at the software centers initiated in FY 2006 for petascale computing.
• SciDAC will foster additional research investments in Applied Mathematics and Computer Science to accelerate efforts in modeling and simulation on the petascale computing facilities.
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 8
ASCR Facilities
• Oak Ridge Leadership Computing Facility
• Argonne Leadership Computing Facility
• NERSC at Lawrence Berkeley National Lab
• ESnet at Lawrence Berkeley National Lab
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 9
FY 2006 ASCR Budget Distribution
Research and Evaluation Prototypes
$13.0
Applied Mathematics $29.4
NERSC $37.5
ESnet $19.0
Distributed Network Environment Research
$13.6
SciDAC $38.1
Computer Science $24.3
SBIR/STTR $6.3
LCF$53.7
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 10
FY 2007 ACSR Budget Status
0
50
100
150
200
250
300
350
FY06 FY07 Request FY07 House FY07 Senate
$235M
$319M $319M$319M
Do
llars
in M
illio
ns
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 11
FY 2007 - House
FY 2007 House Energy and Water Development Appropriations Bill passed both the Appropriations Committee and the House Report language:
“ The Committee commends the Office of Science and the Office of Advanced Scientific Computing Research for their efforts to provide cutting-edge capabilities to meet current scientific computational needs, and at the same time to extend the boundaries of that cutting edge into the next generation of high-performance scientific computers and supporting software.”
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 12
FY 2007 - Senate
FY 2007 Senate Energy and Water Development Appropriations Bill passed the Appropriations Committee Report language
“ …The Committee is concerned with the relationship between the Office of Science and the NNSA. As an example, the ASCR strategic plan discusses the need to work with other Federal agencies including several defense agencies, but only discusses in general terms three areas of research where NNSA and the Office of Science cooperated. In the area of basic research, the strategic plan states that it is an area that is `not important enough to justify ASCI investment at this time.' The Committee is also aware that the Office of Science has budgeted $13,000,000 for the DARPA to support a petaflop computer deployment by 2010. The Committee believes this funding would be better spent within the Department to support a petaflop initiative. The Department is directed to divide the funds equally between the Office of Science and the NNSA Advanced Simulation and Computing activities to support development of component architecture for high-performance software and storage”
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 13
Facilities Update
• NERSC – delivery of NERSC-5 in FY 2007• LCF at ORNL –
– Upgrade Cray XT3 upgrade • Now: 25Teraflop 50Teraflop• By the end of 2007: 50Teraflop 250Teraflop
– Acquire 1 Petascale Cray Baker system by end of 2008• LCF at Argonne –
– Acquire 100 Teraflop IBM Blue Gene/P in FY2007– Upgrade to 250-500 Teraflop IBM Blue Gene/P in 2008– On path to a Petascale IBM Blue Gene /Q by end of the
decade
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 14
Innovative and Novel Computational Impact on Theory
and Experiment- INCITE
• Initiated in 2004• Provides Office of Science computing resources to a small
number of computationally intensive research projects of large scale, that can make high-impact scientific advances through the use of a large allocation of computer time and data storage
• Open to national and international researchers, including industry
• No requirement of DOE Office of Science funding• Peer-reviewed• 2004 Awards: 4.9 Million processor hours at NERSC awarded
to three projects• 2005 Awards: 6.5 Million processor hours at NERSC awarded
to three projects
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 15
INCITE 2006
• Expanded to include SC high end computing resources at PNNL, ORNL and ANL in addition to LBNL and multiple year requests.
• Received 43 proposals requesting over 95 million processor hours.– 60% from Universities– 40% had funding from other
federal research agencies
• 15 Awards for over 18.2 million processor hours
Wired Magazine, August, 2006, pg. 42
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 16
INCITE 2007
• Expanded in 2007 to include 80% of resources at Leadership Computing Facilities in addition to 10% of NERSC and 5% of PNNL
• Call issued July 27, 2006
• Proposals due September 15, 2006• Award announcements in mid-December
http://hpc.science.doe.gov
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 17
Scientific Discoverythrough Advanced Computing
Structure
-- Scientific Application Areas - Breakthrough scientific advances through computer modeling and simulation that
are impossible using theoretical and experimental studies or improve experimental science.
-- Centers for Enabling Technology (CETs)- provide the essential computing and communications infrastructure to support SciDAC applications
-- Institutes- University-led centers of excellence; address specific software issues; Engage a broader community of scientists in modeling and simulations.
www.SciDAC.gov
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 18
Computing/Networking
Applications
Scientific Discovery
Leadership Computing- ANL
100 TF
Leadership Computing- ORNL
250 TF
Production Computing- NERSC
100-150 TF
ESnetOn path toward Dual rings 40Gbps/ 10 Gbps fault tolerant
Path to Petascale
- Centers for Enabling Technology
- Scientific Applications Partnerships - Institutes (University-lead)
- Accelerator science and simulation - Astrophysics- Climate modeling and simulation - Computational Biology- Fusion science - High-energy physics- Petabyte high-energy/nuclear physics - Materials science and chemistry- Nuclear physics - QCD- Radiation transport - Turbulence
- Groundwater reactive transport modeling and simulation
Sci
DA
C-2
INC
ITE
NE
RS
C A
llo
ca
tio
n
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 19
SciDAC-2 Institutes…
… University-led centers of excellence that focus on major software issues through a range of collaborative research interactions.
– Develop, test, maintain, and support optimal algorithms, programming environments, systems software and tools, and applications software.
– Focus on a single general method or technique.
– Focal point for bringing together a critical mass of experts from multiple disciplines to focus on key problems in a particular area of enabling technologies.
– Forge relationships between experts in software development, scientific application domains, high performance computing, and industrial partners.
– Reach out to engage a broader community of scientists in the activities of scientific discovery through advanced computation and collaboration.
– Incorporate training and outreach in high performance computing topics, including for graduate students and postdocs.
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 20
Centers for Enabling Technologies… (CETs)
… provide the essential computing and communications infrastructure to support SciDAC applications; multi-discipline approach with activities in:
– Algorithms, methods, and libraries.– Program development environments and tools -- terascale and petascale program
development and tools provide maximum ease-of-use to scientific end users. – Systems software that provides system stability and functionality needed by users for tera- to
peta- scale simulations. – Visualization and data management systems.
• CETs work directly with applications on: – Development and application of computing systems software that allows scientific simulation
codes to take full advantage of the extraordinary capabilities of terascale and petascale computers.
– Ensuring that the most critical computer science and applied mathematics issues are addressed in a timely and comprehensive fashion.
– Addressing all aspects of the successful research software lifecycle including transition of a research code into a robust production code and long term software evolution and maintenance and end user support.
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 21
SciDAC Accomplishments
• To better understand combustion which provides 80% of the energy used in the U.S., SciDAC teams created first laboratory-scale flame simulation in three dimensions
• Magnetic fusion scientists and applied mathematicians simulated techniques for re-fueling fusion reactors
• Teams developed new methods for simulating improvements in future particle accelerators
• Partnerships improved effectiveness of scientific applications codes between 275% to over 10,000%
Hydroxyl radical in a turbulent jet flame
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 22
The Power and Evolution of Partnerships
Today and Tomorrow
ASCR
HEP
NE
EM
NNSA
BES
BER
FES
NP
Industry
Methane Flames Diesel CombustionUltrafast Science
GCM Performance CloudsSimulation of Microbes
Plasma Core Full Simulation ofBurning Plasma
Weak Matrix ElementsPerformance of Supernova codesAccelerator Design
Increased Accuracy
Elements beyond Fe
LCLS System Design
Neutron Transport Advanced Fuel Cycles
Groundwater Transport
Libraries &Algorithms Science Based Materials
Improvements in Mesh Generation
Virtual Prototypes
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 23
Road to Petascale Computing: Combustion
A 50% increase in efficiency of automobiles could save 21% of U.S. oil consumption for transportation
Direct Numerical Simulation of Turbulent Combustion, a collaborative effort between Sandia National Laboratories, U. of Wisconsin, U. of Michigan, U. of Maryland, U.C. Davis, and Princeton U., has been focusing on advancing science for improved fuel efficiency of gas turbines and technologies for transportation.
Petascale computing will enable fundamental understanding of turbulence-chemistry interactions, e.g. multi-stage auto-ignition and flame stabilization in compression ignition engine environments, in regimes relevant to real devices. This large turbulent CO/H2 jet flame was run
on the CrayX1E at ORNL LCF.
(Credit for the image is to Kwan-Liu Ma, Hongfeng Yu, and Hiroshi Akiba of UC Davis.)
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 24
Volume rendering of scalar dissipation rates in a temporal plane jet flame. High values are shown in red, medium in yellow, and lower values in blue. (Credit: Jackie Chen/SNL)
Turbulence
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 25
Center for Simulation of Wave Interaction with Magneto-hydrodynamics, a collaboration of PPPL, ORNL, MIT, Columbia U., Indiana U., Wisconsin U., etc., is a project for very large plasma wave simulation for fusion-based research.
Using 4,096 processors of Cray XT3 at LCF (ORNL), it is the largest, most-detailed simulation ever done of plasma control waves in a tokamak, the reactor that will eventually form the core of the multinational ITER reactor.
Road to Petascale Computing: Fusion
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 26
A multi-institution project for large-scale simulation of Neutrino-driven Supernova moves forward to discover a new instability that is important to core collapse supernovae – the supernova shock wave instability (SASI) and a plausible mechanism for generating the initial neutron star spin required to give birth to pulsars.
Road to Petascale Computing: Astrophysics
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 27
Researchers at Argonne National Laboratory have recently developed a Portable, Extensive, Toolkit for Scientific (PETSc) numerical library used in dozen of scientific applications world-wide.
PETSc encapsulates the complexity of underlying parallel algorithms and presents them in terms of numerical abstractions familiar to scientists. It has been used to extend fully compressible flow codes, most notably the pressure-oriented implicit continuous Eulerian (PCICE) scheme developed at Idaho National Engineering Laboratory.
Road to Petascale Computing:Modeling for Reactor Design
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research Program
CASC- September 6, 2006 28
Researchers at Lawrence Livermore National Laboratory have recently developed a suite of fast and scalable adaptive algorithms for modeling shocks and detonations interacting with movable rigid bodies in a fluid. Accurate 2D simulations tax existing computer resources; extending to well-resolved 3D simulations will require peta-scale resources.
This new algorithm combines efficient local representations of the embedded bodies with moving geometry and adaptive mesh refinement.
Road to Petascale Computing: Modeling the Interaction b/w Fluids & Embedded
Solid Particles
Shock hitting several movable rigid bodies using a Godunov method with adaptive mesh refinement; density and refinement grids are shown
The same fluid solver can model detonations using an ignition and growth model and JWL equation of state. This mock Schlieren shows a detonation interacting with solid bodies.
(Credit for the images is to Bill Henshaw (PI) and Kyle Chand, LLNL)