overview of the scidac project: collaborative design and development of the ccsm for terascale...

Overview of the SciDAC Project: Collaborative Design and

Development of the CCSM for Terascale Computers

PI: Malone(LANL), Drake(ORNL)Co-I (DOE): Ding(LBL), Duffy(LLNL), Erickson(ORNL), Foster(ANL), Ghan(PNL), Jacob(ANL), Jones(LANL), Larson(ANL), Mirin(LLNL), Rotman(LLNL), Taylor(ANL), Worley(ORNL)Co-I (NCAR&NASA): Bettge(NCAR), Kiehl(NCAR), Craig(NCAR), Deluca(NCAR), Lin(DAO), Washington(NCAR), Williamson(NCAR)

Goals for SciDAC CCSM Collaborative Development

• Comprehensive treatment of physical and chemical processes– High resolution ocean and atmosphere support– Hybrid vertical coordinate in ocean code – Tropospheric chemistry package– Biogeochemistry

• Modular “packages” with well defined interfaces and testing procedures– Atm, Coupler (Avant Garde), Ice, Ocn, Lnd

• Performance optimized yet portable and adaptable for utilization of emerging architectural features of terascale computers

• Ready for DOE, NSF and NASA applications– High resolution historical and climate change scenario studies– Carbon cycle studies and data assimilation for climate

Climate Science Enabled

Science 13April 2001: “Detection of Anthropogenic Climate Change in the Worlds Oceans,” Barnett, Pierce, Schnur

Method: Ensemble simulations of the DOE Parallel Climate Model (PCM)

Results: • Detection of Anthropogenic Climate Change in the Worlds Oceans• Ensembles establish 95% confidence intervals of model predictions• Simulated ocean heat storage matches historical record of rising ocean

temperatures

Enabling Technology: • Parallel Climate Model

developed in collaborative effort lead by Warren Washington (NCAR)

• Terascale computing resources

Firsts: • Ensemble study with US model and computers • Coupled model reproducing ocean response• Establishing new level of US model quality

Software Infrastructure and Applied Math Challenges

Challenge Existing Solutions Enhanced Solutions

Project management, Version control and testing infrastructure for competing, distributed development teams

CVS with monitoring procedures BitKeeper? SourceForge? Variety of systems and institutions.

Evolving Three layer architecture: Library(eg. FFT), Utility(eg. orbital calendar, data transpose) and Model ( eg. radiation physics)

F90 modules, vendor math libraries, customized data movement, PILGRM, MCT, …Open design process

Optimized math libraries, performance monitors, CCA, NASA Earth System Modeling Framework(ESMF)

Accurate, fast dynamical methods: atm, ocn, ice

Spectral horizontal, finite volume, conservative semi-Lagrangian advection, hybrid vertical coordinates, fast Helmholtz solvers, two-time level monotone advection

Scalable algorithms, grid technologies, nonlinear solvers, new discretization techniques, subcycling explicit barotropic modes, new formulations

Incorporating new model components and improving coupled model climate simulation

Control simulations, component working group activities, scientific steering and review of non-linear interactions

Available cycles, improved analysis capabilities and extensive data handling capabilities, collaborative technologies

Unknown source ISIC and external Internal to project

Software Engineering Challenges

Software Tasks• Develop comprehensive design documents for each

component of the model• Implement performance-portable, run-time configurable

CCSM on target machines• Incorporate emerging programming paradigms and

software design practices• Develop testing and validation procedures for all

component models• Incorporate new dynamical components for ocean,

atmosphere and sea ice.• Incorporate new modular physical and chemical process

models

Task List Excerpt and Schedule

Date Component Milestone Deliverable Lead Lab

02Q1 Ocean Reviewed requirements document Report LANL

02Q1 Land Complete requirements document Report NCAR

02Q2 Ocean Hybrid programming model completed in POP Released code LANL

02Q2 OceanComplete blending of orthobaric surfaces with z-levels in HYPOP

Progress report LANL

02Q2 Coupler Load balancing in MCT New release of MCT ANL

02Q3 Atmosphere Demonstrate tuned and optimized model at T85 Validated code ORNL

02Q3 Atmosphere Implement 3D block decomposition in all dycores Benchmarked code ORNL

02Q3 Sea ice Complete requirements document Report LANL

02Q3 Atmosphere Subgrid topography scheme applied to dycores Progress report PNNL

02Q4Atmospheric chemistry

Complete development of chemistry solver Benchmarked code LLNL

03Q1 Atmosphere Separate dynamics and transport New release of dynamical core ORNL

03Q1 Atmosphere Demonstrate T31 performance optimization Validated code ANL

03Q1 Land Complete cache-friendly decomposition Benchmarked code ORNL

03Q1 Sea ice Use automatic differentiation to tune parameters in CICEImproved values of model parameters

ANL

03Q2 Ocean MLP and dynamic load balancing in POP Benchmark report, released code LANL

03Q2 Sea iceComplete hybrid programming model in CICE with subblocked decomposition and load balancing

Benchmarked code LANL


Test GHG distributions with tropospheric chemistry forcing Progress report LLNL

041Q OceanFinish tracer validation tests in HYPOP (ready for biogeochemistry)

LANL

042Q Coupler New data types and structures New realease of MCT ANL

04Q3 All Parallel IO Release of IO library LBNL

05Q1 Ocean Complete validation of HYPOP in coupled model tests Validated code LANL


Test GHG distributions with stratosphereic chemistry forcing added

Progress report LLNL

05Q3 CouplerIntegration of coupler with CCSM utility and machine-specific layers

New release of MCT ANL


Complete addition of aerosol physics Validated code NCAR

• Chemical solver technologies

• GHG with transport

• Interactive ozone : sulfur cycle and aerosols

• Upper ocean dimethyl sulfide

• Load balancing

• Off-line chemical simulations (DAM)

• Support C4MIP

• Multiple resolution atm model– T31, T42, T85 and T170 – Sub-grid orography

precip scheme• Scalable dycores

– 2d, 3d blocks, MLP tests

– Optimized and load balancing physics

• Design documents

• Chunking for cache performance

• Load balanced, m to n transfers

• Land Surface Modeling Toolkit as part of utility layer

Next Generation Couplers

• MCT version 2– Higher level abstractions, component model registry– Scalability to thousands of processors– Dynamic load balancing– 3d fields

• Interoperation of climate, weather and data assimilation functions

• MCT as CCA compliant prototype for Earth System Modeling Framework

SciDAC CCSM – ISIC Collaboration Goals

• Improve performance characterizations• Accelerate development of mathematical software

for climate modeling and analysis• Explore novel methods and advance the theory

and simulation of geophysical flows• Improve software practices for the scientific

endeavor of climate change prediction• Enhance analysis and data handling methods

ISIC Projects

• Performance Evaluation(D. Bailey) 02Q1– PCTM, CCSM Instrumentation - P. Worley (ORNL)

• Earth System Grid (Ian Foster) 02Q3– Data archive and analysis grid - Bernholdt(ORNL)

• Data Access (Arie Shoshani) 02Q3– Improved efficiencies and monitoring - Samatova

(ORNL)

ISIC Projects

• Grids (Glimm, Brown) 03Q1– Ocean nested grids – Smith (LANL), Brown(LLNL)

– Smooth grid transformations – Drake(ORNL), Khamayseh(ORNL)

– Spectral element refinement strategies – Fournier (NCAR, U.Md), Fisher(ANL),Taylor&Wingate(LANL)

– Adaptive models – Joyce Penner (U. Mich)

ISIC Projects

• PDE methods (Phil Collela) 03Q1– Adaptive diagnostics (Drake-Williamson)– Static adapted western boundary currents Malone(LANL)

• Solvers (David Keyes) 03Q4– Baroclinic model (a new discretization) – Krylov methods – Balou Nadiga (LANL)

• Common Component Architecture (Rob Armstrong) 04Q1– Model Coupler Toolkit – Larson (ANL)– CCA compliant CCSM

SciDAC Climate ProjectsProject Title Institutions Involved Principal InvesigatorPredictive Understanding of the Oceans' Wind-Driven Circulation on Interdecadal Time Scales Univ of Indiana Michael Ghil, UCLA

Decadal Variability in the Coupled Ocean-Atmoshere Systems Paola Cessi, UCSDA Geodesic Climate Model with Quasi-Lagrangian Vertical Coordinates LANL, Clarkson University David Randall, Colorado State University

Improving the Processes of Land-Atmoshpere Interactions in CCSM 2.0 at High Resolution University of Arizona Robert Dickinson, Georgia Tech Research Corp.Continuous Dynamic Grid Adaptation in a Global Atmoshperic Model William Gutow ski, Iow a State university

Collaborative Design and Development of the Community Climate System Model for Terascale Computers

ORNL. LBNL, PNNL, ANL, LLNL, National center for Atmospheric Research Robert Malone, LANL

Decadal Regional Climate Studies and Applications with Variable-Resoulation GCMs Using Advanced Numerical Techniques Michael Fox-Rabinovitz, University of Maryland

Multi-Resolution Climate Modeling Ferdinand Baer, University of Maryland

Development of an Atmospheric Climate Model with Self-Adapting Grid and Physics Joyce Penner, University of Michigan

Testing a New Hybrid ocean Circulation Model Based on POP LANL Kirk Bryan, Princeton University

Decadal Regional Climate Studies and Applications with Variable-Resoulation GCMs Using Advanced Numerical Techniques Jean Cote, University of Quebec

Modeling Dynamic Vegetation for Decadal to Multi-Century Climate Change Studies Goddard Institute for Space Studies Andrew Friend, Rutgers University

Towards the Prediction of Decadal to Multi-Century Processes in a High-Throughput Climate System Model

Zhengyu Liu, University of WisconsinJohn Kutzbach, University of Wisconsin

Modeling and Analysis of Global and Regional Hydrologic Processes and Appropriate Conservation of Moist Entropy Donald Johnson, University of Wisconsin

Global Climate Research