E th S t M d liE th S t M d liEarth System Modeling: Earth System Modeling: New Directions for aNew Directions for aNew Directions for a New Directions for a Predictive SciencePredictive Science

J h B D kJ h B D kJohn B. DrakeJohn B. DrakeComputational Earth Science GroupComputational Earth Science Group

Oak Ridge National Laboratory Oak Ridge National Laboratory

PresentationMarch 11, 2009

Wh M d l h Cli ?Wh M d l h Cli ?Why Model the Climate?Why Model the Climate?M ltiM lti scale chaotic s stemscale chaotic s stem MultiMulti--scale chaotic systemscale chaotic system

Internal modes and forced Internal modes and forced modesmodes variability andvariability andmodes modes -- variability and variability and climate changeclimate change

K dKey advances•Weather forecast skill and data assimilation techniques•ENSO phenomena and pteleconnections•Ocean thermohaline circulation and abrupt change•GHG and aerosol effects

The anthropogenic climate change “fingerprint”. In The anthropogenic climate change “fingerprint”. In the absence of humanthe absence of human--induced changes to the induced changes to the atmosphere, the earth would be in a cooling trend atmosphere, the earth would be in a cooling trend (simulations were carried out using NCAR, ORNL (simulations were carried out using NCAR, ORNL and NERSC computing platforms)and NERSC computing platforms)

•GHG and aerosol effects on radiation balance•Coupled GCMs with ensemble studies

GlobalGlobal--average radiative forcing average radiative forcing estimates and rangesestimates and rangesestimates and rangesestimates and ranges

The Earth System is a Coupled System: The Earth System is a Coupled System: Wh t h t di t?Wh t h t di t?What can you hope to predict?What can you hope to predict?

New predictive skill for decadal prediction is needed to inform adaptation decisions and better understand the consequences of climate change

High resolution atmosphereHigh resolution atmosphere

C CC CClimate Change AttributionClimate Change Attribution

Past climates, Milankovich cyclesPast climates, Milankovich cycles

Projections of Future Changes in ClimateProjections of Future Changes in ClimateProjections of Future Changes in ClimateProjections of Future Changes in Climate

For the next two decades a warming of about For the next two decades a warming of about 0.20.2°°C C per decadeper decade is projected for a range of SRES emissionis projected for a range of SRES emissionper decadeper decade is projected for a range of SRES emission is projected for a range of SRES emission scenarios. scenarios.

Even if the concentrations of all greenhouse gases Even if the concentrations of all greenhouse gases g gg gand aerosols had been kept constant at year 2000 and aerosols had been kept constant at year 2000 levels, a further warming of levels, a further warming of about 0.1about 0.1°°C per decadeC per decadewould be expectedwould be expectedwould be expected. would be expected.

Earlier IPCC projections of Earlier IPCC projections of 0.15 to 0.3 0.15 to 0.3 ooCC per decade per decade can now be compared with observed values of 0.2can now be compared with observed values of 0.2 ooCCcan now be compared with observed values of 0.2 can now be compared with observed values of 0.2 C C

Projections of Future Changes in ClimateProjections of Future Changes in ClimateProjections of Future Changes in ClimateProjections of Future Changes in Climate

Projections of Future Changes in ClimateProjections of Future Changes in Climate

Projected warmingin 21st century

j gj g

in 21st century expected to be

t t l dgreatest over land and at most high northern latitudes

and least over the Southern OceanSouthern Ocean and parts of the North Atlantic OOcean

Projections of Future Changes in ClimateProjections of Future Changes in Climate

Precipitation increases very likely in high latitudes

Decreases likely in most subtropical land regions

A i Th ?A i Th ?Arctic Thaw?Arctic Thaw?

NASA Observations of Ice CapNASA Observations of Ice Cap

Low Emission Scenario Topics• Can we stabilize global warming using the new CCSP Report 2.1a scenarios?• Can we limit global warming to 2ºC from years 1870 to 2100?• Can we limit global warming to 2 C from years 1870 to 2100?• What are climate change impacts on surface temperature, precipitation, and sea ice?

ReferenceReference

Level 4Level 4

Temperature Change at the Surface for DJF and JJAL E i i Mi C it t d A2 Mi C it t

Level 1Level 1

Level 3Level 3

Level 2Level 2

Low Emission Minus Commitment and A2 Minus Commitment

CCSP SAP 2.1a: COCCSP SAP 2.1a: CO2 2 EmissionEmissionR d ti S iR d ti S i

Sea Ice Changes for February, March, April and August, Sea Ice Changes for February, March, April and August,

ObsReduction ScenariosReduction Scenarios~ 70% cut in carbon emissions by ~ 70% cut in carbon emissions by

the end of centurythe end of century

Conclusion: It is possible to limit global warming to 2ºC from 1870 to 2100 and reduce Arctic sea ice melt by using the CCSP level 1 scenario This will require a substantial decrease in the use of fossil fuels starting in the next decade or so.

g y, , p g ,g y, , p g ,September, OctoberSeptember, October

What is the Urgency?What is the Urgency?What is the Urgency?What is the Urgency?

• Carbon management is the key challenge for mitigating climate change in the next century

• Predicting regional climate change and its ill h t tilit iconsequences will have great utility in

adapting to climate change

Can we evaluate bioenergy scenarios? Can we evaluate bioenergy scenarios? An ORNL LDRD projectAn ORNL LDRD project

Considerations of biological Considerations of biological suitabilitysuitability

Quantifying spatial distribution ofQuantifying spatial distribution ofQuantifying spatial distribution of Quantifying spatial distribution of land that:land that:–– Is availableIs available–– Has suitable biological, Has suitable biological, g ,g ,

environmental conditions to meet environmental conditions to meet demanddemand

Balancing economic constraintsBalancing economic constraints

Carbon Land Model Carbon Land Model Intercomparison (CIntercomparison (C--LAMP)LAMP)

What are the What are the l tl trelevant processes relevant processes

for carbon in the for carbon in the next version of the next version of the CCSM?CCSM?

Comparison of Comparison of CASA’ CN andCASA’ CN andCASA’, CN, and CASA’, CN, and IBISIBIS

Ice Sheet ModelIce Sheet ModelWilliam Lipscomb (LANL)William Lipscomb (LANL)William Lipscomb (LANL)William Lipscomb (LANL)

• The rate of 21st century ice sheet melting and sea level rise is extremely uncertain and is now recognized as a high priority for climate models.

• We have coupled the GLIMMER model to CCSM and will• We have coupled the GLIMMER model to CCSM and will soon begin climate experiments with a dynamic Greenland ice sheet.

•GLIMMER has been added to CCSM, with wrapper code for exchanging fields between GLC and the couplerexchanging fields between GLC and the coupler.

• The Community Land Model, CLM, has been modified to compute the ice sheet surface mass balance in glaciated columns and pass the mass balance to GLC via the coupler.

• More detailed ice dynamics is being added and we will soon• More detailed ice dynamics is being added and we will soon begin to address ocean/ice interactions.

Figure 8: Greenland topography in GLIMMER

Figure (a) Reconstructed GIS from the last interglacial (Cuffey and Marshall, 2000), when sea level was about 6 m higher than today (b) Effect of a 6 m seahigher than today. (b) Effect of a 6 m sea level rise on the southeast United States (Weiss and Overpeck, University of Arizona).

Before and AfterBefore and Afterthe IPCC AR4the IPCC AR4

Observational record establishes global warmingObservational record establishes global warming Atmospheric climate models establish role of GHG forcingAtmospheric climate models establish role of GHG forcing Coupled modeling identifies warming signatures in ocean Coupled modeling identifies warming signatures in ocean p g g gp g g g

basins, troposphere basins, troposphere Coupled models project future climates with uncertaintiesCoupled models project future climates with uncertainties High resolution studies project first regional effects, heat wavesHigh resolution studies project first regional effects, heat waves Lack of clarity on hurricane frequencyLack of clarity on hurricane frequency Unable to provide good sea level rise estimates …Unable to provide good sea level rise estimates … Unable to balance the carbon budgetUnable to balance the carbon budgetgg Unable to incorporate biogeochemistry for impacts on air quality, Unable to incorporate biogeochemistry for impacts on air quality,

ecosystems, agricultureecosystems, agricultureThe questions we are being asked after AR4 areq gmore challenging and of a different character

Historical Development of Weather and Cli t M d li

Mid-1970s

Atmosphere

Mid--1980s

Atmosphere

Early 1990s

Atmosphere

Late 1990s

Atmosphere

Present Day

Atmosphere

Late 2000’s

Atmosphere Weather

Climate Modeling

Atmosphere Atmosphere

Land Surface

Atmosphere

Land Surface

Ocean & Sea Ice

Atmosphere

Land Surface

Ocean & Sea Ice

Sulphate

Atmosphere

Land Surface

Ocean & Sea Ice

Sulphate

Atmosphere

Land Surface

Ocean & Sea Ice

Sulphate

Weather

ClimateVariability

SulphateAerosol

SulphateAerosol

Non-sulphateAerosol

Carbon Cycle

SulphateAerosol

Non-sulphateAerosol

Carbon Cycle

ClimateChange

y y

Dynamic

Earth System St di

Ice Sheets

Ice Sheets

DynamicVegetation

AtmosphericChemistry

Ocean

Managed Systems

Human Factors

Studies

Sustainability Studies

Ocean Ecology

Economics

CCSM Development, the Climate End Station, and pIPCC AR5: THE BIG PICTURE

Cli t h•Climate change simulations are compute and storage intensive•Simulation rates are 5-20

/dSlide 8

years/day•Single project is producing 5TB/day in archival output•Ensembles are requiredWorkflo and data•Workflow and data

management are a challenge for single investigators

Earth System GridEarth System Grid InternationalInternationalEarth System Grid Earth System Grid ---- International International Distribution of Simulation ResultsDistribution of Simulation Results

International central site: Earth System Grid International central site: Earth System Grid –– Sponsored by DOE SciDAC project. Integrates major centersSponsored by DOE SciDAC project. Integrates major centers

for supercomputing and analysis coordinated internationally through PCMDIfor supercomputing and analysis coordinated internationally through PCMDI–– IPCC AR4: 12 experiments, 24 models, 17 climate centers, 13 nationsIPCC AR4: 12 experiments, 24 models, 17 climate centers, 13 nations

CC LAMP i tLAMP i t–– CC--LAMP experimentsLAMP experiments Archive status and activity:Archive status and activity:

–– 6000 registered users6000 registered users–– Downloaded: Downloaded: 250 Terabytes in 2007250 Terabytes in 2007–– Current contents:Current contents: 100,000 simulated years of data100,000 simulated years of data–– Data sets:Data sets: 1M files, 180 Terabytes1M files, 180 Terabytes–– New portals:New portals: ORNL, NCARORNL, NCAR

Access point: Access point: https://www.earthsystemgrid.org/https://www.earthsystemgrid.org/

Computational RequirementsComputational RequirementsIssue Motivation Compute FactorSpatial resolution Provide regional details 103-105

Model completeness Add “new” sc ience 102

New parameterizations Upgrade to “better” science 102

Run length Long-term implications 102

Ensembles, scenarios Range of model variability 10Total Compute Factor 1010-1012

A Science Based Case for Large-Scale Simulation(SCaLeS), SIAM News, 36(7), 2003 - David Keyes

Establishing a PetaScale Collaboratory for the GeosciencesUCAR/JOSS May 2005UCAR/JOSS, May 2005

Early Computers Early Computers -- Weather and Climate are Weather and Climate are y py pSignature ApplicationsSignature Applications

Computational Power Computational Power -- Factor of Factor of a billion growth since 1960a billion growth since 1960

Fixed clock speed implies exponential growth of processing threads

YearYear 20052005 20102010 20152015 20202020

ThreadsThreads 768768 184000184000 793600793600 3.2M3.2M

Baker 1 PF

Performance portability, extensibility and scalability

Performance portable: transfers between Performance portable: transfers between architectures maintaining performancearchitectures maintaining performance

MPI–– Cache Cache –– vector reconfigurable data structurevector reconfigurable data structure–– TunableTunable-- Hybrid MPIHybrid MPI--OpenMP parallelismOpenMP parallelism

22--d patches blocks clumps stripsd patches blocks clumps strips

MPI

22--d patches, blocks, clumps, strips, …d patches, blocks, clumps, strips, … Vertical columns (load balanced), segmentedVertical columns (load balanced), segmented Efficient transpose and regriding operatorsEfficient transpose and regriding operators

E t ibl t li ftE t ibl t li ftOpenMP

Extensible: component coupling softwareExtensible: component coupling software–– Model coupling toolkit (MCT): utility layer for writing Model coupling toolkit (MCT): utility layer for writing

couplerscouplerspp–– Attribute vectors and conservative regriding via parallel Attribute vectors and conservative regriding via parallel

matrix multiplymatrix multiplyStatic load balancingStatic load balancing

26

Static load balancingStatic load balancing

Th Ch ll f S l biliTh Ch ll f S l biliThe Challenge of ScalabilityThe Challenge of Scalability Scalability for petascale science Scalability for petascale science

applicationsapplicationspppp–– Physical/ chemical/ biologicalPhysical/ chemical/ biological–– Fine grain parallelism to ~100K Fine grain parallelism to ~100K

processorsprocessors–– What about 1M processors?What about 1M processors?

Fl ibilitFl ibilit FlexibilityFlexibility–– Component parallelsim (MPI Component parallelsim (MPI ––

overlapping), sequential/ overlapping), sequential/ concurrentconcurrent

–– Roadrunner/ Cell processorRoadrunner/ Cell processor–– IBM BG/P,Q,…IBM BG/P,Q,…–– Load balanceLoad balance–– Paralllel I/O layerParalllel I/O layer–– Checkpoint fault recoveryCheckpoint fault recovery

Reproducibility traditionReproducibility tradition–– Reproducibility traditionReproducibility tradition

Eliminating Algorithmic Scalability Bottlenecks

CAM scaling for both spectral and finite volume dynamical cores are now limited by performance, not artificial algorithmic limitations. Are now addressing these performance limiters.

For T85L26 grid and spectral Eulerian dynamics, can use 1024 MPI tasks productively, 8 times that of the original 128 MPI task limit.

For 2 degree grid resolution and Finite Volume dynamics, can use 1024 MPI tasks productively, 4 times that of the original 256 MPI task limit.

28

Evaluation of Methods:Evaluation of Methods:PDEs on the SpherePDEs on the SpherePDEs on the SpherePDEs on the Sphere

April 6April 6--9, 2009, Santa Fe9, 2009, Santa Fe Finite volume withFinite volume with

Advection schemesAdvection schemes Discretization methodsDiscretization methods Comparison study of Comparison study of

methodsmethods

Finite volume with Finite volume with Lagrangian verticalLagrangian vertical

–– LatLat--lon grid (now the lon grid (now the standard CAM3.5 dycore)standard CAM3.5 dycore)methodsmethods

Computational Computational performanceperformance

Discussion and proposal of Discussion and proposal of t tt t

–– Cubed sphere grid Cubed sphere grid (hydrostatic and non(hydrostatic and non--hydrostatic hydrostatic -- GFDL)GFDL)

Spectral element andSpectral element andtest casestest cases Global hydrostatic and Global hydrostatic and

nonnon--hydrostatic modelshydrostatic models

Spectral element and Spectral element and discontinuous Galerkindiscontinuous Galerkin

–– Cubed sphere Cubed sphere (HOMME/CCSM (HOMME/CCSM f k)f k)framework)framework)

SemiSemi--Lagrangian spectral Lagrangian spectral with Lagrangian verticalwith Lagrangian vertical

The choice of numerical method and the resolution still has an appreciable effect on the solution even for simple cases.

Computational Methods Computational Methods ResearchResearch

Implicit methods (Evans)Implicit methods (Evans)–– SemiSemi--implicit and fully implicitimplicit and fully implicit–– NewtonNewton--Krylov and hierarchical preconditionersKrylov and hierarchical preconditioners

Long timestep methods (Archibald, Drake, Fann,Long timestep methods (Archibald, Drake, Fann, Long timestep methods (Archibald, Drake, Fann, Long timestep methods (Archibald, Drake, Fann, White)White)

–– SemiSemi--LagrangianLagrangian–– Exact Linear Part (ELP) methodsExact Linear Part (ELP) methods–– Krylov Differed Corrections (KDC) method Krylov Differed Corrections (KDC) method y ( )y ( )–– Parallel in timeParallel in time

New vertical discretizations (Drake)New vertical discretizations (Drake)–– Lagrangian and isentropic systemsLagrangian and isentropic systems–– DivergenceDivergence--vorticity formulationsvorticity formulationsDivergenceDivergence vorticity formulationsvorticity formulations

MultiMulti--resolution horizontal representations resolution horizontal representations (Archibald, Drake, Fann)(Archibald, Drake, Fann)

–– Wavelets and curveletsWavelets and curvelets–– Adaptive SpectralAdaptive SpectralAdaptive SpectralAdaptive Spectral

CAM Scalable Dycore Integration & EvaluationCAM Scalable Dycore Integration & EvaluationM. Taylor (Sandia), K. Evans (ORNL)M. Taylor (Sandia), K. Evans (ORNL)

Cubed-sphere dycores in CAM (with J. Edwards IBM/NCAR)

– Motivation: more scalable dycores using NCAR's HOMME

P lit d l ith f ll d i b li– Process split model with full dynamics subcycling

– Next steps: evaluation of aqua planet results, interpolation to/from other CCSM component grids. Possible other dycores: GFDL cubed-sphere, CSU Icosahedral

Cubed sphere dycore improvements:Cubed-sphere dycore improvements: – Developed conservative formulation of spectral elements based on

compatibility. First dycore in CAM to locally conserve both mass and energy. Shape preserving.

– Developed efficient hyper-viscosity to replaced element based filtering. p yp y p gThe filter was causing bad grid imprinting in moisture and other fields.

snapshot monthly mean

SummarySummaryS iDAC2 CCSM C ti ill ll b tS iDAC2 CCSM C ti ill ll b t SciDAC2 CCSM Consortium will collaborate SciDAC2 CCSM Consortium will collaborate with NSF and NASA to build the next with NSF and NASA to build the next generation Earth System Modelgeneration Earth System Modelgeneration Earth System Modelgeneration Earth System Model

The Climate End Station will provide a The Climate End Station will provide a significant portion of the development andsignificant portion of the development andsignificant portion of the development and significant portion of the development and climate change simulation resourcesclimate change simulation resources

The Earth System Grid will distribute highThe Earth System Grid will distribute high The Earth System Grid will distribute high The Earth System Grid will distribute high quality climate change simulation resultsquality climate change simulation results