climate modeling and ipcc data management - ncar library · 2020-01-06 · climate modeling and...

Climate Modeling andIPCC Data Management

DCERC Data Curation WorkshopGary Strand

[email protected]

Monday, June 11, 2012

mailto:[email protected]

mailto:[email protected]

A brief history of climate modeling

1904 Vilhelm Bjerknes1922 Lewis Fry Richardson

2013 IPCC AR5, petascale

1940s ENIAC1950s NWP and climate1960s NCAR; ocean modeling1970s Land and sea ice modeling1980s Early coupled models1990s First IPCC report2007 IPCC AR4, Nobel Prize


ATMOSPHERE

SEA ICE

BIOGEOCHEMICALCYCLES

SULFATEAEROSOLS

2010-2012

INTERACTIVEVEGETATION

CARBON/NITROGEN

CYCLE

DUST/SEA SPRAY/

MINERAL AEROSOLS

SOLAR FORCINGVOLCANICAEROSOLS

ICE SHEET

OCEAN

SULFATEAEROSOLS

OCEAN

2000-2008

SOLAR FORCINGVOLCANICAEROSOLS

CARBONCYCLE

DUST/SEA SPRAY/

MINERAL AEROSOLS

VEGETATION

ATMOSPHERE

SEA ICE

1990s

SULFATEAEROSOLS

ATMOSPHERE/LAND SURFACE/

VEGETATION

OCEAN

SEA ICE

1970s-1980sATMOSPHERE/

LAND SURFACE/VEGETATION

OCEAN

SEA ICE

Mid-1960s

ATMOSPHERE/LAND SURFACE

OCEAN

The Development of Climate Models

Past, Present, and FutureMonday, June 11, 2012

Schematically

Processes

Grid


supercomputer

A machine that takes one problem (computation) and turns it into

more - I/O, storage, and management.

|ˈsoōpərkəmˌpyoōtər|


What is “CESM”?CommunityEarthSystemModel

CESM belongs to an elite category of computer-based simulations known as earth system models. Such models use mathematical formulas to recreate the chemical and physical processes that drive Earth’s climate. Extraordinarily sophisticated, they incorporate phenomena ranging from the effect that volcanic eruptions have on temperature patterns to the impact of shifting sea ice on sunlight in the atmosphere. What emerges from trillions of computer calculations is a picture of the world’s climate in all its complexity.


Improved geographic resolution

1990s(~300 km)

2000s(~150 km)

2010s(~30 km)

1980s(~800 km)


A brief history of the IPCC & data1990 - First Assessment (FAR)

5 modeling groups, 8 models, 7 simulations

2013 - Fifth Assessment (AR5)21 modeling groups, 25+ models, 45 simulation types (decadal prediction and long-term)

1995 - Second Assessment (SAR)8 modeling groups, 10 models, 6 “IS92a” simulations

2001 - Third Assessment (TAR)7 modeling groups, 8 models, 6 “SRES” simulations

2007 - Fourth Assessment (AR4)16 modeling groups, 24 models, 12 simulation types


Impact of IPCC on data volume

0

1,200,000

2,400,000

3,600,000

4,800,000

6,000,0002

00

0-0

1

20

00

-07

20

01

-01

20

01

-07

20

02

-01

20

02

-07

20

03

-01

20

03

-07

20

04

-01

20

04

-07

20

05

-01

20

05

-07

20

06

-01

20

06

-07

20

07

-01

20

07

-07

20

08

-01

20

08

-07

20

09

-01

20

09

-07

20

10

-01

20

10

-07

20

11

-01

20

11

-07

20

12

-01 0

400,000

800,000

1,200,000

1,600,000

2,000,000

CMIP5 (IPCC AR5)

CMIP3 (IPCC AR4)

cumulative files

cumulative volume


IPCC AR5 data format standard

• Single variable per netCDF-3 file• File sizes must be < ~4 GB (as practical)• Stringent file naming convention• Defined temporal, horizontal and vertical

resolutions• Specific fields (can be the same as model

output but also derived)

Quite detailed (PDF is 167 pages):


IPCC AR5 data format standard


IPCC AR5 data format standardCMOR2: Climate Model Output Rewriter (v2)

• Each file contains a single primary output variable, with coordinate/grid variables, attributes and other metadata, with flexibility in how many time slices are stored

• Metadata is defined in MIP-specific tables• For IPCC AR5, CMOR2 uses netCDF-3•CMOR uses udunits2 to verify (and possibly convert)

to the “units” attribute specified by the MIP table•CMOR uses uuid to produce a unique tracking

number for each file


IPCC AR5 variable counts1 hour 3 hour 6 hour daily month annual totals

aerosol

atmosphere

land

land ice

ocean

biogeochemistry

0 0 0 0 81 0 81

75 104 9 76 184 0 448

0 3 0 2 58 0 63

0 0 0 2 13 0 15

0 1 0 3 116 0 120

0 0 0 0 88 71 159

sea ice

totals

0 0 0 4 47 0 51

75 108 9 87 587 71 937


float TS(time, lat, lon) TS:units = "K" TS:long_name = "Surface temperature (radiative)" TS:cell_method = "time: mean"

Standard CESM output for a specific variable

IPCC AR5 metadata standard


float ts(time, lat, lon) ts:standard_name = "surface_temperature" ts:long_name = "Surface Temperature" ts:comment = "skin" temperature (i.e., SST for open ocean)" ts:units = "K" ts:original_name = "TS" ts:cell_methods = "time: mean (interval: 30 days)" ts:cell_measures = "area: areacella" ts:history = "2011-07-22T00:05:32Z altered by CMOR: replaced missing value flag (-1e+32) with standard missing value (1e+20)." ts:missing_value = 1.e+20f ts:_FillValue = 1.e+20f ts:associated_files = "baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_CCSM4_historical_r0i0p0.nc areacella: areacella_fx_CCSM4_historical_r0i0p0.nc"

IPCC AR5 metadata standardSame variable as required by IPCC AR5


http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation




:Conventions = "CF-1.0":source = "CAM":case = "b40.20th.track1.1deg.006":title = "UNSET":logname = "mai":host = "be0809en.ucar.ed":Version = "$Name$":revision_Id = "$Id$":initial_file = "b40.1850.track1.1deg.006.cam2.i.0893-01-01-00000.nc":topography_file = "/fis/cgd/cseg/csm/inputdata/atm/cam/topo/USGS-gtopo30_0.9x1.25_remap_c051027.nc":nco_openmp_thread_number = 1

Standard CESM global attributes



:project_id = "CMIP5":product = "output":frequency = "mon":modeling_realm = "atmos"

:institution = "NCAR (National Center for Atmospheric Research), Boulder, CO, USA":institute_id = "NCAR"

:model_id = "CCSM4":source = "CCSM4 (tag: ccsm4_0_beta43 compset: B20TRCN)":references = "Gent P. R., et.al. 2011: The Community Climate System Model version 4. J. Climate, doi: 10.1175/2011JCLI4083.1":resolution = "f09_g16 (0.9x1.25_gx1v6)":title = "CCSM4 model output prepared for CMIP5 historical":contact = "[email protected]"

:acknowledgements = "The CESM project is supported by the National Science Foundation and the Office of Science (BER) of the U.S. Department of Energy. NCAR is sponsored by the National Science Foundation. Computing resources were provided by the Climate Simulation Laboratory at the NCAR Computational and Information Systems Laboratory (CISL), sponsored by the National Science Foundation and other agencies."

Global attributes as required by IPCC AR5


:experiment = "historical":experiment_id = "historical":forcing = "Sl GHG Vl SS Ds SD BC MD OC Oz AA LU"

:realization = 1:initialization_method = 1:physics_version = 1

:parent_experiment = "pre-industrial control":parent_experiment_id = "piControl":parent_experiment_rip = "r1i1p1":branch_time = 937.

:tracking_id = "d33ccf77-a73c-4f55-8f02-3a0734d51151"

:creation_date = "2011-07-22T00:05:32Z":history = "2011-07-22T00:05:32Z CMOR rewrote data to comply with CF standards and CMIP5 requirements.":Conventions = "CF-1.4":table_id = "Table Amon (27 April 2011) a5a1c518f52ae340313ba0aada03f862":cmor_version = "2.7.1"

Global attributes as required by IPCC AR5


:cesm_casename = "b40.20th.aero.1deg.008" ;:cesm_repotag = "ccsm4_0_beta56" ;:cesm_compset = "B1850CN" ;

:resolution = "f09_g16 (0.9x1.25_gx1v6)" ;

:forcing_note = "Additional information on the external forcings used in this experiment can be found at http://www.cesm.ucar.edu/CMIP5/forcing_information" ;

:processed_by = "strandwg on silver.cgd.ucar.edu at 20120205 -073243.338" ;:processing_code_information = "Last Changed Rev: 525 Last Changed Date: 2012-02-04 13:11:55 -0700 Repository UUID: d2181dbe-5796-6825-dc7f-cbd98591f93d" ;

Additional global attributes


http://www.cesm.ucar.edu/CMIP5/forcing_information





b40.20th.track1.1deg.006.cam2.h0.TS.1850-01_cat_2005-12.nc

CESM filename

/CCSM/csm/b40.20th.track1.1deg.006/atm/proc/tseries/monthly

CESM archival location

ts_Amon_CCSM4_historical_r1i1p1_185001-200512.nc

IPCC AR5 filename

.../CMIP5/output/NCAR/CCSM4/historical/mon/atmos/ts/r1i1p1

IPCC AR5 location


0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000B

CC

R

CA

WC

R

CC

CM

A

CN

RM

CSI

RO EC

GFD

L

GIS

S

IAP

ING

V

INM

CM

3

IPSL

MET

RI

MIR

OC

3

MIU

B

MPI

MR

I

NC

AR

Nor

Clim

U R

eadi

ng

UK

MO

AR4 data volumes by groupIPCC AR4 by group total: 35 TB


IPCC AR4 distribution

Modeling centers (16) Users (1000s)

Gateway (1)


0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

900,000B

CC

R

CA

WC

R

CC

CM

A

CN

RM

CSI

RO EC

GFD

L

GIS

S

IAP

ING

V

INM

CM

3

IPSL

MET

RI

MIR

OC

3

MIU

B

MPI

MR

I

NC

AR

Nor

Clim

U R

eadi

ng

UK

MO

IPCC AR5 volumes by group

total: 2,200 TB

total: 35 TBIPCC AR4 by group

IPCC AR5 by group


IPCC AR5 distribution

Modeling centers (24)Gateways (9)Nodes (14)

Users (1000s)


Distribution: The ESG federation

!Monday, June 11, 2012

METAFORCommon Metadata for Climate Modelling Digital Repositories

• Develop a Common Information Model (CIM) to describe climate data and the models that produce it in a standard way

• Ensure the wide adoption of the CIM

• Addresses the fragmentation and gaps in availability of metadata as well as duplication of information collection and problems of identifying, accessing or using climate data that are currently found in existing repositories.


METAFORCommon Information Model (CIM)


METAFOR - CIMThe CIM (Common Information Model) is a domain model of

the concepts and relationships used in climate modeling:•It includes descriptions not only of the data, but also of the

models that generated that data, the simulations those models implemented, the experiments for which those simulations were run, the people and institutions involved, and why they bothered.

•It tries to describe the full provenance of climate modeling artifacts

•It is an emerging standard•It’s the core of a related set of tools and services•It’s the infrastructure around which IPCC AR5 metadata is

based


Data QC


Data QCQC Level 1 QC Level 2 QC Level 3

DescriptionCMOR2 and ESG publisher conformance checks

Data consistency checksDouble- and cross-checks of data and metadata and data publication as DataCite DOI

Data

preliminary; no user notification about changes;performed for all data;metadata may not be complete

no user notification about changes;performed for CMIP5 requested metadata and data

published and persistent data with version and unique DOI as persistent identifier;user notification about changes;performed for replicated data

Accessconstrained to CMIP5 modeling centers

constrained to non-commercial research and educational purposes

constrained to non-commercial research and educational purposes, or open for unrestricted use (as specified by the modeling centers)

Access Control

PCMDI on behalf of WMO/WGCM

PCMDI, BADC, WDCC/DKRZ core data archives on behalf of WMO/WGCM

IPCC-DDC on behalf of TGICA

Citation no citation reference informal citation reference formal citation reference

Quality Flag "automated conformance checks passed"

"subjective quality control passed"

"approved by author" (in case of newer DOI available: "approved by author, but suspended")


Possible futuresAssuming there is an IPCC AR6...• The current model, even with federated

distribution, will likely not work• Model resolution will preclude access of

global fields• “Little big iron”• More-exotic solutions


CESM websitehttp://cesm.ucar.edu

CESM Data Management Planhttp://www.cesm.ucar.edu/management/docs/data.mgt.plan.2011.pdf

CMIP5 websitehttp://cmip.llnl.gov/cmip5

Websites


http://cf-pcmdi.llnl.gov

http://cf-pcmdi.llnl.gov

http://www.cesm.ucar.edu/management/docs/data.mgt.plan.2011.pdf

http://www.cesm.ucar.edu/management/docs/data.mgt.plan.2011.pdf

http://cmip.llnl.gov

http://cmip.llnl.gov

climate modeling and ipcc data management - ncar library · 2020-01-06 · climate modeling and...

Documents