T i t l e

Climate Change

ECMWFandtheCopernicusClimateChangeService:netCDF

KevinMarshCopernicusProductionSection

kevin.marsh@ecmwf.int

(WiththankstocolleaguesatECMWFandCopernicus)

ClimateChange

E C M W F B a c k g r o u n dEuropeanCentreforMedium-RangeWeatherForecasts

Anindependent,intergovernmentalorganisationsupportedbymanyEuropeannations,basedatReadingUK.

Mission:• Produce numericalweatherforecasts;• Scientific/technicalresearchtoimproveforecastskill;• Maintainanarchiveofmeteorologicaldata.

ECMWFprovidesarangeorproductsandservicesincluding:Globalnumericalweatherforecasts,Airqualityanalysis,Atmosphericcompositionmonitoring,Climatemonitoring,Oceancirculationanalysis,Hydrologicalprediction…andmanymore

https://www.ecmwf.int

ClimateChange

R e q u i r e m e n t s f o r s t o r i n g n e t C D F i n M A R S

• ThemainrepositoryofmeteorologicaldataatECMWFisMARS (theMeteorologicalArchivalandRetrievalSystem);

• Itcontains>180petabytesofoperationalandresearchdata,mainlyinGRIB (1&2)andBUFR;

• netCDF isbecomingincreasingly usedforstoringdataatECMWF(e.g NEMOoceandatafromCERA-20C)

• WewouldliketobeabletoarchivethesenetCDF datainMARSdirectly;

• ECMWFwanttomakeCF-netCDF astandardsbased“firstclasscitizen”bypromotingitsuseandincreasingtheabilityofourtoolstosupportit.

ClimateChange

R e q u i r e m e n t s f o r s t o r i n g n e t C D F i n M A R S

• Filesmustbecompatible withexistingMARSinfrastructureandsystems;

• Neededtoprovideguidance onhowCF-netCDF filesshouldbeformedforuseatECMWF;

• Issuesaddressedincludestructureofthefiles,standardnames,useofmultipletimecoordinates,cellmethods,etc…;

• A“Recommendations”documenthasbeenproducedtosummarisethiswork:https://software.ecmwf.int/wiki/display/DGOV/Metadata+recommendations+for+encoding+NetCDF+products+based+on+CF+convention(ECMWFwebloginrequired&mayneedtorequestaccess)

ClimateChange

E C M W F M e t a d a t a R e c o m m e n d a t i o n s

Discusses:• Standardnames• SpatialCoordinates• Multipletimecoordinates• Realization• Cellmethods• Aggregations• Attributes• Metadata

V1.0ReleasedJuly2017

netCDF datainMARS~end2017

ClimateChange

C o p e r n i c u s S e r v i c e s B a c k g r o u n d• CopernicusistheEU’searthobservationprogramme,andisdirectedbyEU

andESAhttp://www.copernicus.eu/;

• Ithas3components:i)Spacecomponentii)In-situmeasurementsiii)Servicestousers;

• Thereare6services tousers(eachaddressingathematicarea)whichhavebeenfundedbytheEU;

• ECMWFistheentrustedentitytoruntheCopernicusClimateChangeService(C3S)andtheCopernicusAtmosphericMonitoringService (CAMS);

• TheClimateDataStore willbeattheheartoftheC3Sinfrastructureandwillprovideinformationaboutpast,present andfuture climateintermsofEssentialClimateVariables andderivedclimateindicators,andmanymore…

https://climate.copernicus.eu/climate-data-store

ClimateChange

C o p e r n i c u s S t r u c t u r e

http://www.copernicus.eu/main/copernicus-brief

C3S:https://climate.copernicus.eu

CAMS:https://atmosphere.copernicus.eu

ClimateChange

C 3 S S e a s o n a l F o r e c a s t D a t a• SeasonalForecastdataisamajordatasetandservice forC3S/CDS

• 5institutions (ECMWF,MetOffice,MF,DWD,CMCC)willprovideSeasonalForecastdatatoC3S,whichwillbeusedtoproduceproducts

• Needtobuildaconsistent archiveforusers,andoperationally producemulti-systemproducts

Wehavethereforedeveloped:• A netCDF standard forC3SSeasonalForecastdataproviders

• UsesnetCDF4“classic”format• basedonCF-1.6,SPECS,CMIP…

https://software.ecmwf.int/wiki/display/C3SS/Guide+to+NetCDF+encoding+for+C3S+providers(ECMWFwebloginrequiredandmayneedtorequestaccess)

ClimateChange

C 3 S n e t C D F s t a n d a r d f o r S e a s o n a l D a t a

Defines:• Exactstructureoffiles• Standardnames• SpatialCoordinates• Multipletimecoordinates• Realization• Cellmethods• Attributes• Metadata

V0.1ReleasedJune2017

Expectoperational netCDF datainthisformbySeptember2017

ClimateChange

C 3 S S e a s o n a l F o r e c a s t D a t a

FortheSeasonalForecastdatawehavealso developed:

Afilename/directory nameconvention• BasedonCMIPapproach.

https://software.ecmwf.int/wiki/display/C3SS/NetCDF+Dataset+Design+Overview(ECMWFwebloginrequiredandmayneedtorequestaccess)

AnetCDF filechecker• ChecksconformancetoCF-1.6andC3S-0.1;• Commandlinetool;• Pythoncode,highlyconfigurableusingJSONconfig filesformetadataand

dataconstraints.https://software.ecmwf.int/stash/projects/CDS/repos/checker/browse

ClimateChange

n e t C D F C h e c k e r

Provides:• Filesummary• ResultsofCF-1.6checks• ResultsofUser/projectdefinedchecksCheckerreleasedJuly2017

ClimateChange

T h e C D S T o o l b o x• Thetoolboxwillmakeuseofall thedatasetsavailableintheCDS;

• Thewidevarietyofvolumes,datatypes,formatsandstructuresmakestheircombinedusehighlychallenging;

• Willallowtheuserstodevelopwebbasedprocessing(applications);

• Availableoperationsinc. differences,regridding,statisticalcomputations(tools);

• Combinetoolsandpresentresultstousers(workflows);

ToachievethisaCommonDataModelisalsobeingdeveloped:• BuiltonnetCDF recommendationsdocumentwehaveproduced(referredto

earlier)

ClimateChange

T o o l b o x C D M

• BasedontheECMWF“Recommendations”document

• BasisforToolboxdatahandling

ExpectedRelease~endof2017

ClimateChange Wewelcomeanyfeedbackyouhave…

Thankyou

https://climate.copernicus.eu https://www.ecmwf.int