hnilo rdap11 data archives in federal agencies
DESCRIPTION
National Climate Model Portal, Jay Hnilo, NOAA NOMADS; Data Archives in Federal Agencies; RDAP11 Summit The 2nd Research Data Access and Preservation (RDAP) Summit An ASIS&T Summit March 31-April 1, 2011 Denver, CO In cooperation with the Coalition for Networked Information http://asist.org/Conferences/RDAP11/index.htmlTRANSCRIPT
The National Climate Model Portal
Overview & NCDC Archive and Access Challenges
Dr. Jay Hnilo NCMP Senior Scientist
NOAA’s Cooperative Institute for Climate and Satellites (CICS-NC)National Climatic Data Center (NCDC) Asheville, NC 28801
ASIS&T Research Data Access and Preservation Summit ASIS&T Research Data Access and Preservation Summit Denver CO. March 31, 2011Denver CO. March 31, 2011
M. Sutton 1995
The National Oceanic and Atmospheric Administration
2
NOAA National Climate Model Portal
NCMP
Background: NOMADS - A Data Access System
NCMP and NOMADS - Goals and Motivation
NCDC Archive Processes - Archive Processes
- Distributed Access Philosophy
OutlineOutline
3
NOAA National Climate Model Portal
NCMP
Until 2002 there existed no long-term archive for Climate and Weather models in NOAA.
Retrospective analysis and model inter-comparison are necessary to verify and improve short term NWP models, seasonal forecasts, climate simulations, assessments and detection.
University and Institutional research goes largely untapped by NOAA scientists. Effort is wasted on data receipt and format issues with no infrastructure to collaborate.
BackgroundNOMADS Data Access System
BackgroundNOMADS Data Access System
4
NOAA National Climate Model Portal
NCMP
• In 2002 to overcome a deficiency in model data access, some of the Nations top scientists actively engaged in a grass-roots framework to share data and research findings over the Internet.
• NCDC, NCEP and GFDL initiated the NOAA Operational Model Archive and Distribution System.
• NOMADS is a distributed data service providing format independent access to climate and weather models and associated data.
BackgroundNOMADS Data Access System
BackgroundNOMADS Data Access System
5
NOAA National Climate Model Portal
NCMP
• foster research within the geo-sciencecommunities (ocean, weather, and climate)to study multiple earth systems using collections of distributed data,
• promote model evaluation and product development
• develop institutional partnerships via distributed open technologies.
• provide distributed access to models and associated data. Begin to scale to petabyte.
BackgroundProject GoalsProject Goals
BackgroundProject GoalsProject Goals
6
NOAA National Climate Model Portal
NCMP
Pare down large file sizes of high resolution data and products- and provide flexible inter-operable access.
(re-) Group different data sets to create needed products – such as initialization files for model development, analysis, or by forecast projection.
Subset and aggregate the data: - in parameter space
- in physical space - in temporal space
BackgroundMotivation: Tools for Users
BackgroundMotivation: Tools for Users
7
NOAA National Climate Model Portal
NCMP
NCDC Archive Motivation for Archive Stewardship
NCDC Archive Motivation for Archive Stewardship
8
NOAA National Climate Model Portal
NCMP
NOAA-wide procedure to identify, appraise, and decide what scientific records are preserved in a NOAA Facility. Then a Submission Agreement (SA) outlines details of dataset.
Reviewed by a cross-NOAA working group- the Environmental Data Management Committee (EDMC). Long-term stewardship the goal.
Criteria developed using guidelines from National Archives and Records Administration (NARA), and National Research Council (NRC) reports on NOAA data management, and from other related reviews.
Technology to provide access and value-added products to deep archive an on-going NCDC activity (NOMADS-NCMP etc.).
NCDC Archive Archive Procedures
NCDC Archive Archive Procedures
9
NOAA National Climate Model Portal
NCMP
The Submission Agreement and “What to Archive Process” allows data center to make informed planning decisions
Provides a formal way to be selective about how data are supported
Documents the justification for allocating archive support for the data
Data Reduction policies and recommendations now underway with NOMADS (e.g., remove fcsts > 5 years).
NCDC Archive Benefits of Archive Procedures
NCDC Archive Benefits of Archive Procedures
NOAA National Climate Model Portal
NCMP
Private orpublished
results
Searchtools
Metadata
Ontologies
Public and private
catalogue
Workflowgeneration
tools
Private virtual
workspaces
Sharedvirtual
workspaces
Monitoring & controlservices
Workfloworchestration
engine
Observing System Simulation Experiments
Other (e.g.Unique
Instrumentation)
ModelingSystems
AnalysesDatasets
Eventdetection
D D D
M
Earth Systems Modeling Framework
National Climate Model Portal
NCMP
D data
Q Q/A
M model
A analysis
Observing Systems
Real-time data streams
Mid
dle
wa
re, a
ccess p
roto
cols, se
cure
da
ta tra
nsp
ort
Use
r au
the
ntica
tion
, acce
ss con
trol lo
gic
Me
tad
ata
voca
bu
larie
s, on
tolo
gy sta
nd
ard
s
Users
NOAA
Climate
Services
Portal
Ed
uca
tion
an
d tra
inin
g, u
ser su
pp
ort
Compute serversNCDC ArchiveDigital
libraries
Task
Reanalysis and
Climate Clearing-
houseCommunity
vetted observational
database
GEOUS-GEO
Adaptive
DOE Earth SystemGrid &iRODS
Q A
Pre / Post Processing
Rutledge/Meacham/Fontaine 2006
U.S. GEO Modeling Infrastructure VisionU.S. GEO Modeling Infrastructure Vision
Access
Data Ingest
NCDC Archive
National Climate Model Portal
Web Based Data Services
Community
• Priority Technologies & Partners - GO-ESSP Community - NCSP National Climate Service Portal
- NCPP National Climate Prediction and
Projections Center (ESRL prototype)- OPeNDAP, OGC, CF, NetCDF, TDS…- iRODS Renaissance Computing Institute- ESGF Earth System Grid Federation- IPCC & LLNL/PCMDI Archive
The NOMADS-NCMP System
12
NOAA National Climate Model Portal
NCMP
British Atmospheric Data Centre◦ Bryan Lawrence – Director, British Atmospheric Data Centre
Geophysical Fluid Dynamics Laboratory◦ V. Balaji, Head, Modeling Group, Princeton/GFDL
The German Climate Computing Centre◦ Michael Lautenschlager (NeRC Grid)
Lawrence Livermore National Laboratory◦ Dean Williams, PCMDI, Chief Archive Services/CMIP5 , ESGF
National Center for Atmospheric Research◦ Don Middleton, Senior Manager, Enabling Technologies, ESGF
Pacific Marine Environmental Laboratory◦ Steve Hankin (Unified Access Framework, DMIT)
NOAA/Earth Systems Research Laboratory◦ Cecelia Deluca (National Climate Projection and Prediction NCPP prototype)
NOAA/National Climatic Data Center◦ Glenn Rutledge, (Program Manager NOMADS/NCMP)
Global Organization for Earth Systems Science Portals
Related Workshop: 2011 GO-ESSP Workshop
Related Workshop: 2011 GO-ESSP Workshop
NCDC hosts the2011 GO-ESSP WorkshopMay 9-10, Asheville NC
http://go-essp.gfdl.noaa.gov/
NOAA National Climate Model Portal
NCMP
Questions?
[email protected]@noaa.gov
NCDC Asheville, NC
http://nomads.ncdc.noaa.gov
M. Sutton 1995
NCMP
NOAA National Climate Model Portal
Thank you