the cf conventions: options for sustained support involving unidata russ rew unidata policy...
TRANSCRIPT
The CF Conventions: Options for Sustained Support
Involving Unidata
Russ Rew
Unidata Policy Committee
May 12, 2008
What are the CF Conventions?• An international community standard for encoding Climate and weather
Forecast metadata in netCDF files: cfconventions.org• Metadata conventions supporting interoperability for earth science data
from different sources• An information model for representing geoscience metadata in terms of
netCDF attributes and coordinate variables• Intended for both model output and observational datasets• Examples of CF metadata
– Coordinate information needed to locate data in space and time– Standard names for quantities – to determine whether data from
different sources are comparable– Additional grid information (e.g., grid cell bounds, cell averaging
methods)
An abstraction-level analogy
ConventionsConventionsCF
conventions Normalization
rules
InformationInformationModelsModels
RelationalRelational netCDF-CFnetCDF-CF
ExchangeExchange FormatsFormats
netCDFXML
Where is CF metadata used?
• Growing use, acceptance, and dependence on CF– Climate: Coupled Model Intercomparison Project phase 3 (CMIP3) multi-
model dataset for IPCC, Paleoclimate Modeling Intercomparison Project (PMIP), Hemispheric Transport of Air Pollution (HTAP), EU-funded ENSEMBLES prediction system for climate change
– Ocean: IOOS DMAC, GHRSST
– GIS: ESRI ArcGIS, OGC GALEON, WCS
• Various archives at ESMF, GFDL, Hadley Centre, NCAR, NOAA, …
• Software for analyzing, visualizing, subsetting, regridding, and aggregating data
Guiding principles of CF
Data should be self-describing, without external tables needed for interpretation.
Conventions should only be developed as needed.
Conventions should not be onerous for either data-writers or data-readers.
Metadata should be readable by humans as well as interpretable by programs.
Redundancy should be minimized to avoid inconsistencies when writing data.
A brief history of CF
• Evolved from simple netCDF User Guide conventions (1989), COARDS standard (1995), Gregory-Drach-Tett (1999), and NCAR CSM (1999) conventions
• 2000-2003: Developed by volunteer efforts (Brian Eaton, Jonathan Gregory, Bob Drach, Karl Taylor, and Steve Hankin)
• 2003: CF 1.0 released• 2005: CF white paper discussing future governance circulated• 2006: Revised white paper presented to WCRP WGCM• 2007: Rules for community-initiated changes to CF
conventions agreed upon• 2008: CF 1.2 released, active community involvement in
proposals for additions
Some strengths of CF
• Successful international collaboration to codify best practices into a community standard
• Proven record of enhancing interoperability• Engagement of diverse communities to capture expertise for
standard names• Agreement on open process for evolving conventions and
reaching consensus
• Continued active interest and involvement from user community
• Commitment of some organizational resources– British Atmospheric Data Centre: Standard names – (50% FTE)
– Lawrence Livermore: Web site and discussion forum support – (20% FTE)
– Unidata: Library development (libcf) – (10% FTE)
• CF issues aired at Global Organization for Earth System Science Portals annual meetings
Future directions for CF
• Implementing CF metadata conventions for other file formats (besides netCDF)
• Supplying both data providers and application developers with library support for using CF
• Providing improvements for representing observational data and metadata
• Supporting more types of grids (staggered, curvilinear, nested)
• Supporting mappings between CF and other metadata standards and conventions
• Use of netCDF-4 data model and format
CF still needs …
• A CF Coordinator (scientist, data specialist) to support CF development– Understanding, reviewing, summarizing,
shepherding, moderating discussion for, coordinating, and improving proposals
– Promoting CF use and compliance in Unidata software
• Committed institutional resources for– Support of libcf– Tests of proposed extensions– Data examples – Stewardship
Proposal for CF support at Unidata
• CF governing and conventions committees seeking – 0.5 FTE CF coordinator with broad scientific knowledge, data
expertise, and passion– Trusted institutional support for hosting CF coordinator
(Unidata)– Shared funding from CF beneficiaries (BADC, NOAA/NCEP,
NOAA/NCDC, NOAA/OAR, DoE, Navy, NASA, GHRSST, NCAR, ...)?
• Propose exploring a new partnership between CF community and Unidata
• CF conventions governance teleconference scheduled on May 16, 2008, to discuss strategies for getting resources to advance CF development, including seeking funding for 0.5 FTE at Unidata
Some Policy Committee options
• Endorse a greater emphasis on both the CF conventions layer in Unidata software and more focus on helping to advance CF, for the benefits it brings to the current Unidata community and to interoperability of tools and data in other communities.
• Endorse specific arrangement of an additional CF coordinator and data specialist at Unidata Program Center, supported by non-NSF funds.
• Suggest other possible sources of funding support for CF infrastructure.
• … ?
Concluding comments
• CF has undergone a two-year transition from informal maintenance by its authors to community governance.
• The CF Standard Names transition, with 0.5 FTE dedicated at BADC, has been very successful.
• The CF Conventions transition has been moderately successful, but needs more resources.
• Wide usage and real-world experience suggests CF metadata conventions are highly suitable for a broad community of data providers and users.
• To guarantee maintenance and ensure persistence as an internet resource, CF will need both– Continued community participation– Resources and trusted organizations to provide stewardship