THREDDS Catalogs

Ethan Davis

UCAR/Unidata

NASA ESDSWG Standards Process Group meeting, 17 July 2007

THREDDS Catalogs

• XML documents over HTTP

• Named datasets with access URLs (typed for various protocols)

• Datasets can be hierarchically organized (good for browsing)

• Metadata at collection (inheritable) or atomic dataset level.

THREDDS Catalogs

Catalog Services

• Catalog Validation

• Catalog Subsetting

• HTML generation

• Automatic/dynamic catalog generation and dataset metadata harvesting

• Direct metadata harvesting

• OAI-PMH harvesting (DIF, ADN)

Implementations – Servers

• THREDDS Data Server (TDS)

• OPeNDAP Hyrax server

• GrADS Data Server (GDS)

• IRI/LDEO server (Ingrid)

• NCAR Community Data Portal (CDP)

Implementations - Clients

• IDV

• MyWorld GIS

• netCDF-Java ToolsUI

• NCAR Community Data Portal (CDP)

• THREDDS Data Server (TDS)

Installations

• NCAR/UCAR• NOAA – NCDC, NGDC, PMEL, …• COLA• FNMOC• BOM (Aust)• GMU• IRI/LDEO• …

Leader For SPGEndorsement Process

• ???

• User community member?

CF (Climate and Forecast) Metadata Convention for netCDF

Ethan Davis

UCAR/Unidata

NASA ESDSWG Standards Process Group meeting, 17 July 2007

CF Design Goals“The Climate and Forecast (CF) metadata conventions are designed to promote the processing and sharing of data stored in files created with the netCDF API. Fundamental features of the conventions are that CF-aware software can automatically determine the space-time location of variables (facilitating analysis and graphical display), and metadata describing each variable is sufficiently detailed to determine whether variables from different sources are comparable.”

-- Lawrence, B.N., R. Drach, et al, “Maintaining and Advancing the CF Standard for Earth System Science Community Data”. http://cf-pcmdi.llnl.gov/documents/white-papers/cf2_whitepaper_final.pdf

CF Guiding Principles

1. Data should be self-describing. No external tables are needed to interpret the file. For instance, CF encodings do not depend upon numeric codes (by contrast with GRIB).

2. Conventions have been developed only for known issues. Instead of trying to foresee the future, features are added as required.

3. The convention should be easy to use, both for data-writers and users of data.

4. The metadata and the semantic meaning encoded through the metadata should be readable by humans as well as easily utilized by programs.

5. Redundancy should be minimised as much as possible (so as to reduce the chance of errors of inconsistency when writing data).

-- Gregory (2003), http://www.cgd.ucar.edu/cms/eaton/cf-metadata/clivar_article.pdf (broken link)

CF netCDF Metadata Conventions

• Determine if variables are comparable:– standard names

– units

• Locate in space and time:– Coordinate variables

– Projections

– etc

• Data types (gridded, ...)

CF, not just for netCDF anymore?

• CF specification defined for netCDF-3• But much of CF is format agnostic, e.g.,

CF-style metadata has been added to HDF data

• The netCDF-4 data model has significant changes from the netCDF-3 data model.

• Won’t be sure how netCDF-4 will affect CF until we have more experience with it

CF Community

• Initially dominated by climate and forecast modelers

• Community is broadening: oceanography, atmospheric chemistry, biology, ...

• Governance: Conventions Committee, Standard Name Committee

• http://www.cfconventions.org/

Implementations

• CF compliance checker• libcf (support read/write of CF compliant netCDF)• netCDF-Java (read, some support for write)• ArcGIS 9.2+ reads CF-netCDF files• CF-netCDF being developed as OGC WCS data

format profile • CF-netCDF as TDS netCDF Subset Service

response

Leader For SPGEndorsement Process

• User community member

• Governing committee member?