sjsk 1 hdf-eos workshop ii metadata requirements for eosdis data providers siri jodha singh khalsa...

SJSK 1HDF-EOS Workshop II

Metadata Requirements for EOSDIS Data Providers

Siri Jodha Singh Khalsa

[email protected]


Topics

•Why metadata is important

•Types of metadata in HDF-EOS files

•Required metadata

•How metadata is encoded and delivered


What is Metadata?

•Metadata is information that identifies and characterizes an information product.

•Sometimes called “data about data”


Users Need Metadata

•Metadata is needed to answer questions such as:

- What time and location does this data apply to?

- Why type of instrument and processing produced the data?

- What other inputs were used to generate the data?

- What QA has been performed on this data?

- Who do I contact if I have questions about this data?


Metadata is Essential

•Large data archive systems cannot function without metadata.

•Metadata is used to keep track of such things as:

- where the data is

- what type of operations are possible on the data

- whether there are any access restrictions on the data

- how individual data files are logically grouped into “collections.”

SJSK 6

Key Concepts

•A granule is the smallest aggregation of data that is independently described and inventoried by the ECS. A granule consists of 1 or more physical files.

•A collection is a logical grouping of granules.

•The ECS Data Model allows for:- “Core” attributes

- “Product-Specific” Attributes (PSAs)


Types of Metadata

•Metadata in HDF files- stored as global text attributes

•Types of Metadata used in HDF-EOS files:- Structural Metadata

- Core Metadata (inventory, can include PSAs)

- Archive Metadata (non-searchable, product-specific)

•Collection level metadata- core and product-specific


Required Metadata

•Origins of metadata requirements:- what is required to archive and retrieve files

- what is required to provide search and other services on data

- what is federally mandated (FGDC)

•There are 287 attributes in the ECS data model- only a subset are used for any given product

- 101 are applicable at the granule level


Metadata Coverage

•Science Data that are delivered for archiving in ECS must meet what is called the Intermediate level of metadata coverage. This involves as few as:

- 31 collection level attributes

- 4 granule level attributes

•Compliance at this level is not enforced by the system.


Collection-Level Metadata for Intermediate Coverage

- ShortName- LongName - CollectionDescription - VersionID- ArchiveCenter- RevisionDate - VersionDescription- CollectionState- MaintenanceandUpdateFrequency- ECSDisciplineKeyword- ECSTopicKeyword- ECSTermKeyword- ECSVariableKeyword- ContactOrganizationName- Role

- SpatialCoverageType- PointLatitude- PointLongitude- TimeType- DateType- TemporalRangeType- PrecisionofSeconds- EndsatPresentFlag- CalendarDate- TimeofDay- GuideName- DataCenter- DocumentVersion- DocumentUpdated- Title- DocumentCreated


Granule-Level Metadata for Intermediate Coverage

•There are only four granule-level metadata attributes required:

- ShortName- VersionID- SizeMBECSDataGranule- ProductionDateTime

•ShortName and VersionID are identical to the collection-level attributes with these names.

•For granules coming into ECS, SizeMBECSDataGranule and ProductionDateTime are supplied by the system upon insertion.


How is Metadata Supplied?

•Collection-level metadata is carried in an Earth Science Data Type (ESDT) Descriptor file.

•Granule-level metadata is defined in the descriptor file and populated using a Metadata Configuration File (MCF).

•Granule-level metadata is delivered in the HDF-EOS granule *or* in a populated MCF accompanying a non-HDF granule.

•The DAAC where a collection will reside is responsible for descriptors and ingest routines.


DataProvider

DAAC

Analysis

Specs

Populat ion

MDWorks

PSA_Reg

Descriptor

ODL Parser

Validat ion

MCF Build

MCF Validated Desc.

DLL coding

Data/Docs

Tools

Tasks

ScienceSoftware

SDP Toolkit

DAAC Data ArchiveHDF-EOS file

Data BaseLoad File

Test & Valid.

ESDTI nsert

Constraintschecks

ODLsyntaxcheck

granule core valuesPSA valuesstructural metadata

Data ModelMDWorks

collect ion core at tributes +valuesgranule coreattributesPSA def init ions

type and formatcheck

Responsibility

I ngestSubsystem

Metadata Work Flow for External Data Providers

SJSK 14

Metadata Resources on the Web

•ECS Metadata Homepagehttp://ecsinfo.hitc.com/metadata/metadata.html

•Metadata Works (ESDT Descriptor Tool)http://et3ws1.HITC.COM/metadata_works/

•EOSDIS Information Architecturehttp://spsosun.gsfc.nasa.gov/InfoArch.html

•Federal Geographic Data Committeehttp://www.fgdc.gov/

SJSK 15

Q&A w/ Experts Panel

•Q: “If you are a new data provider, how do you get your data into an HDF-EOS granule, given the bewildering array of utilities and tools available? What is the simplest solution for this?”

•A: The recommended solution is to obtain the HCR package, which includes the HDF-EOS and HDF libraries. For populating the required metadata in the granule, obtain the Metadata/Time Toolkit_MDT. The steps would be:

1. Write an HCR and use the tools to turn this into a skeletal HDF-EOS granule. (This step is optional).

2. Use the HDF-EOS library to create a granule. (If starting with a skeletal HDF-EOS file generated from an HCR then plain HDF calls can be used to insert data into the granule ).

3. Use Toolkit_MDT calls to insert metadata into the granule. This requires generation of an MCF in ODL. Metadata_Works is available for doing this. As an alternative, a simple HDF call can be used to attach minimum metadata (in ODL) to an HDF file.

Note: if the data are going to reside in a DAAC, or in an archive that must be interoperable with ECS, you will need to generate collection-level metadata. Metadata_Works is the recommended tool for this.

sjsk 1 hdf-eos workshop ii metadata requirements for eosdis data providers siri jodha singh khalsa...

Documents