Meteorological Assimilation Data Ingest System (MADIS) and ISO Data QualityTed HabermannNOAA National Data Centers MADIS observations on April 29, 2004

MADIS and ISO MetadataMADIS produces and distributes thousands of quality reports for stations in U.S. meteorological mesonets every week. See http://madis.noaa.gov/ for MADIS information.

The ISO 19115 Metadata Standard includes an extensive section devoted to data quality reports.

How can these fit together?

MADIS mesonet stations in the North-American domain on

May 26, 2004

Generating ISO Metadata

RichInventory

MADIS TrackingNMMR

Existing Quality Resources

Existing Metadata Resources

ISO Generator

ISOMetadata

RESTViews

This schematic shows our approach. Existing quality resources are queried using a web service (REST) for the “beef” of the quality report. The “bun” is provided from views into existing FGDC or new ISO sources. These two are combined into a complete ISO report.

The “Beef”

The “Bun”

Data Quality Granularity

MADIS integrates observations from stations in many meteorological mesonets around the U.S.

The MADIS Quality Repository includes:nine statistics for six parameters For thousands of stations from~150 data networks every week.

Quality reports can be generated at any level (or granularity) from this hierarchy.

ISO Quality GranularityDQ_Scope+ level: MD_ScopeCode+ extent [0..1] : EX_Extent+ levelDescription [0..*] : MD_ScopeDescription

MD_ScopeDescription+ attributes : Set<GF_AttributeType>+ features : Set<GF_FeatureType>+ featureInstances : Set<GF_FeatureType>+ attributeInstances : Set<GF_AttributeType>+ dataset : CharacterString+ other : CharacterString

+ nonGeographicDataset+ dimensionGroup+ feature+ featureType+ propertyType+ fieldSession

MD_ScopeCode+ attribute+ attributeType+ collectionHardware+ collectionSession+ dataset+ series

+ software+ service+ model+ tile

ISO Quality Granularity

In the geographic jargonan ISO feature is a station andan ISO attribute is a parameter

MD_ScopeDescription+ attributes : Set<GF_AttributeType>+ features : Set<GF_FeatureType>+ featureInstances : Set<GF_FeatureType>+ attributeInstances : Set<GF_AttributeType>+ dataset : CharacterString+ other : CharacterString

MD_ScopeCode = dataset (data network in this case)

We considered several options for expressing report granularity in ISO and ended up with a general approach that seems to make sense (at least as a first draft):

= list ofparameters

= list of stations

Select Report

Select threeweeks

Select threestations

Select twoparameters

http://ngdc.noaa.gov/idb/struts/form?t=102711&s=14&d=35,36,38

The HTML View

Three weeks

Two Parameters

Three Stations

The Quality Information

The “Beef”

Three weeks

Two Parameters

Three Stations

The Quality Information

The ISO Report

<scope><DQ_Scope>

<level><MD_ScopeCode>dataset</MD_ScopeCode>

</level><levelDescription>

<MD_ScopeDescription><dataset><gco:CharacterString>IODOT</gco:CharacterString></dataset>

</MD_ScopeDescription>/levelDescription><levelDescription>

<MD_ScopeDescription><attributeInstances><gco:CharacterString>SLP,THETA</

gco:CharacterString></attributeInstances></MD_ScopeDescription>

</levelDescription><levelDescription>

<MD_ScopeDescription><featureInstances><gco:CharacterString>RAMI4,RAVI4,RBUI4</

gco:CharacterString></featureInstances></MD_ScopeDescription>

</levelDescription></DQ_Scope>

</scope>

ISO Report Scope

Two Parameters

Three Stations

One Network

ISO Temporal Extent

Week One

Week Two

The ISO Quantitative Report<DQ_QuantitativeResult>

<value><gco:Record>

"Start Date","End Date","Data Network","Station","Parameter","Total Count","Bad Count","Accum_Sum","Accum_Sum2","Percent Bad","Mean","RMS“

"2007-01-22", "2007-01-28","IODOT","RAMI4","THETA",294,0,0,0,0,0,0"2007-01-22", "2007-01-28","IODOT","RAMI4","SLP",0,0,0,0,0,0,0"2007-01-22", "2007-01-28","IODOT","RAVI4","SLP",0,0,0,0,0,0,0"2007-01-22", "2007-01-28","IODOT","RAVI4","THETA",282,5,36.96,276.1,1.77,7.39,7.43"2007-01-22", "2007-01-28","IODOT","RBUI4","THETA",308,0,0,0,0,0,0"2007-01-22", "2007-01-28","IODOT","RBUI4","SLP",0,0,0,0,0,0,0"2007-01-29", "2007-02-04","IODOT","RAMI4","THETA",314,0,0,0,0,0,0"2007-01-29", "2007-02-04","IODOT","RAMI4","SLP",0,0,0,0,0,0,0"2007-01-29", "2007-02-04","IODOT","RAVI4","THETA",291,0,0,0,0,0,0"2007-01-29", "2007-02-04","IODOT","RAVI4","SLP",0,0,0,0,0,0,0"2007-01-29", "2007-02-04","IODOT","RBUI4","THETA",316,0,0,0,0,0,0"2007-01-29", "2007-02-04","IODOT","RBUI4","SLP",0,0,0,0,0,0,0"2007-02-05", "2007-02-11","IODOT","RAMI4","THETA",671,0,0,0,0,0,0"2007-02-05", "2007-02-11","IODOT","RAMI4","SLP",0,0,0,0,0,0,0"2007-02-05", "2007-02-11","IODOT","RAVI4","THETA",633,0,0,0,0,0,0"2007-02-05", "2007-02-11","IODOT","RAVI4","SLP",0,0,0,0,0,0,0"2007-02-05", "2007-02-11","IODOT","RBUI4","THETA",622,0,0,0,0,0,0"2007-02-05", "2007-02-11","IODOT","RBUI4","SLP",0,0,0,0,0,0,0

</gco:Record></value>

</DQ_QuantitativeResult>

The Quality Information

Conclusions• Yes, MADIS Quality Information can be reported as ISO 19115

• The ScopeCode is general with details in the LevelDescription

• We need to learn about describing feature and attribute sets in Feature Catalogs. At present we are using comma separated lists.

• We need to learn about how to describe the formats of reports. At present we are using CSV (comma separated values)

Ted.Habermann@NOAA.GOV