hdf-eos aura file format guidelines

HDF-EOS Aura File Format Guidelines

Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 1


Cheryl Craig

September 20, 2000



Preliminary• Aura Data System Working Group will

meet in October to discuss and hopefully adopt

• Authors from each instrument team are in agreement



What Is Aura?

• Third in series of EOS atmospheric chemistry satellites (1:45 PM equator crossing time)

• Formerly known as CHEM• To be launched June 2003• Level 2 data products include:

– Vertical profiles of temperature, pressure, chemical and aerosol species

– Column amounts of chemical species– Cloud properties



Aura Instruments• HIRDLS (High Resolution Dynamics Limb Sounder)

– Limb infrared sounder– University of CO and Oxford

• MLS (Microwave Limb Sounder)– Limb microwave sounder– JPL

• OMI (Ozone Monitoring Instrument)– Nadir wide-field-imaging spectrometer– Netherlands, Finland and US

• TES (Tropospheric Emission Spectrometer)– Nadir and limb infrared-imaging spectrometer– JPL

All instruments have world-wide co-investigators



Why Guidelines for Aura Instruments?

• HDF-EOS Swath allows a lot of flexibility

• Ease cross-platform use of Aura files– Ease development of software– Easier to understand

• Intended to standardize Level 2 files (may apply to Level 3 and above as well)



Areas of Potential Mismatch

• Organization of data fields and attributes• Dimension names• Geolocation names and dimension ordering• Data field names and dimension ordering• Units for data fields• Attribute names, values and units



Guidelines List

• Instrument teams will use the same major version of HDF-EOS

• Aura Level 2 files will use HDF-EOS Swath• HDF-EOS structure names will strictly adhere

to Valids list• Data will be reported on a pressure grid and be

ordered from ground to space



Guidelines List (continued)

• Data fields will be ordered with pressure coordinate being the fastest incrementing coordinate

• Data fields will be in units specified• HDF fill value and MissingValue will have the

same value• Instrument data fields unique to particular

instrument will be prefixed with instrument name



Swathname: Meas. Species Name

Dimensions:nTimesnLevels etc.

Geolocation Fields:TimeLatitude etc.

Data Fields:L2gpValueL2gpPrecision

Global Attributes:Instrument NameProcess Level etc.

Swathname: Instrument Name

Dimensions:nTimesnLevels etc.

Geolocation Fields:TimeLatitude etc.

Data Fields:TemperatureO3 etc.

Global Attributes:Instrument NameProcess Level etc.

Organization of Data Fields and AttributesType 1 Type 2

HDF-EOS V3.x Note – Two types may be merged into one using links for Data Field Names



Standard Names

Adhere to strict names (spacing and capitalization required)

• Dimension names

• Geolocation Fields – Used to hold descriptor data (includes SolarZenithAngle, OrbitAscendingFlag, etc.)

• Data Fields – Contains measured and derived data

• Global Attributes – Includes InstrumentName and ProcessLevel

• Data Field Attributes – only if use HDF-EOSV3 (includes MissingValue, ScaleFactor, Units, etc.)



HDF-EOS V2.x Versus V3.x

HDF-EOS V2.x

PROS

• Currently used by Terra and Aqua teams

• Currently available

CONS

• Based on HD4 – supported by HDF group in maintenance only mode

HDF-EOS V3.xPROS• Contains new functionality

– local data attributes – more than one unlimited

dimension– compression with unlimited

dimensions

• Based on HDF5 CONS• IDL support (unknown?)

Aura will decide at future date. At January, 2000 DSWG, indicated desire to used V3.x if packages like IDL will be available in V3.x



Conclusions

• Preliminary standardization complete – meeting in October to discuss and adopt

• HDF-EOS version decision to be decided at a future date

• Some decisions postponed until prototyping efforts complete

• Standard may evolve as instrument teams get better understanding of their data storage needs

• URL of document:http://www.eos.ucar.edu/hirdls/HDFEOS_Aura_File_Format_Guidelines.pdf



Valids from Document

Swath Name: Instrument Name - or - Measured Field Name (see Data Field Names for list of names) Dimensions: These are the actual dimensions of the Geolocation and Data field quantities. nTimes Number of times (profiles) in data set (this may be unlimited) nLevels Number of pressure levels nWavel Number of wavelengths nXtrack Number of pixels in the across track direction (OMI specific)



Geolocation Fields: This is data which describe the scientific measured quantities. They provide information to aid in describing the data’s “location”.

Instrument HDF-EOS Name Dimension Suggested units H M O T Notes Time (nTimes) seconds from Jan 1, 1993 X X X X time in TAI units Latitude (nTimes) degrees X X X Geodetic Latitude Latitude (nXtrack,nTimes) degrees X Geodetic Latitude Longitude (nTimes) degrees (-180 to 180) X X X Longitude (nXtrack,nTimes) degrees (-180 to 180) X Pressure+ (nLevels) hPa (ordered from ground to

space) X X X Pressure will be a superset of

the UARS pressure levels Altitude (nLevels,nTimes) meters X X SecondsInDay (nTimes) seconds X Seconds from midnight of

day listed in global attributes SolarZenithAngle (nTimes) degrees X X X X LocalSolarTime (nTimes) hours X X X SpacecraftLatitude (nTimes) degrees X X A SpacecraftLongitude (nTimes) degrees (-180 to 180) X X A SpacecraftAltitude (nTimes) meters X X A OrbitAscendingFlag (nTimes) logical X A if true then orbit is ascending OrbitGeodeticAngle (nTimes) degrees X LineOfSightAngle (nTimes) degrees (East of North) X SolarAzimuthAngle (nXtrack,nTimes) degrees (East of North) X A ViewingZenithAngle (nXtrack,nTimes) degrees X ViewingAzimuthAngle (nXtrack,nTimes) degrees (East of North) X SpacecraftAzimuth (nTimes) degrees (East of North) A SpacecraftZenith (nTimes) degrees A HIRDLSScienceScanMode (nTimes) short integer X HIRDLS Science Scan Mode

identifier HIRDLSScanUpFlag (nTimes) logical X HIRDLS Scan Up identifier



Geolocation Fields (continued):

Instrument HDF-EOS Name Dimension Suggested units H M O T Notes HIRDLSScanElevationAtNominalAltitude (nTimes) degrees X HIRDLSScanAzimuthAtNominal Altitude (nTimes) degrees X HIRDLSTangentHeightAtNominalAltitude (nTimes) meters X HIRDLSViewDirectionAtNominalAltitude (nTimes) degrees (East of North) X TESDate (nTimes) A TES survey may cover

multiple days TESSequenceID (nTimes) sequence within current survey X TESBoresightAngle (nTimes) degrees A TESFiltersUsed (nTimes) bit mask A TESPixelsUsedFlag (nTimes) bit mask A

X – Field in standard file A – Data items which will be carried in an ancillary file that is created only once per set of standard products + nLevels for TES is currently set to 14 levels (based on UARS standard) but may be changed to 85 levels. nLevels for HIRDLS is set to 145 (1000. * 10 ** (-i/24) i=0,144)



Data Field Names: This is the actual scientific data. IMPORTANT NOTE – Array ordering is Fortran ordering, the first dimension is the most rapidly incrementing. For C, reverse the array indices. HDF-EOS Name Dimension Suggested units H M O T Notes L2gpValue (nLevels,nTimes) varies X MLS Generic Data Field Name (species

identified by SwathName) L2gpPrecision (nLevels,nTimes) varies X MLS Generic Precision Name (species

identified by SwathName) Temperature (nLevels,nTimes) K X S X O3 (nLevels,nTimes) parts per volume X S X O3 (nLevels,nXtrack,nT

imes) parts per volume X OMI imager data

H2O (nLevels,nTimes) parts per volume X S X ClONO2 (nLevels,nTimes) parts per volume X Sp N2O5 (nLevels,nTimes) parts per volume X Sp N2O nLevels,nTimes) parts per volume X S Sp TES special product, tropospheric control NO2 (nLevels,nTimes) parts per volume X X TES limb only CH4 (nLevels,nTimes) parts per volume X X HNO3 (nLevels,nTimes) parts per volume X S X TES limb only CFC11 (nLevels,nTimes) parts per volume X CFC12 (nLevels,nTimes) parts per volume X OH (nLevels,nTimes) parts per volume S HO2 (nLevels,nTimes) parts per volume S CO (nLevels,nTimes) parts per volume S X HCN (nLevels,nTimes) parts per volume S Sp NO (nLevels,nTimes) parts per volume X TES limb only HCl (nLevels,nTimes) parts per volume S Sp



Data Field Names (continued): HDF-EOS Name Dimension Suggested units H M O T Notes HOCl (nLevels,nTimes) parts per volume S ClO (nLevels,nTimes) parts per volume S BrO (nLevels,nTimes) parts per volume S SO2 (nLevels,nTimes) parts per volume Sp H2O2 (nLevels,nTimes) parts per volume Sp HDO (nLevels,nTimes) parts per volume Sp C2H6 (nLevels,nTimes) parts per volume Sp C2H2 (nLevels,nTimes) parts per volume Sp HCOOH (nLevels,nTimes) parts per volume Sp CH3OH (nLevels,nTimes) parts per volume Sp PAN (nLevels,nTimes) parts per volume Sp CH3C(O)CH3 (nLevels,nTimes) parts per volume Sp C2H4 (nLevels,nTimes) parts per volume Sp H02N02 (nLevels,nTimes) parts per volume Sp NH3 (nLevels,nTimes) parts per volume Sp CCl4 (nLevels,nTimes) parts per volume Sp CCl3F (nLevels,nTimes) parts per volume Sp CCl2F2 (nLevels,nTimes) parts per volume Sp CHCl2F (nLevels,nTimes) parts per volume Sp CHClF2 (nLevels,nTimes) parts per volume Sp COS (nLevels,nTimes) parts per volume Sp H2S (nLevels,nTimes) parts per volume Sp TES special product, volcanic plumes SF6 (nLevels,nTimes) parts per volume Sp XX.XXMicronAerosolExtinction (nLevels,nTimes) extinction (1/km) X Aerosol at XX.XX yyyPrecision (nLevels,nTimes) X Random Errors for species yyy yyyNormChiSq (nTimes) X Normalized Chi2 for species yyy



Data Field Names (continued): HDF-EOS Name Dimension Suggested units H M O T Notes CirrusIceContent (nLevels,nTimes) g/m3 S CloudTopHeight (nTimes) meters X CloudTopPressure (nXtrack,nTimes) hPa X (OMI imager data) CloudTopPressure (nTimes) hPa X X ColumnAmountO3 (nXtrack,nTimes) Dobson units S X ColumnAmountNO2 (nXtrack,nTimes) molec/cm2 X ColumnAmountSO2 (nXtrack,nTimes) molec/cm2 X ColumnAmountBrO (nXtrack,nTimes) molec/cm2 X ColumnAmountHCHO (nXtrack,nTimes) molec/cm2 X ColumnAmountOCLO (nXtrack,nTimes) molec/cm2 X CloudFraction (nXtrack,nTimes) dimensionless X AerosolOpticalThickness (nWavel,

nXtrack,nTimes) dimensionless X

SurfaceTemperature (nTimes) K X (May be cloud top for limb or may be omitted for limb)

RandomError (nLevels,nTimes) range: 0-1 X LandSurfaceEmissivity (nTimes) A SystematicError (nLevels,nTimes) range: 0-1 X TotalColumnDensity (nTimes) moles/cm2 X TES may still split into Trop/Strat TotalColumnDensityError (nTimes) moles/cm2 X TES may still split into Trop/Strat EstRandomErrors (nLevels,nTimes) range: 0-1 X EstSystematicErrors (nLevels,nTimes) range: 0-1 X EstFractionalExplainedVariance (nLevels,nTimes) range: 0-1 X EstCorrelationLength (nLevels,nTimes) 0-1000 km X Convergence (nTimes) X (Chi2) final retrieval – not normalized CompConvergenceToEstimate (nTimes) A Comparison between actual Chi2 and

estimated Chi2 – not normalized



Data Field Names (continued): HDF-EOS Name Dimension Suggested units H M O T Notes ConvergenceFlag (nTimes) 0-1 (Boolean) A Final retrieval converged nIterations (nTimes) 0-8 small int. A Number of model iterations for final

retrieval nonConvergenceFlag (nTimes) 0-1 (Boolean) X Did not converge at least during an

intermediate step OutofExpectedRange (nTimes) X AverageRMSError (nTimes) X MaxRMSError (nTimes) X CloudCover (nTimes) 0-1 (Boolean) A Identified through retrieval or L1B ProcessingTime (nTimes) A X – Field in standard file S – Swathname (MLS will identify field with the swathname rather than the fieldname) A – Data items which will be carried in an ancillary file that is created only once per set of standard products Sp – TES special data product



Global Attributes: This is information which helps to describe this particular data set. Attribute Name Attribute Type Attribute Values InstrumentName character “HIRDLS”, “MLS”, “TES” or “OMI” ProcessLevel HDFE_NATIVE_SHORT 2, 3, etc. GranuleMonth HDFE_NATIVE_INT 1-12 GranuleDay HDFE_NATIVE_INT 1-31 GranuleYear HDFE_NATIVE_INT Version HDFE_NATIVE_CHAR VerticalCoordinate HDFE_NATIVE_CHAR “Pressure”, “Altitude”, “Potential Temperature” Data Field Attributes

HDF5 Note This is information which helps to describe the individual data fields. Data Field Attributes are a feature which can be useful in annotating plots as well as describing the data product to input routines. It is a feature which was cumbersome to implement and retrieve in HDF-EOS V2.x, but is a subroutine call in HDF-EOS V3.x. For this reason, it is proposed that we only use data field attributes if the Aura team adopts HDF-EOS V3.x as the version to use.

Data Field Attributes: Attribute Name Attribute type Attribute Description MissingValue Same type as Data Field Contains the value for missing data ScaleFactor HDFE_NATIVE_FLOAT Factor for scaling data (if applicable) Offset HDFE_NATIVE_FLOAT Value to add to the data (if applicable) Title HDFE_NATIVE_CHAR For labeling a plot or axis Units HDFE_NATIVE_CHAR Labeling units (for labeling color bars, converting between units, etc). Values for Units: "m", "km", "hPa", "Pa", "mAtm-cm", "ppbv", “K”, “micron”, “degree”, “second”, “hour”, “m-1”, “dimensionless”, etc.

hdf-eos aura file format guidelines

Documents