hdf-eos apis, tools, etc
DESCRIPTION
HDF-EOS APIs, tools, etc. Larry Klein, Abe Taaheri, and Cid Praderas L-3 Communications Government Services, Inc. November 30, 2005. What is HDF-EOS ?. HDF (Hierarchical Data Format) is the format chosen by NASA as the standard for the EOS program - PowerPoint PPT PresentationTRANSCRIPT
1
HDF-EOS APIs, tools, etc.
Larry Klein, Abe Taaheri, and Cid Praderas
L-3 Communications
Government Services, Inc.
November 30, 2005
2
What is HDF-EOS ?• HDF (Hierarchical Data Format) is the format
chosen by NASA as the standard for the EOS program
• HDF-EOS is an extension to HDF which standardizes storage of and access to common Earth Sciences data structures– By providing a single interface to data structures
common to earth science– By providing a container for EOS inventory,
archive and product specific metadata– By establishing a consistent relationship between
geolocation and science data.
3
HDF-EOS Grid Structure
• Usage - Data which is organizedby regular geographic spacing, specified by projection parameters.
• Structure– Any number of 2-D to 8-D data arrays per structure, one
per data type (e.g. temperature) – Geolocation information contained in projection formula,
coupled by structural metadata.– Any number of Grid structures per file allowed.
4
HDF-EOS Swath Structure
• Data which is organized by time, or other track parameter.
• Spacing can be irregular.• Structure
– Geolocation information stored explicitly in Geolocation Field (2-D array)
– Data stored in 2-D or 3-D arrays– Time stored in 1-D or 2-D array, TAI is standard. – Geolocation/science data connected by structural
metadata
5
Swath Data Example
6
HDF-EOS Point Structure
• Data is specified temporally and/or spatially, but with no particular organization
• Structure– Tables used to store science data at a particular
Lat/Long/Height
– Up to eight levels of data allowed. Structural
metadata specifies relationship between levels.
7
Zonal Average Structure
• Generalized array structure with no geolocation linkage
• Data can be organized by time or track parameter
• Data spacing can be irregular
• Structure– Data stored in multidimensional arrays– Time stored in 1-D or 2-D array
8
HDF-EOS 2• HDF4 - based, storage format for EOS standard
products. • Used operationally by MODIS, MISR, ASTER,
Landsat, AIRS and other EOS instruments• Support for Grid/Point/Swath structures
• HDF4 and HDF-EOS 2 will be supported by NCSA/NASA for the fore-seeable future.
9
HDF-EOS 5
• Based on HDF5, a complete rewrite of HDF4 with a different user interface. – First released in 2000. – Format used by EOS Aura instruments
• Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. – Supports same data structures– Added prefix ‘HE5_’ to HDF-EOS 2 functions. – Doesn’t preclude HDF5 functionality.
10
Top Level of HDF-EOS 5 File
STRUC. METADATASTRUC. METADATA
HDFEOSHDFEOS
SWATHSWATH GRIDGRID POINTPOINT
Root -- “/”Root -- “/”
ADDITIONALADDITIONAL
Global (File) AttributesGlobal (File) Attributes
HDFEOSINFORMATION HDFEOSINFORMATION
ARRAYARRAY
ECS METADATAECS METADATA
11
Basic HDF-EOS Functions
• Access - Open, create, close objects/files
• Definition - Set key parameters of data sets
• Input/Output - Read/Write data
• Inquiry - Return information about data
• Subset - Read data from specified region
12
File/Object Access
GRID:
• HE5_GDopen/he5_gdopen
• HE5_GDcreate/he5_gdcreate
• HE5_GDattach/he5_gdattach
• HE5_GDdetach/he5_gddetach
• HE5_GDclose/he5_gdclose
13
File/Object Access
POINT:
• HE5_PTopen/he5_ptopen
• HE5_PTcreate/he5_ptcreate
• HE5_PTattach/he5_ptattach
• HE5_PTdetach/he5_ptdetach
• HE5_PTclose/he5_ptclose
14
File/Object Access
SWATH:
• HE5_SWopen/he5_swopen
• HE5_SWcreate/he5_swcreate
• HE5_SWattach/he5_swattach
• HE5_SWdetach/he5_swdetach
• HE5_SWclose/he5_swclose
15
File/Object Access
ZA:
• HE5_ZAopen/he5_zaopen
• HE5_ZAcreate/he5_zacreate
• HE5_ZAattach/he5_zaattach
• HE5_ZAdetach/he5_zadetach
• HE5_ZAclose/he5_zaclose
16
Definition
GRID:
• HE5_GDdefdim/he5_gddefdim
• HE5_GDdefproj/he5_gddefproj
• HE5_GDdeforigin/he5_gddeforigin
• HE5_GDdefpixreg/he5_gd_defpreg
17
Definition
POINT:
• HE5_PTdeflevel/he5_ptdeflevel
• HE5_PTdeflinkage/he5_ptdeflinkage
18
Definition
SWATH:
• HE5_SWdefdim/he5_swdefdim
• HE5_SWdefdimmap/he5_swdefmap
• HE5_SWdefidxmap/he5_swdefimap
• HE5_SWdefgeofield/he5_swdefgfld
• HE5_SWdefdatafield/he5_swdefdfld
19
Definition
ZA:
• HE5_ZAdefdim/he5_zadefdim
• HE5_ZAdefine/he5_zadefine
• HE5_ZAdefcomp/he5_zadefcomp
• HE5_ZAdefchunk/he5_zadefchunk
20
Input/Output
GRID:
• HE5_GDwritefield/he5_gdwrfld
• HE5_GDreadfield/he5_gdrdfld
• HE5_GDwriteattr/he5_gdwrattr
• HE5_GDreadattr/he5_gdrdattr
21
Input/Output
POINT:
• HE5_PTwritelevel/he5_ptwritelevel
• HE5_PTreadlevel/he5_ptreadlevel
• HE5_PTwriteattr/he5_ptwrattr
• HE5_PTreadattr/he5_ptrdattr
22
Input/Output
SWATH:
• HE5_SWwritefield/he5_swwrfld
• HE5_SWreadfield/he5_swrdfld
• HE5_SWwriteattr/he5_swwrattr
• HE5_SWreadattr/he5_swrdattr
23
Input/Output
ZA:
• HE5_ZAwrite/he5_zawrite
• HE5_ZAread/he5_zaread
• HE5_ZAwriteattr/he5_zawrattr
• HE5_ZAreadattr/he5_zardattr
24
Inquiry
GRID:
• HE5_GDinqdims/he5_gdinqdims
• HE5_GDdiminfo/he5_gddiminfo
• HE5_GDprojinfo/he5_gdprojinfo
• HE5_GDorigininfo/he5_gdorigininfo
• HE5_GDpixreginfo/he5_gdpreginfo
25
Inquiry
POINT:
• HE5_PTlevelinfo/he5_ptlevelinfo
• HE5_PTinqattrs/he5_ptinqattrs
• HE5_PTattrinfo/he5_ptattrinfo
26
Inquiry
SWATH:
• HE5_SWinqdatafields/he5_swinwdflds
• HE5_SWinqgeofields/he5_swinqgflds
• HE5_SWinqattrs/he5_swinqattrs
• HE5_SWattrinfo/he5_swattrinfo
27
Inquiry
ZA:
• HE5_ZAinquire/he5_zainquire
• HE5_ZAinfo/he5_zainfo
• HE5_ZAinqattrs/he5_zainqattrs
• HE5_ZAattrinfo/he5_zaattrinfo
28
Programming Model
• Open the HDF-EOS5 file• Create or Attach to a specified
Grid/Swath/Point/ZA object• Define key object features and/or
perform necessary operations (with the data sets, attributes, etc)
• Detach from the Grid/Swath/Point/ZA object
• Close the HDF-EOS5 file
29
HDF-EOS Functionality
• Basic File I/O• Swath Interface• Grid Interface• Point Interface• Profile Interface• Generalized Array
Interface (Zonal Avg. (HDF5))
• Subsetting
• Fill Values• Compression (gzip,
szip, shuffling)• Chunking/Tiling• Aliases• Global (File), Group
& Local Attributes• Threadsafe Version
30
Applications
• HE5View (HDF5 - based browser)• EOSView (HDF4 - based browser)• HDFView (with HDF-EOS Plug-in), Java EOS
Browser (HDF-EOS 2 and 5 access)• heconvert (converts HDF4 - based Grid/Point/Swath
structures to HDF-EOS 5 equivalents) – Compliments HDF4 -> HDF5 conversion tool
– Not tested on all products
• HEG (HDF-EOS to GeoTIFF Converter/Subsetter)
31
HDFView• HDFView: A Java-based browser from NCSA for
HDF4 and HDF5 – based data.• HDF-EOS plug-in extends HDFView to browse HDF-
EOS data in both versions.– One reason: EOSView is becoming obsolete
• The extension is a plug-in module and does not change NCSA HDFView functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects remains.
• Identical look and feel when displaying HDF and HDF-EOS objects.
• Users can access a complete breakdown of HDF-EOS objects. e.g. grid information, projection information, dimensions, datafields, and attributes are displayed.
32
HDFView (Cont.)• Release 2.2 October 2005
- A separate HDF-EOS plug-in: Users install HDFView 2.2 and integrates HDF-EOS plug-in into it.- Ability to do "Save as" for HDF-EOS5: modify individual values in grid or swath data. Delete entire grids or swaths. Save the result. - Separate help window for HDF-EOS objects
• Release 2.3 January 2006 - Point Object viewing capability
- Geolocating pixels for swath and grid imagesFuture Plans:
- Ability to modify input HDF-EOS2 granules. - Ability to cut/paste objects, modify/delete attributes.- Ability to create new grids/swaths
33
HDFView showing ASTER L1B swath
34
• A utility that converts EOSDIS data from HDF-EOS format to common Geographical Information System (GIS)-compatible formats.
• Two versions available:– A downloadable desktop version – Access through NASA archive online storage (Data
Pools)– Implemented also in prototypes for Stand-alone Data
Pool, OGC (Open Geospatial Consortium) for Web Services Chaining, and DOWS (Deploy OGC Web Services) on the DAACs
• Details of internal file structures are transparent to users.
HDF-EOS to GeoTIFF Converter (HEG)
35
HEG Functionality• Subsetting (spatial, field, band)• Performs conversion to a single-band or multi-
band GeoTIFF’s• HDF-EOS Swath to HDF-EOS Grid and Swath• HDF-EOS Swath or Grid to GeoTIFF• HDF-EOS Swath or Grid to generic Binary (with
metadata file)• Stitching (mosaicing) + subsetting + Reprojecting• Running from Command Line (batch jobs) or with
Java-based GUI• Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ,
GEO, SIN)• Subsampling (Subsample stacks)
36
HEG Functionality• Metadata preservation/creation.• Currently supports MODIS, MISR, ASTER, AIRS and AMSR-E
products on TERRA and AQUA (>110 products), AURA/HDF-EOS5 support in progress
• Allows file selection from users local storage• Operable on Sun, SGI, Win, Linux (64-bit soon), MAC• Integrated into ECS Data Pool
– Reduces the transfer time of HDF-EOS data sets (if subset is requested)
– Provides the end-user with the exact file required by their application• User Interface
– Portable, written in Java.– Not dependent on COTS (eg. IDL).
37
Availability
• Access to libraries and applications: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.htmlhttp://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html
http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.htmlhttp://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html
• Email– [email protected] – [email protected]– [email protected]– [email protected]
38
Mt. Etna Eruption: MISR L1B Swaths: Stitched
39
Mt. Etna: Stitched, Subsetted ASTER L1B
40
MODIS L1B: Reprojected, Co-Registered with ASTER, La Plata Tornado
41
La Plata Tornado Damage: ASTER L1B reprojected/stitched/subsetted
42
MLS L2GP-H2O Product (2004, day 243), Geographic Projection, Full Globe
43
MLS L2GP-H2O Product, PS Projection, Northern Hemisphere
44
2004 Hurricane Francis, MODIS
(MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5)
45
AST_L1B, 2003 California FiresGeo. projection, Multi-Band GeoTIFF,
(3 bands: Visibile Near Infrared (VNIR))
Two Swaths Stitched …..
….. and Subsetted