hdf-eos status, related tools and issues

HDF-EOS Status, Related Tools and Issues

Abe Taaheri, Raytheon IIS2009 THG Briefing to ESDIS

TOOLKIT / HDF-EOS Support

• Maintenance support is funded on EMD Evolution through May 22, 2009

- Bridge contract until EODDIS Evolution & Development (EED) award in 2010

• Current support HDF-EOS 2 & 5 SDP/MTD Toolkit HDF-EOS plug-in for HDFView HDF-EOS2 to GeoTIFF Converter (HEG) Other Toolkit/HDF-EOS related software Toolkit/HDF-EOS/HEG helpdesk

ECS Support of HDF-EOS

• Both HDF4 and HDF5 based flavors of HDF-EOS are part of ECS baseline.

• OS’s: Solaris (9, 10), Irix6.5 (64, n32), Linux (including 64-bit Opteron and Itanium),

Mac OS X (Power PC and Mac Intel), Windows XP(MTD Toolkit, HDF-EOS2 & 5, HEG, HDFView)

• added full Cygwin support for next release

• Compilers: Fortran 77/90 & g77/pgf90 , C, C++, gcc, g++,Intel Fortran (SGI Altix), gfortran (Mac

Intel)

Downloadable Software

• SDP/MTD Toolkit (Science Data Processing Tools)

• HDF-EOS2 (HDF4 based)

• HDF-EOS5 (HDF5 based)

• heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents)

Downloadable Applications

• HEG

- GeoTIFF conversion

- Spatial subsetting, Field Subsetting

- reprojection

- stitching, sub-sampling and more ….

• DumpMetadata (Dump HDF-EOS metadata)

• HDFView Plug-in, Java-based browser (HDF4, HDF5, HDF-EOS 2 and 5 access)

• HE5View* (HDF5 - based browser)

• EOSView* (HDF4 - based browser)* Dropped support for OS updates on July 2006

Software Releases

Toolkit/HDF-EOS:

• Last release – March 2008– TOOLKIT 5.2.15 – HDF-EOS 2.15, HDF- EOS5.1.11

with HDF4.2r3, HDF5-1.6.7, Szip2.1

• Next release – May 2009– TOOLKIT 5.2.16 – HDF-EOS 2.16, HDF-EOS5.1.12

with new releases of HDF4 (2r4), 5 (1.8.3)

Software Releases (Cont.)

HEG (HDF-EOS2 to GeoTIFF Converter):

• Last release – August 2008

– Version 2.9

• Next release – May 2009

– Version 2.10

– Already have released Datapool version to the DAACs (Feb. 2009)


HDF-EOS Plug-in for HDFView:

• Last release – September 2008

– Version 2.4

• Next release – June 2009

– For HDFView Version 2.5


HECONVERT (HDF-EOS2 to HDF-EOS5 Converter):

• Last releases

– 2001, 2004 Version ?

– Oct. 2008 Version 2.0

DumpMetadata (ASCII Dump of HDF-EOS metadata)

• Last release

–Dec. 2008 Version 1.0

SDP Toolkit

• Science Data Production (SDP) Toolkit is a collection of tools used by

- Science software developers who produce

code to process instrument data.

- HDF-EOS users who need to• Access metadata in HDF-EOS files• Perform Time/Date conversion• Access Digital Elevation Model Data• Access Ancillary Data, such as Digital

Chartof the World database (DCW), Olson World Ecosystem files, DEM data files, etc.

SDP Toolkit (cont.)

• Tools provided in (SDP, MTD) Toolkit:- AA (Ancillary Data Access)- CBP (CELESTIAL Body Position)- CSC(Coordinate System Conversion)- CUC (Constant and Unit Conversions)- DEM (Digital Elevation Model Access)- EPH (Ephemeris Data Access)- GCT (Geo Coordinate Transformation)- IO (Input Output (File I/O))- MEM (Memory Management)- MET (Metadata Access)- PC (Process Control)- SMF (Status Message File (Error/Status))- TD (Time and Date Conversion)

What is HDF-EOS ?

• HDF-EOS is an extension to HDF which standardizes storage of and access to common Earth Sciences data structures

–By providing a single interface to data structures common to earth science

–By providing a container for EOS inventory, archive and product specific metadata

–By establishing a consistent relationship between geolocation and science data.

HDF-EOS 2

• HDF4 - based, storage format for EOS standard products.

• Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments

• Support for Grid/Point/Swath structures

• HDF4 and HDF-EOS 2 will be supported by THG/NASA for the fore-seeable future.

HDF-EOS 5

• Based on HDF5, a complete rewrite of HDF-EOS2 with a different user interface.

–First released in 2000.

–Format used by EOS Aura instruments

• Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible.

–Supports same data structures

–Added prefix ‘HE5_’ to HDF-EOS 2 functions.

HDFView

• HDFView: A Java-based browser from THG for HDF4 and HDF5 – based data.

• HDF-EOS plug-in extends HDFView to browse HDF-EOS data in both versions.

• The extension is a plug-in module and does not change THG HDFView functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects remains.

HDFView

• Identical look and feel when displaying HDF and HDF-EOS objects.

• Users can access a complete breakdown of HDF-EOS objects. e.g. grid information, projection info., dimensions, datafields, and attributes are displayed.

• Current Support: SUN, SGI, LINUX, Mac, WINDOWS platforms

Swath, Grid, and Point Objects

AMSR-E Swath

Tableview Features in Version 2.4

HEG: Utility that converts EOSDIS data from HDF-EOS format to common Geographical Information System (GIS)-compatible formats.

• Two versions available:–A downloadable desktop version –Access through NASA archive online storage (Data Pools)

– Implemented also in some prototypes:▪ Stand-alone Data Pool▪ OGC (Open Geospatial Consortium) for Web Services Chaining▪ DOWS (Deploy OGC Web Services) on the DAACs

HDF-EOS to GeoTIFF Converter (HEG)

HEG Functionality

• Conversion:

– HDF-EOS (Swath or Grid ) to a single-band or multi-band GeoTIFF’s

– HDF-EOS Swath to HDF-EOS Grid

– HDF-EOS Swath or Grid to generic Binary

– Metadata preservation/creation.

– Allows file selection from users local storage

HEG Functionality

• Subsetting (spatial, field, band)

• Stitching (mosaicing) + subsetting + Reprojecting

• Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ,

GEO, SIN, Albers)

• Subsampling (Subsample stacks)

• Running from Command Line (batch jobs)

or with Java-based GUI

HEG Functionality

• Currently supports MODIS, MISR, ASTER, AIRS and AMSR-E products on TERRA and AQUA (>110 products),

• Operable on Sun, SGI, Win, Linux, MAC

• Integrated into ECS Data Pool– Reduces the transfer time of HDF-EOS data sets (if subset is requested)

– Provides the end-user with the exact file required by their application

• User Interface– Portable, written in Java.

– Not dependent on COTS (eg. IDL).

Mt. Etna Eruption: MISR L1B Stitched Blocks

2004 Hurricane Francis, MODIS

MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5

AST_L1B, 2003 California FiresGeo. projection, Multi-Band GeoTIFF,

(3 bands: Visible Near Infrared (VNIR))

Two Swaths Stitched …..

….. and Subsetted

TOOLKIT/HDF-EOS:

SZIP Insatallation: Default decoder,

Choice for encoder

Choice for configuring HDF4 with/without SZIP

Choice for disabling HDF4’s internal netCDF

Choice for installing Toolkit/HDF-EOS in 32 or 64-bit modes in 64-bit Linux platforms

(except for Itanium; no –m32 flag)

Recent Major Development/Fixes

Support for new releases of HDF4/HDF5/SZIP

(HDF4.2r4, HDF5-1.8.X, SZIP-2.1)

Auto Configuration capability (like HDF) for

HDF-EOS2 and HDF-EOS5

Support for Mac Intel platform. Support for CYGWIN


Added APIs GDij2ll and GDll2ij to HDF-EOS2 and HDF-EOS5

Added HDF_EOS 2 & 5 utilities for dumping grid lat/lon into ASCII or binary files

Bug fixes such as:- HDFEOS5: Attribute data type inquiry (code returned not useful for reuse)

- HDFEOS5: Tiling/comp. consistency between dataset and structmetadata

- HDF-EOS2: Correct diminfo for a field (value in structmetadata is wrong?)

- HDF-EOS5: wrong datatype info for INT8 data (C code)

- Toolkit: Problem in pccheck utility( hangs if a blank space at a line start )

- Fix for ODL core dump (too many file names for Input Pointer)


HDF-EOS Plug-in for HDFView:

Current Support: SUN, LINUX, WINDOWS, and MAC (Power and Intel)

– Auto installation of plug-in into HDFView running self-extracting archives.

– Adding to plug-in the fixes and enhancement that went into HDFView2.4 and 2.5

– Display image for selected pixels in the viewed tables


– Display Lat/Lon for a selected pixel in the

data field tables

– Full Mac OSX support

– Improved Help Window

– New icons for activating Lat/Lon display for

Swath & Grid

– Format conversion: HDF-EOS2 -> HDF-EOS5

– Correction for compression and tiling values for grid fields


HEG: Product Support:

AMSR-E:- Land products (CEA Projection added)- AMSR-E 5 day Snow, and Monthly Snow- Weekly Ocean, and Monthly Ocean- Daily 12 Km, and Daily 6 Km Sea Ice- L2 Rain, and Rain Grid- ASTR_07XT, ASTR_09XT, ASTR_L1BE,

MOD29E1D- More new versions and products tested. Will

update Users’ Guide and HEG web site


• Stitching fixes:

− MOD14A1 3-D fields (band selection)

− TM grids

• Multi-band geotif/HDF-EOS in Datapool

• Other Fixes:

− Interpolation correction with AIRS products

containing fillvalue in the data fields

− Pixel size unit consistency in conversion and stitch GUIs.


− Correction for droping a column when

stitching MODIS SIN tiles.

− Fixes for processing with default pixel size instead of ignoring it for Geographic projection

− Spatial subsetting fix for global AMSR-E garanules

− Fix for ignoring re-projection of type NN interpolation after stitching

− support pixel resampling for gridded data


Future Plans

TOOLKIT/HDF-EOS:

• Auto Configuration for TOOLKIT

• XML output for Metadata in TOOLKIT

• Update of DEM land-sea mask

• Support for HDF5-1.8.X

• Validation tool for HDF-EOS5 files

Future Plans (cont.)

HDF-EOS Plug-in for HDFView:• More functionalities :

- Ability to modify input HDF-EOS2 granules. - Ability to cut/paste objects, modify/delete attributes.- Ability to create new grid/swath

HEG:- Support for AMSR-E Level 3 Land (CEA projection)- New MRT (MODIS Reprojection Tool) release updates- Support HEG for MAC Intel

Availability

• Access to libraries, applications, Documentation:

TOOLKITTOOLKIT:: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.htmlhttp://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html

HDFViewHDFView::http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/

HDFView_hdfeos_plugin.htmlHDFView_hdfeos_plugin.html

HEGHEG: : http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.htmlhttp://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html

• Email–[email protected]–[email protected]

hdf-eos status, related tools and issues

Documents