status of hdf-eos, related software, and tools

Status of HDF-EOS,Related Software, and Tools

Abe Taaheri, Raytheon IIS

HDF & HDF-EOS Workshop XII

October 2008

TOOLKIT / HDF-EOS Support

• Maintenance support was funded on EMD Evolution through May 2008

• One year extension until May 2009

• Current support HDF-EOS2 & 5 SDP/MTD Toolkit HDF-EOS plug-in for HDFView HDF-EOS2 to GeoTIFF Converter (HEG) Other Toolkit/HDF-EOS related software Toolkit/HDF-EOS/HEG helpdesk

ECS Support of HDF-EOS

• Both HDF4 and HDF5 based flavors of HDF-EOS are part of ECS baseline.

• OS’s: Solaris (9, 10), Irix6.5 (64, n32), Linux (including 64-bit Opteron and Itanium),

Mac OS X (Power PC and Mac Intel), Windows XP(MTD Toolkit, HDF-EOS2 & 5, HEG, HDFView)

(Dropped support for Solaris 8, HP, HP11, Dec, IBM in 2.15 release)

• Planned full Cygwin support for late 2008

• Compilers: Fortran 77/90 & g77/pgf90 , C, C++, gcc, g++,Intel Fortran (SGI Altix), gfortran (Mac Intel)

Downloadable Software

• SDP/MTD Toolkit (Science Data Processing Tools)

• HDF-EOS2 (HDF4 based)

• HDF-EOS5 (HDF5 based)

• heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents)

Downloadable Applications

• HEG

- GeoTIFF conversion

- Spatial subsetting, Field Subsetting

- reprojection

- stitching, sub-sampling and more ….

• HDFView Plug-in, Java-based browser (HDF4, HDF5, HDF-EOS 2 and 5 access)

• HE5View* (HDF5 - based browser)

• EOSView* (HDF4 - based browser)* Dropped support for OS updates on July 2006

Software Releases

Toolkit/HDF-EOS:

• Last release – March 2008– TOOLKIT 5.2.15 – HDF-EOS 2.15, HDF- EOS5.1.11

with HDF4.2r3, HDF5-1.6.7, Szip2.1

• Next release – March 2009– TOOLKIT 5.2.16 – HDF-EOS 2.16, HDF-EOS5.1.12

with new releases of HDF4, 5 (possibly 1.8.X)

Software Releases (Cont.)

HEG (HDF-EOS2 to GeoTIFF Converter):

• Last release – August 2008

– Version 2.9

• Next release – May 2009

– Version 2.10


HDF-EOS Plug-in for HDFView:

• Last release – September 2008

– Version 2.4

• Next release –

– For HDFView Version 2.5


HECONVERT (HDF-EOS2 to HDF-EOS5 Converter):

• Last release – 2001, 2004

– Version ?

• Next release – Dec. 2008

– Version ?

SDP Toolkit

• Science Data Production (SDP) Toolkit :collection of tools used by

- Science software developers who producecode to process instrument data.

- HDF-EOS users who need to• Access metadata in HDF-EOS files• Perform Time/Date conversion• Access Digital Elevation Model Data• Access Ancillary Data, such as Digital

Chartof the World database (DCW), Olson World Ecosystem files, DEM data files, etc.

SDP Toolkit (cont.)

• Tools provided in (SDP, MTD) Toolkit:- AA (Ancillary Data Access)- CBP (CELESTIAL Body Position)- CSC(Coordinate System Conversion)- CUC (Constant and Unit Conversions)- DEM (Digital Elevation Model Access)- EPH (Ephemeris Data Access)- GCT (Geo Coordinate Transformation)- IO (Input Output (File I/O))- MEM (Memory Management)- MET (Metadata Access)- PC (Process Control)- SMF (Status Message File (Error/Status))- TD (Time and Date Conversion)

What is HDF-EOS ?

• HDF-EOS is an extension to HDF which standardizes storage of and access to common Earth Sciences data structures

– By providing a single interface to data structures common to earth science

– By providing a container for EOS inventory, archive and product specific metadata

– By establishing a consistent relationship between geolocation and science data.

HDF-EOS 2

• HDF4 - based, storage format for EOS standard products.

• Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments

• Support for Grid/Point/Swath structures

• HDF4 and HDF-EOS 2 will be supported by THG/NASA for the fore-seeable future.

HDF-EOS 5

• Based on HDF5, a complete rewrite of HDF-EOS2 with a different user interface.

– First released in 2000.

– Format used by EOS Aura instruments

• Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible.

– Supports same data structures

– Added prefix ‘HE5_’ to HDF-EOS 2 functions.

HDFView

• HDFView: A Java-based browser from THG for HDF4 and HDF5 – based data.

• HDF-EOS plug-in extends HDFView to browse HDF-EOS data in both versions.

• The extension is a plug-in module and does not change THG HDFView functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects remains.

• Identical look and feel when displaying HDF and HDF-EOS objects.

• Users can access a complete breakdown of HDF-EOS objects. e.g. grid information, projection info., dimensions, datafields, and attributes are displayed.

• Current Support: SUN, SGI, LINUX, Mac, WINDOWS platforms

Swath, Grid, and Point Objects

AMSR-E and OMI Point objects

AMSR-E Swath

New Features in Release Version 2.4

HE2 To HE5 Conversion

New Features in Release Version 2.4

HEG: Utility that converts EOSDIS data from HDF-EOS format to common Geographical Information System (GIS)-compatible formats.

• Versions Available:– Stand-alone: A downloadable desktop version – Data Pool: Access through NASA archive online storage

– Implemented also in prototypes for Stand-alone Data Pool, OGC (Open Geospatial Consortium) for Web Services Chaining, and DOWS (Deploy OGC Web Services) on the DAACs

HDF-EOS to GeoTIFF Converter (HEG)

HEG Functionality

• Conversion:–HDF-EOS (Swath or Grid ) to a single-band or multi-

band GeoTIFF’s–HDF-EOS Swath to HDF-EOS Grid or Swath–HDF-EOS Swath or Grid to generic Binary (with

metadata file)• Subsetting (spatial, field, band)• Stitching (mosaicing) + subsetting + Reprojecting• Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN,

Albers)• Subsampling (Subsample stacks)

• Running from Command Line (batch jobs) or with Java-based GUI

HEG Functionality

• Metadata preservation/creation.

• Currently supports MODIS, MISR, ASTER, AIRS and AMSR-E products on TERRA and AQUA (>115 products),

• Allows file selection from users local storage

• Operable on Sun, SGI, Win, Linux, MAC

• Integrated into ECS Data Pool– Reduces the transfer time of HDF-EOS data sets (if subset is requested)

– Provides the end-user with the exact file required by their application

• User Interface– Portable, written in Java.

– Not dependent on COTS (eg. IDL).

Mt. Etna Eruption: MISR L1B Stitched Blocks

2004 Hurricane Francis, MODIS

MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5

AST_L1B, 2003 California FiresGeo. projection, Multi-Band GeoTIFF,

(3 bands: Visible Near Infrared (VNIR))

Two Swaths Stitched …..

….. and Subsetted

TOOLKIT/HDF-EOS:

SZIP Insatallation: Default decoder,

Choice for encoder

Choice for configuring HDF4 with/without SZIP

Choice for disabling HDF4’s internal netCDF

Choice for installing Toolkit/HDF-EOS in 32 or 64-bit modes in 64-bit Linux platforms

(except for Itanium; no –m32 flag)

Major Development/FixesLast Releases

Support for new releases of HDF4/HDF5/SZIP

(HDF4.2r3, HDF5-1.6.7, SZIP-2.1)

Installation fix for a problem caused by limits.h

in some Linux platforms

Auto Configuration capability (like HDF) added

for both HDF-EOS2 and HDF-EOS5

Major Development/Fixes (cont.)

Support for Mac Intel platform. Drop of support for Sun5.8, Dec, HP, HP 11, IBM

Fixes for Lat/Lon to pixel conversions in DEM tools for Linux to get results similar to those in Unix platforms Add GDij2ll and GDll2ij APIs to HDF-EOS and HDF-EOS5

Modified accuracy check for quaternion in CSC tools


HDFEOSVersion string clean up in HDF-EOS5

Added new utilities to dump lat/lon arrays for grids in HDF-EOS and HDF-EOS5

Fixed several other bugs for Grid. Swath and Point objects (complete list in release notes)



HDF-EOS Plug-in for HDFView:

Current Support: SUN, LINUX, WINDOWS, and MAC (Power and Intel)

– Auto installation of plug-in into HDFView running self-extracting archives.

– Adding to plug-in the fixes and enhancement that went into HDFView2.4

– Display image for selected pixels in the viewed tables


– Display Lat/Lon for a selected pixel in the

data field tables

– Full Mac OSX support

– Improved Help Window

– New icons for activating Lat/Lon display for

Swath & Grid

– Format conversion: HDF-EOS2 -> HDF-EOS5


HEG: Release Spatial subsetting:

- AE-DyOcn, AE_DySno• New Product: AMSR-E and ASTER (On-Demand L2)

- AMSR-E 5 day Snow- AMSR-E Monthly Snow- AMSR-E Weekly Ocean- AMSR-E Monthly Ocean- AMSR-E Daily 12 Km Sea Ice- AMSR-E Daily 6 Km Sea Ice- AMSR-E L2 Rain, AMSR-E Rain Grid- ASTR_07XT, ASTR_09XT


• Stitching:

− MOD14A1 3-D fields (band selection)

− TM grids

• Multi-band geotif/HDF-EOS in Datapool

• Other Fixes:

− Interpolation correction with AIRS products

containing fillvalue in the data fields

− Pixel size unit consistency in conversion and stitch GUIs.


− Correction for droping a column when

stitching MODIS SIN tiles.

− Correction to ignoring of Geographic pixel size

and processing with default pixel size

− Spatial subsetting fix for global AMSR-E garanules

− Correction for ignoring re-projection interpolation type NN after stitching

Future Plans

TOOLKIT/HDF-EOS:• Auto Configuration for TOOLKIT

• Full Support for Cygwin platform

• Support for HDF5-1.8

• Validation tool for HDF-EOS5 files

• Performance improvement

– reading HDF format eph/att files in Toolkit

– HDF-EOS5 by redesigning some functions

Future Plans (cont.)

HDF-EOS Plug-in for HDFView:• More functionalities :

- Ability to modify input HDF-EOS2 granules.

- Ability to cut/paste objects, modify/delete attributes.

- Ability to create new grid/swath

HEG:- Support for AMSR-E Level 3 Land (CEA projection)

- New MRT (MODIS Reprojection Tool) release updates

Availability

• Access to libraries, applications, Documentation:

TOOLKITTOOLKIT:: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.htmlhttp://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html

HDFViewHDFView::http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/

HDFView_hdfeos_plugin.htmlHDFView_hdfeos_plugin.html

HEGHEG: : http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.htmlhttp://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html

• Email–[email protected]–[email protected]

status of hdf-eos, related software, and tools

Technology

hdfeos files

hdfeos format

hdfeos objects

hdfeos users

hdf eos5

status of hdf

flavors of hdf

hdfview hdfeos2