www.hdfgroup.org the hdf group november 3-5, 2009 hdf-opendap project update hdf/hdf-eos workshop...

www.hdfgroup.org

The HDF Group

November 3-5, 2009

HDF-OPeNDAP Project Update

HDF/HDF-EOS Workshop XIII 1

Joe Lee and Kent Yang The HDF Group

James Gallagher OPeNDAP, Inc.

The 13th HDF and HDF-EOS Workshop November 3-5, 2009

www.hdfgroup.orgNovember 3-5, 2009

Outline

• Introduction to OPeNDAP

• What’s New?

oHDF5-OPeNDAP Handler (minor)

oHDF4-OPeNDAP Handler (major)

HDF/HDF-EOS Workshop XIII 2

www.hdfgroup.orgNovember 3-5, 2009

Access HDF via FTP

HDF/HDF-EOS Workshop XIII 3

Users

FTP Clients

IDL

FTP Servers

HDF Files

AIRS/OMI

FTP

Local HDF Files

Aqua/Aura

www.hdfgroup.orgNovember 3-5, 2009

Client Library(libnc_dap)

Data Access Protocol (DAP) via Internet

Server(Hyrax)

Handler(hdf4/5_handler)

Remote Data(HDF4/5)

View Data (User)

OPeNDAP Concept

4

Visualization Tools

(gradsdap)

Typical Remote Access

FTP/HTTP

Visualization tools

(e.g., grads)

HDF/HDF-EOS Workshop XIII

www.hdfgroup.orgNovember 3-5, 2009

Access HDF via OPeNDAP

HDF/HDF-EOS Workshop XIII 5

Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF Files

Hyrax AIRS/OMI

DAP

libnc-dap

Visualization Tools

HDF HANDLER

Aqua/Aura

www.hdfgroup.orgNovember 3-5, 2009

What’s New?

HDF5-OPeNDAP Handler Work

HDF/HDF-EOS Workshop XIII 6

www.hdfgroup.orgNovember 3-5, 2009

HDF5 Handler Update

HDF/HDF-EOS Workshop XIII 7

Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF5 Files

Hyrax OMI Grid

DAP

libnc-dap

Aura

OMI Swath

HDF5 HANDLER Update

Visualization Tools

www.hdfgroup.orgNovember 3-5, 2009

HDF5-OPeNDAP Handler Update

• Support Aura HDF-EOS5 OMI Swath

• Access some files that follow HDF5

Dimension Scale Specification

• Change in Short Name Option

HDF/HDF-EOS Workshop XIII 8

RadiativeCloudFraction

A3RadiativeClou

www.hdfgroup.orgNovember 3-5, 2009

Summary: Updated HDF5 Handler

• Official Release: October 6, 2009

• http://opendap.org/hdf5_handler

HDF/HDF-EOS Workshop XIII 9

www.hdfgroup.orgNovember 3-5, 2009

What’s New?

HDF4-OPeNDAP Handler Work

HDF/HDF-EOS Workshop XIII 10

www.hdfgroup.orgNovember 3-5, 2009

HDF4-OPeNDAP Handler Work

• Problems with the old HDF4 Handler

• Our Work

• Limitations

• Future Work

HDF/HDF-EOS Workshop XIII 11

www.hdfgroup.orgNovember 3-5, 2009

Old HDF4 Handler Problem

• OPeNDAP Visualization Tools CAN

NOT display most NASA HDF-EOS2

and HDF4 data

HDF/HDF-EOS Workshop XIII 12

www.hdfgroup.orgNovember 3-5, 2009

Old HDF4 Handler Problem

HDF/HDF-EOS Workshop XIII 13

Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF4 Files

Hyrax AIRS Grid

DAP

libnc-dap

Aqua

Old HDF4 HANDLER

Visualization Tools

www.hdfgroup.orgNovember 3-5, 2009

Our Work

• Let Visualization Clients Access HDF-EOS2 and HDF4 files

HDF/HDF-EOS Workshop XIII 14

www.hdfgroup.orgNovember 3-5, 2009

What’s Missing?

• My house costs 1. • Data – Computer Science

• int my_house = 1; // Use boolean to save?

HDF/HDF-EOS Workshop XIII 15

• My house in Hawaii costs $1M. • Information – Earth Science

• Geo-Location, Geo-Location, Geo-Location

• Unit, Scale factor

www.hdfgroup.orgNovember 3-5, 2009

What’s Missing?

• Clients expect certain data structureso DAP Grids and Shared dimensions

• Clients expect some key attributeso CF conventions

HDF/HDF-EOS Workshop XIII 16

www.hdfgroup.orgNovember 3-5, 2009

Solution: Metadata Parser

HDF/HDF-EOS Workshop XIII 17

• Goal: Meet the Client’s Requirements

• Parse the structMetadata too generate the DAP Grids and Shared Dims

o generate the CF-compliant attributes

www.hdfgroup.orgNovember 3-5, 2009

New HDF4 Handler (parser)

HDF/HDF-EOS Workshop XIII 18

Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF4 Files

Hyrax AIRS Grid

DAP

libnc-dap

Aqua

Old HDF4 HANDLER

Visualization Tools

New HDF4 HANDLER (parser)

www.hdfgroup.orgNovember 3-5, 2009

Demo: Parser Solution

HDF/HDF-EOS Workshop XIII 19

Problem: Only a few HDF-EOS2 files work!

AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdfhttp://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10MODIS GridMOD08_D3.A2008284.005.2008289105354.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html

AIRS Swath ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/

AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G08331121849.hdfMODIS Swath

MOD06_L2.A2001222.0025.004.2003127111841.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.html

www.hdfgroup.orgNovember 3-5, 2009

New Challenges for HDF4 Handler

• Hundreds of HDF-EOS2 and HDF4 products

oGrid Projections

o Swath Dimension Map

HDF/HDF-EOS Workshop XIII 20

www.hdfgroup.orgNovember 3-5, 2009 HDF/HDF-EOS Workshop XIII 21

HDF-EOS2 Grid projections• Geographic• Sinusoidal• Polar Stereographic• etc.

New Challenges

HDF-EOS2 Swath Dimension Map

www.hdfgroup.orgNovember 3-5, 2009

Solution: Use HDF-EOS2 Library

HDF/HDF-EOS Workshop XIII 22

• Goal: Support more HDF-EOS2 products

• Use HDF-EOS2 Library too handle different Grid projections

o handle Swath dimension map

www.hdfgroup.orgNovember 3-5, 2009

New HDF4 Handler (EOS2 library)

HDF/HDF-EOS Workshop XIII 23

Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF4 Files

Hyrax MODIS Swath

DAP

libnc-dap

Aura

MODIS Swathw/ Dim. Map

New HDF4 HANDLER (parser)New HDF4 HANDLER (EOS2 library)

Visualization Tools

New HDF4 Handler (Parser)

www.hdfgroup.orgNovember 3-5, 2009

Demo: HDF-EOS2 Library Solution

HDF/HDF-EOS Workshop XIII 24

Swath with Dimension Map

www.hdfgroup.orgNovember 3-5, 2009

Two Solutions

Metadata Parser

• Pro: Simple installation

• Con: Limited products

HDF-EOS2 Library

• Pro: More products

• Con: HE2 library installation

HDF/HDF-EOS Workshop XIII 25

www.hdfgroup.orgNovember 3-5, 2009

Two Solutions - Implementation

Metadata Parser

• hdf5_handler

HDF-EOS2 Library

• h4toh5 tool

HDF/HDF-EOS Workshop XIII 26

• Adapted the implementation of previous software packages

www.hdfgroup.orgNovember 3-5, 2009

Solution Comparison

HDF/HDF-EOS Workshop XIII 27

www.hdfgroup.orgNovember 3-5, 2009

Live Demo

• Tutorial Exampleo http://hdfeos.org/software/idv.php

• Demo Serverso http://hdfgroup.org/projects/opendapo http://hdfgroup.org/projects/opendap4

HDF/HDF-EOS Workshop XIII 28

www.hdfgroup.orgNovember 3-5, 2009

Limitations

• 7 Reasons for Visualization Failures

• Sample Files for Visualization Failures

o http://hdfgroup.org/projects/opendap4/limits.html

HDF/HDF-EOS Workshop XIII 29

www.hdfgroup.orgNovember 3-5, 2009

The Tip of the Iceberg

HDF/HDF-EOS Workshop XIII 30

The current HDF4 handler

CAN NOT visualize most of NASA HDF-EOS2 / HDF4 files

www.hdfgroup.orgNovember 3-5, 2009

More Challenges for HDF4 Handler

• Tackle limitations and other challenges.

• Handle NASA HDF4 (e.g.,CERES) files.

• In 2010

HDF/HDF-EOS Workshop XIII 31

www.hdfgroup.orgNovember 3-5, 2009

Summary: Enhanced HDF4 Handler

• Official Release on Oct. 6, 2009.

• http://opendap.org/hdf4_handler

• Support some HDF-EOS2 Grid / Swath files

• Use HDF-EOS2 library

• Future work: support more NASA HDF-EOS2

/ HDF4 files

HDF/HDF-EOS Workshop XIII 32

www.hdfgroup.org

The HDF Group

November 3-5, 2009

Thank You!

HDF/HDF-EOS Workshop XIII 33

www.hdfgroup.orgNovember 3-5, 2009

Acknowledgements

This work was supported by cooperative agreement number NNX06AC83A, NNX08AO77A and

NNX06AG75A from the National Aeronautics and Space Administration (NASA).

Any opinions, findings, conclusions, or recommendations expressed in this material are

those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space

Administration.

HDF/HDF-EOS Workshop XIII 34

www.hdfgroup.org

The HDF Group

November 3-5, 2009

Questions/comments?

HDF/HDF-EOS Workshop XIII 35