hdf5 opendap project update and demo
DESCRIPTION
HDF5 OPeNDAP Project Update and Demo. MuQun Yang and Hyo -Kyung Lee (The HDF Group) James Gallagher ( OPeNDAP , Inc.). OPeNDAP Review. A software framework that allows simple access to remote data Data Access Protocol Client-server model Widely used by Earth Science Community. - PowerPoint PPT PresentationTRANSCRIPT
HDF5 OPeNDAP ProjectUpdate and Demo
MuQun Yang and Hyo-Kyung Lee (The HDF Group)
James Gallagher (OPeNDAP, Inc.)
1HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
OPeNDAP Review
• A software framework that allows simple access to remote data
• Data Access Protocol• Client-server model• Widely used by Earth Science Community
2HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
OPeNDAP client library
DAP 2.0/4.0
dap server(e.g. Apache)
OPeNDAP HDF/NetCDF
server
Visualization tools
(e.g. GrADS)
HDF/NetCDFView HDF/NetCDF
OPeNDAP Concept
3HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
OPeNDAP client Visualization
Tools
Example Usage
OPeNDAP Client
OPeNDAPHDF4 Server
HDF4
Syntactic Structure of DataSemantic Meaning of Data
Actual Content of Data
OPeNDAP Visualization Client4
HDF and HDF-EOS Workshop XII, Aurora, CO10/17/2008
Why Important?
• HDF5: NASA EOS / NPOESS
• NASA’s MISSION: ACCESS!
• Our MISSION: Build a DAP-HDF5 Bridge
5HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes into DAP Datatypes
2. Plus, some enhancements to make OPeNDAP Clients to access HDF-EOS5 Files
Goal: Access HDF5 data via OPeNDAP
6HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Mapping HDF5 Datatypes to DAP• Compound Datatype
• Support for Group
• Object/regional References
7HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Some OPeNDAP Clients
• Cannot handle long variable names.
• Need special attributes on dataset.
• Handle only well-formed attributes.
Not all OPeNDAP clients are created equal!
8HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Tweaks for OPeNDAP Clients Two configuration options
• --enable-short-name
• --enable-CF
9HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
HDF EOS
• Grid with No Dimension data Clients expect Grid with Dimension data
• Some Attributes stored in Extremely Long String(s) (e.g. StructMetada.0) Clients expect well-structured attributes
Not all HDF5 files are created for DAP!
10HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Tweaks for HDF-EOS Two more configuration options
• --enable-eos-grid
• --enable-eos-meta
11HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Day After Server Tweaks
• Finally, Happy Clients!
GrADSNCLFerretMATLAB
ncBrowse
IDVODC
IDL
12HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Goals
•1st : Access HDF5 via OPeNDAPThe OPeNDAP HDF5 handler was released on March, 2008 at OPeNDAP website http://www.opendap.org/
• 2nd: Construct HDF5 data from OPeNDAP Bonus: Help clients to better access EOS
Swath data 13
HDF and HDF-EOS Workshop XII, Aurora, CO10/17/2008
HDF5-Friendly OPeNDAP client library
14HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Example: Group in HDF5Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.
HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5
15HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Example: Reference in HDF5
• Object / Regional Reference
• Map to DAP URL
• Important for NPOESS
16HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
• Only ODC can display Swath properly.• One more reason: Help Clients to view Swath
HDF5-Friendly OPeNDAP Client Library
No Latitude and Longitude
Courtesy of NASA
17HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
libnc-dap
???
liboc-dap
NetCDF OPeNDAP Client Library
libdap
dapserver
OPeNDAP NC Server
GrADS
NetCDFView NetCDFHDF5
Group/Ref/Swath
OPeNDAP HDF5 Server
View HDF5
GrADS
View G/R/Swath
HDF5-Friendly OPeNDAP Client Library
18HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Summary of Client Library Prototype
• Finished coding • Tested with the GrADS visualization client• Documented • Working on a demo DAP to HDF5 tool by
using the client library prototype• May test with NCL
10/17/2008HDF and HDF-EOS Workshop XII, Aurora,
CO 19
Example: dap2h5
• A test application for the client library
• It can construct HDF5 from DAP output
ORIGINAL
CONSTRUCTED
20HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Demo: MLS swath via GrADS
• GrADS is modified to use our library
• The client library provides grid mapping
from swath data
• GrADS displays swath through HDF-EOS5
specific client library API calls
21HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Caution
• Our Client Library is ONLY PROTOTYPE! It does NOT support all DAP data types
It does NOT support all AURA files It does NOT support all Visualization clients
22HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Summary
• HDF5 Access via OPeNDAP Is Easy Is used by GES DISC to serve Aura files
• HDF5-Friendly OPeNDAP Client Library Prototype Serves HDF5 better(EOS swath) Benefits visualization clients
23HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Future Work• HDF5 to DAP2 Mapping Document
• Release HDF5-friendly OPeNDAP Client Library Prototype
• URL: http://hdfgroup.org/projects/opendap/index.html
24HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Credits
Mike Folk (The HDF Group)Robert McGrath (NCSA)
Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang (NASA)
Jennifer Adams (GrADS)Dave Brown (UCAR)
25HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008
Acknowledgement
This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX06AG75A. . Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.
26HDF and HDF-EOS Workshop XII, Aurora,
CO10/17/2008