hdf5 opendap project
DESCRIPTION
MuQun Yang and Hyo -Kyung Lee (THG) James Gallagher ( OPeNDAP , Inc.). HDF5 OPeNDAP Project. Question of the Day. How can I get One Sub from SUBWAY? C hicken teriyaki with no onion . How can I get One Subset of EOS data image from NASA? - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/1.jpg)
HDF5 OPeNDAP Project
MuQun Yang and Hyo-Kyung Lee (THG)
James Gallagher (OPeNDAP, Inc.)
![Page 2: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/2.jpg)
Question of the Day
• How can I get One Sub from SUBWAY?• Chicken teriyaki with no onion
• How can I get One Subset of EOS data image from NASA?• Today’s satellite image right above the roof
![Page 3: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/3.jpg)
Subway
• Easy: anywhere / accepts Visa|MC
• Efficient: your own recipe• Cool: eat fresh and look good
How can I get One Sub from SUBWAY?
![Page 4: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/4.jpg)
Dumb-way
• Difficult: bring a USB drive and visit NASA
• Inefficient: download 10 Gbyte and search for 10 byte dataset
• Not cool: wrong or useless dataset
How can I get One Subset of EOS data image from NASA?
![Page 5: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/5.jpg)
DAP-way!
• Easy: anywhere / accepts IE|FF
• Efficient: WYSIWYG – less fat!
• Cool: Visualization clients – see fresh!
How can I get One Subset of EOS data image from NASA?
![Page 6: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/6.jpg)
DAP-way Implementation
• OPeNDAP: Free DAP-way
Software
• Server-Client Model APIs
• Widely used by Earth Science
Community
![Page 7: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/7.jpg)
OPeNDAP
• Market Dominance:
•
• http://www.opendap.org
OPeNDAP server sites
Courtesy of OPeNDAP, Inc.
![Page 8: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/8.jpg)
OPeNDAP Servers
Data Data Data Data Data Data Data
Matlab
HDF4 JDBC
FreeFromFITS
CDF CEDAR
Data
netCDF
netCDF HDF4 Matlab
Data
DSP
DSP
Data
JGOFS
Tables SQL FITS CDFFlat
Binary CEDAR
Data
CODAR
Data
ESML
GeneralCODAR
Courtesy of OPeNDAP, Inc.
![Page 9: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/9.jpg)
OPeNDAP Clients
netCDF C netCDF Java
IDVFerret GrADS VisAD ncBrowse Matlab ExcelIDL Access
MatlabClient
IDLClient
Courtesy of OPeNDAP, Inc.
![Page 10: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/10.jpg)
How OPeNDAP works
OPeNDAP Client
OPeNDAPHDF4 Server
HDF4
OPeNDAPNetCDF Server
NetCDF
![Page 11: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/11.jpg)
Architecture of OPeNDAP Server• Server 3 (CGI)
• C/C++ based HTTP Server• Insecure / Inflexible
• Server 4 (Hyrax)• Java based TomCat Servlet Engine
• Secure / Flexible
![Page 12: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/12.jpg)
Server 3 (CGI) Architecture
OLFS
Java Servlet Engine
BES
Unix Daemon
BES Commands
XML- encapsulated object
DataStore
File system with data files like HDF, NetCDF and SQL Database, …
DAP
XML, GIS, KML
HTML
Optional catalogs: XML, GIS, KML
Server 4 (Hyrax) ArchitectureHTTP Server
Courtesy of OPeNDAP, Inc.
http://docs.opendap.org/index.php/Hyrax
![Page 13: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/13.jpg)
Example Usage
OPeNDAP Client
OPeNDAPHDF4 Server
HDF4
Syntactic Structure of DataSemantic Meaning of Data
Actual Content of Data
OPeNDAP Visualization Client
![Page 14: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/14.jpg)
What about HDF5?
OPeNDAP Client
OPeNDAPHDF4 Server
HDF4
OPeNDAPNetCDF Server
NetCDF
OPeNDAPHDF5 Server
HDF5
![Page 15: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/15.jpg)
Why Important?
• HDF5: NASA EOS / NPOESS
• NASA’s MISSION: ACCESS!
• Our MISSION: Build DAP-way Bridge
![Page 16: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/16.jpg)
Goals
•1st: Transform HDF5 access in DAP-way
• 2nd: Yet preserve the beauty of HDF5
![Page 17: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/17.jpg)
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS Files
Goal: Transform HDF5 access in DAP-way
![Page 18: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/18.jpg)
Mapping HDF5 in DAP-way• Prototype server in 2001
• NASA Grant from Nov. 2006
• Support for Compound Datatype
• Support for Group
• Support for References / Links
• First Product Release in Mar. 2008
![Page 19: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/19.jpg)
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS Files
Goal: Transform HDF5 access in DAP-way
![Page 20: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/20.jpg)
Subway Customers
• I want 3 inch-long sub
• I need Wasabi sauce
• I eat only wheat bread
• I eat only meat balls
![Page 21: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/21.jpg)
OPeNDAP Clients
• I hate foot-long variable names.• I need special attributes on dataset.• I care pre-defined Grid data type.• I care only well-formed attributes.
Not all OPeNDAP clients are created equal!
![Page 22: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/22.jpg)
Tweaks for OPeNDAP Clients Two configuration options• --enable-short-name = cut bread• --enable-CF = put Wasabi sauce
![Page 23: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/23.jpg)
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS5 Files
Goal: Transform HDF5 access in DAP-way
![Page 24: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/24.jpg)
Subway Customers
• I want 3 inch-long
• I need Wasabi sauce
• I eat only wheat bread
• I eat only meat balls
![Page 25: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/25.jpg)
Subway Suppliers Problem
• No breads, only wheat!
• No meat balls, only meat!
• I eat only wheat bread
• I eat only meat balls
Subway Customers - OK
![Page 26: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/26.jpg)
HDF EOS
• Grid with No Dimension data • Clients expect Grid with Dimension data
• Metadata Attribute in Two Strings• Clients expect One Structured Format
Not all HDF5 files are created for DAP-way!
![Page 27: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/27.jpg)
Tweaks for HDF-EOS Two more configuration options• --enable-eos-grid = bake bread• --enable-eos-meta = make balls
![Page 28: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/28.jpg)
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS5 Files
Goal: Transform HDF5 access in DAP-way
![Page 29: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/29.jpg)
Day After Server Tweaks• Finally, Happy Clients!
GrADSNCLFerretMATLAB
ncBrowse
IDVODC
![Page 30: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/30.jpg)
Live DemoOzone concentration level over the south pole
![Page 31: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/31.jpg)
Goals
•1st: Transform HDF5 access in DAP-way
• 2nd: Yet Preserve the beauty of HDF5
![Page 32: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/32.jpg)
HDF5 Served in DAP-way (Yet Preserve the beauty of HDF5)
• There are some things money can’t buy. For everything else, there is MasterCardTM.
• There are some things DAP can’t serve. For everything else, there is _________?
Minimize
Maximize
![Page 33: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/33.jpg)
Some Things
• Hard: Opaque, Bitmap, Enum, 64 bit Integer, Variable Length types
• Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5
(that OPeNDAP HDF5 server can’t serve)
![Page 34: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/34.jpg)
For Everything Else
• Let there be HDF5-friendly OPeNDAP client library!
• Package and Deliver in DAP-way first.• Then, let the client library handle it.• Concept of Trojan Horse
![Page 35: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/35.jpg)
Example: Group in HDF5Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.
HDF5-friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5
![Page 36: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/36.jpg)
Example: Reference in HDF5
• Object / Regional Reference • Map to DAP URL at server• No de-referencing of URL at client library• Important for NPOESS
![Page 37: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/37.jpg)
HDF5-Friendly OPeNDAP Client Library
• One more reason: Tame Clients • Only ODC can display Swath properly.
• Easy but Evil OPeNDAP NC Client Library! easy: nc style API evil: fixed dimensional attributes
• Our client library must be Easy but Good
No Latitude and Longitude
Courtesy of NASA
![Page 38: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/38.jpg)
libnc-dap
???
liboc-dap
NC-Friendly OPeNDAP Client Library
libdap
dapserver
OPeNDAP NC Server
GrADS
NetCDFView NetCDFHDF5
Group/Ref/Swath
OPeNDAP HDF5 Server
View HDF5
GrADS
View G/R/Swath
HDF5-Friendly OPeNDAP Client Library
![Page 39: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/39.jpg)
Summary
• DAP-way access of HDF5 Is Easy / Efficient / Cool Loses some things Requires HDF5-friendly OPeNDAP Client Library
• HDF5-friendly OPeNDAP Client Library Serves HDF5 better Tames evil visualization clients
![Page 40: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/40.jpg)
Future Work• HDF5 DAP Mapping Document
• Finish HDF5-friendly OPeNDAP Client Library prototype
• Test it on GrADS and display Swath
![Page 41: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/41.jpg)
Project Website http://hdfdap.hdfgroup.uiuc.edu/joomla
• Beta preview• Feedbacks are more than welcome!
![Page 42: HDF5 OPeNDAP Project](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816592550346895dd85e62/html5/thumbnails/42.jpg)
Credits
Mike Folk (THG)Robert McGrath (NCSA)
Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)Christopher Lynnes, James Johnson, Denis Nadeau (NASA)
Jennifer Adams (GrADS)Dave Brown (UCAR)