www.hdfgroup.org the hdf group esip summer meeting easy access hdf files via hyrax kent yang the hdf...
TRANSCRIPT
www.hdfgroup.org
The HDF Group
1ESIP Summer Meeting
Easy access HDF files via Hyrax
Kent YangThe HDF Group
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 2
Outline
• Why OPeNDAP?• Hyrax features • Hyrax modules• Demos• Suggestions
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 3July 8 – 11, 2014
Why OPeNDAP?
• Check metadata remotely (in various forms)• Obtain the subset of data easily and efficiently• Hide the original data sources
• netCDF, HDF4, HDF5 or GeoTiFF, GRIB • Many popular earth science tools can visualize
and analyze the data via OPeNDAP• OPeNDAP output(including subsets) can be
downloaded as other formats
www.hdfgroup.orgESIP Summer Meeting 4July 8 – 11, 2014
Doesn’t helpGreat!
www.hdfgroup.orgESIP Summer Meeting 5July 8 – 11, 2014
• The visualization of HDF(5) data via Hyrax• What NASA users request the most
HDF(5) Hyrax modules
www.hdfgroup.orgESIP Summer Meeting 6
Helpful Hyrax feature – BES keys
• BES keys to dynamic change the program flow• Easy for users to choose what they want• Easy for developers to add/revise features
requested by users
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 7
Example on using BES keys
July 8 – 11, 2014
Float /CERES_Unfiltered_Radiances/all_toa_sw_rad[24][180][360]
Float CERES_Unfiltered_Radiances_all_toa_sw_rad[gmt_hour_index = 24][latitude =180][longitude= 360]
www.hdfgroup.orgESIP Summer Meeting 8
Hyrax modulation
• Core framework implemented in C++• Just need to provide DDS, DAS and Data to Hyrax
libdap and BES• A handler can be sophisticated • HDF handler examples
• Retrieve latitude and longitude for HDF-EOS grid• Interpolate the latitude and longitude for HDF-EOS
swath with dimension map• Make sure the dimensions, variable names and
attributes follow CF conventions
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 9July 8 – 11, 2014
Non-CF CF
Float CERES_Unfiltered_Radiances_all_toa_sw_rad[gmt_hour_index = 24][latitude =180][longitude= 360]
Float /CERES_Unfiltered_Radiances/all_toa_sw_rad[24][180][360]Float /CERES_Unfiltered_Radiances/all_toa_sw_rad[24][180][360]
www.hdfgroup.orgESIP Summer Meeting 10
Demos – HDF-EOS2 (AIRS grid)
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 11
Demo 2 – HDF-EOS2 (MODIS swath)
July 8 – 11, 2014
Swath with Dimension Map
www.hdfgroup.orgESIP Summer Meeting 12
Demo 3 – HDF-EOS5 (OMI grid)
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 13
Hyrax modules that help the easy access
• Filenet CDF module• Can download DAP output as a netCDF-3 or netCDF-
4 classic files
• NcML module• Add/delete/revise metadata information of the data
source• Aggregate granules
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 14
Demo
July 8 – 11, 2014
• GPM level 1B(HDF5)
www.hdfgroup.orgESIP Summer Meeting 15
HDF(5) and file netCDF demo
• FilenetCDF module can work with the HDF handlers to download the HDF files or subset of the files as netCDF-3 or netCDF-4 classic files that follow the CF conventions
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 16
HDF(5) and NcML demo
• NcML module can be used with HDF handlers to provide the missing CF information
• AVHRR example
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 17
HDF(5), file netCDF and NcML demo
• HDF handlers, NcML module and file netCDF module can work together
• AVHRR example
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 18
Summary of modules to serve HDF files
July 8 – 11, 2014
Hyrax core
HDF(5) files
Main engineHDF(5) handlers
Supplement Info
NcML
DAP clients(IDV, Ferret, Panoply, IDL…...)
netCDFfiles
Alternative outputFile netCDF
www.hdfgroup.orgESIP Summer Meeting 19
Demo - Beyond the web service
• BES standalone program • Connect the HDF handlers, the file netCDF module
and the NcML module• End users can build Hyrax and use this program as a
command-line tool
• Can it be a standalone tool?
July 8 – 11, 2014
besstandalone -c /etc/bes/bes.conf -i in4-GPM-l1b > GPM-l1b.nc4
www.hdfgroup.orgESIP Summer Meeting 20
Future - Performance
• Improve the BES caching mechanism• Critical for large file access and aggregation service
July 8 – 11, 2014
www.hdfgroup.org
The HDF Group
21ESIP Summer Meeting
Earth Science GroupTed Habermann
Aleksandar JelenakH. Joe Lee
Joel PlutchakJohn Readey
Kent Yang
July 8 – 11, 2014