![Page 1: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/1.jpg)
www.hdfgroup.org
The HDF Group
1ESIP Summer Meeting
Easy access HDF files via Hyrax
Kent YangThe HDF Group
July 8 – 11, 2014
![Page 2: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/2.jpg)
www.hdfgroup.orgESIP Summer Meeting 2
Outline
• Why OPeNDAP?• Hyrax features • Hyrax modules• Demos• Suggestions
July 8 – 11, 2014
![Page 3: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/3.jpg)
www.hdfgroup.orgESIP Summer Meeting 3July 8 – 11, 2014
Why OPeNDAP?
• Check metadata remotely (in various forms)• Obtain the subset of data easily and efficiently• Hide the original data sources
• netCDF, HDF4, HDF5 or GeoTiFF, GRIB • Many popular earth science tools can visualize
and analyze the data via OPeNDAP• OPeNDAP output(including subsets) can be
downloaded as other formats
![Page 4: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/4.jpg)
www.hdfgroup.orgESIP Summer Meeting 4July 8 – 11, 2014
Doesn’t helpGreat!
![Page 5: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/5.jpg)
www.hdfgroup.orgESIP Summer Meeting 5July 8 – 11, 2014
• The visualization of HDF(5) data via Hyrax• What NASA users request the most
HDF(5) Hyrax modules
![Page 6: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/6.jpg)
www.hdfgroup.orgESIP Summer Meeting 6
Helpful Hyrax feature – BES keys
• BES keys to dynamic change the program flow• Easy for users to choose what they want• Easy for developers to add/revise features
requested by users
July 8 – 11, 2014
![Page 7: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/7.jpg)
www.hdfgroup.orgESIP Summer Meeting 7
Example on using BES keys
July 8 – 11, 2014
Float /CERES_Unfiltered_Radiances/all_toa_sw_rad[24][180][360]
Float CERES_Unfiltered_Radiances_all_toa_sw_rad[gmt_hour_index = 24][latitude =180][longitude= 360]
![Page 8: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/8.jpg)
www.hdfgroup.orgESIP Summer Meeting 8
Hyrax modulation
• Core framework implemented in C++• Just need to provide DDS, DAS and Data to Hyrax
libdap and BES• A handler can be sophisticated • HDF handler examples
• Retrieve latitude and longitude for HDF-EOS grid• Interpolate the latitude and longitude for HDF-EOS
swath with dimension map• Make sure the dimensions, variable names and
attributes follow CF conventions
July 8 – 11, 2014
![Page 9: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/9.jpg)
www.hdfgroup.orgESIP Summer Meeting 9July 8 – 11, 2014
Non-CF CF
Float CERES_Unfiltered_Radiances_all_toa_sw_rad[gmt_hour_index = 24][latitude =180][longitude= 360]
Float /CERES_Unfiltered_Radiances/all_toa_sw_rad[24][180][360]Float /CERES_Unfiltered_Radiances/all_toa_sw_rad[24][180][360]
![Page 10: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/10.jpg)
www.hdfgroup.orgESIP Summer Meeting 10
Demos – HDF-EOS2 (AIRS grid)
July 8 – 11, 2014
![Page 11: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/11.jpg)
www.hdfgroup.orgESIP Summer Meeting 11
Demo 2 – HDF-EOS2 (MODIS swath)
July 8 – 11, 2014
Swath with Dimension Map
![Page 12: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/12.jpg)
www.hdfgroup.orgESIP Summer Meeting 12
Demo 3 – HDF-EOS5 (OMI grid)
July 8 – 11, 2014
![Page 13: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/13.jpg)
www.hdfgroup.orgESIP Summer Meeting 13
Hyrax modules that help the easy access
• Filenet CDF module• Can download DAP output as a netCDF-3 or netCDF-
4 classic files
• NcML module• Add/delete/revise metadata information of the data
source• Aggregate granules
July 8 – 11, 2014
![Page 14: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/14.jpg)
www.hdfgroup.orgESIP Summer Meeting 14
Demo
July 8 – 11, 2014
• GPM level 1B(HDF5)
![Page 15: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/15.jpg)
www.hdfgroup.orgESIP Summer Meeting 15
HDF(5) and file netCDF demo
• FilenetCDF module can work with the HDF handlers to download the HDF files or subset of the files as netCDF-3 or netCDF-4 classic files that follow the CF conventions
July 8 – 11, 2014
![Page 16: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/16.jpg)
www.hdfgroup.orgESIP Summer Meeting 16
HDF(5) and NcML demo
• NcML module can be used with HDF handlers to provide the missing CF information
• AVHRR example
July 8 – 11, 2014
![Page 17: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/17.jpg)
www.hdfgroup.orgESIP Summer Meeting 17
HDF(5), file netCDF and NcML demo
• HDF handlers, NcML module and file netCDF module can work together
• AVHRR example
July 8 – 11, 2014
![Page 18: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/18.jpg)
www.hdfgroup.orgESIP Summer Meeting 18
Summary of modules to serve HDF files
July 8 – 11, 2014
Hyrax core
HDF(5) files
Main engineHDF(5) handlers
Supplement Info
NcML
DAP clients(IDV, Ferret, Panoply, IDL…...)
netCDFfiles
Alternative outputFile netCDF
![Page 19: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/19.jpg)
www.hdfgroup.orgESIP Summer Meeting 19
Demo - Beyond the web service
• BES standalone program • Connect the HDF handlers, the file netCDF module
and the NcML module• End users can build Hyrax and use this program as a
command-line tool
• Can it be a standalone tool?
July 8 – 11, 2014
besstandalone -c /etc/bes/bes.conf -i in4-GPM-l1b > GPM-l1b.nc4
![Page 20: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/20.jpg)
www.hdfgroup.orgESIP Summer Meeting 20
Future - Performance
• Improve the BES caching mechanism• Critical for large file access and aggregation service
July 8 – 11, 2014
![Page 21: Www.hdfgroup.org The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014](https://reader034.vdocuments.us/reader034/viewer/2022042613/551c2ad6550346b24f8b60a8/html5/thumbnails/21.jpg)
www.hdfgroup.org
The HDF Group
21ESIP Summer Meeting
Earth Science GroupTed Habermann
Aleksandar JelenakH. Joe Lee
Joel PlutchakJohn Readey
Kent Yang
July 8 – 11, 2014