esdis 2008 tasks status
DESCRIPTION
ESDIS 2008 tasks status. Kent Yang, Mike Folk The HDF Group March 31, 2009. Summary of tasks. Total 28 tasks: Most tasks received from the annual briefing and the workshop 4 NPP/NPOESS tasks transferred to NPOESS project 10 tasks have been finished 6 tasks are on-going activities - PowerPoint PPT PresentationTRANSCRIPT
04/20/23 1
ESDIS 2008 tasks status
Kent Yang, Mike Folk
The HDF Group
March 31, 2009
Annual briefing to ESDIS
04/20/23 2
Summary of tasks
• Total 28 tasks: Most tasks received from the annual briefing and
the workshop 4 NPP/NPOESS tasks transferred to NPOESS
project 10 tasks have been finished 6 tasks are on-going activities 8 tasks are in progress
Annual briefing to ESDIS
04/20/23 3
Finished tasks highlights
Annual briefing to ESDIS
HDF-EOS automatic configuration
• Both HDF-EOS2 and HDF-EOS5• Added the szip support
Both encoder and decoder Decoder only Without szip
• Added Fortran wrapper support• Updated source code under Fortran testdrivers
directory • Updated installation files for automatic
configuration
04/20/23 4Annual briefing to ESDIS
04/20/23 5
Investigation of HDF Support in Some Open
Source Software Packages
Annual briefing to ESDIS
04/20/23 6
Five open source packages
• PyHDF Python interface to HDF4 http://pysclint.sourceforge.net/pyhdf/
• Geospatial Data Abstraction Library (GDAL) Translator library for Raster Geospatial Data Formats Supports about 100 file formats http://gdal.org/
• NCAR Common Language (NCL) Interpreted Language for Data Analysis and Visualization http://ncl.ucar.edu/
• Grid Analysis and Display System (GrADS) Interpreted Language for Data Analysis and Visualization http://iges.org/grads/
• GNU Data Language (GDL) Interpreted Language for Data Analysis and Visualization Data Analysis and Visualization http://gnudatalanguage.sourceforge.net/
Annual briefing to ESDIS
04/20/23 7
Evaluation - File Formats
HDF4 HDF5
NCL SDSNetCDF4 classic
GrADS SDS
PyHDFSDS, Vdata, Vgroup
GDL SDS, VdataGeneral HDF5 objects
GDAL SDS, GR, EOS2General HDF5 objects
04/20/23 7Annual briefing to ESDIS
04/20/23 8
Evaluation - Documentation
Document Example
NCL sufficientMany examples with screenshots
GrADS sufficient Several screenshots
PyHDFsimple, but sufficient
Several simple examples
GDL IDL documents IDL documents
GDAL SufficientA few examples in TutorialWorking applications included
04/20/23 8Annual briefing to ESDIS
04/20/23 9
Evaluation - Installation
binaries source build
NCLAIX, IRIX, Linux, MacOS X, Solaris, Cygwin
not tested
GrADSLinux, MacOS X, SunOS
not tested
PyHDF Requires few settings
GDLRequires several packagesAssumes HDF4 and HDF5 without SZIP
GDAL Requires one patch
04/20/23 9Annual briefing to ESDIS
Usage example - GrADS
04/20/23 10
$ gradsga> xdfopen AE_RnGd.xdfga> set lon -180 180ga> set gxout shadedga> display RrLandRainga> draw title Total Rain Rate over Land in July 2007
Annual briefing to ESDIS
04/20/23 11
Performance evaluation of using SSHFS-FUSE to
access HDF5 files
Annual briefing to ESDIS
• NFS requires a special permission. General users are not allowed.
remote systemremote system local systemlocal system
Using NFS
04/20/23 12
NFSNFS
Annual briefing to ESDIS
• What if only a small part is necessary from a huge file?
local systemlocal system
1. Downloading a whole file
04/20/23 13
localfilesystem
localfilesystem
remote systemremote system
ftp, sftp, …
Annual briefing to ESDIS
• Given that permission to access FUSE is granted, general users can mount remote filesystems.
remote systemremote system local systemlocal system
2. Using FUSE+SSHFS
SSHFSFUSE
SSHFSFUSE
04/20/23 14Annual briefing to ESDIS
Accessing HDF5 through SSHFS-FUSE
• Experiment:• For 4 files,
Copy a whole file Copy one dataset in a file Copy one small hyperslab in a dataset
04/20/23 15Annual briefing to ESDIS
• elapsed time ratio
- SSHFS / downloading a whole file and subsetting
04/20/23 16
Results of the experiment
sshfs/download&local File 1
Whole file 3.42
One dataset 0.23
One hyperslab 0.10
SSHFSconsumedless time
SSHFSconsumedless time
SSHFSconsumedmore time
SSHFSconsumedmore time
Annual briefing to ESDIS
Other finished tasks
• The third dimension support of HDF-EOS The documentation of HDF-EOS needed to be
updated
• HDF4.2r4 release Static library with static external libraries C shared libraries
• HDFView 2.5 release Fixed bugs of ‘switch base index’ and ‘fill value’
• Help build EOS toolkit on solaris• NetCDF4/NetCDF3/HDF5 performance study
http://www.hdfgroup.org/pubs/papers/2008-06_netcdf4_perf_report.pdf
04/20/23 17Annual briefing to ESDIS
04/20/23 18
On-going tasks overview
Annual briefing to ESDIS
On-going tasks overview
• Applications/other technologies(in separate talks) Support OPeNDAP HDF4 handler for OPeNDAP Independent HDF4 mapping
• HDF5 library Support backward/forward compatibility
• Tests with other software packages Maintain HDF-EOS2 and HDF-EOS5 daily tests Regression tests with IDL and Matlab
04/20/23 19Annual briefing to ESDIS
04/20/23 20
Tasks in progress• Some tasks almost finished• Some tasks we need to know the next step• Some tasks are re-scoped into our next year’s plan
Annual briefing to ESDIS
Special value RFC
• 90% done. We have written the RFC We then sent out to NASA users and hdf-forum A decision has been made based on feedback
o Provide examples to demonstrate how to handle special values
04/20/23 21
Hi Kent,
Thank you for your message. I appreciate your taking the time to
communicate with us.
I appreciate even more that your decision was guided by the needs,
concerns, and interests expressed by hdf5 data producers.
- Paul
Annual briefing to ESDIS
High Level Language Support
• Motivation A better high-level language support for HDF5
such as Python, Perl, Ruby etc.
• Progress Investigated SWIG-HDF5 interface Held a discussion in HDF-EOS workshop Formed a High level language wiki to further
discuss the general issuehttp://wiki.hdfgroup.org/High+level+language+supports+for+HDF5
Suggestions ?
04/20/23 22Annual briefing to ESDIS
Simplifying installation procedures
• Request Investigate simplifying build installation procedures,
including 3rd party libraries
• Progress The HDF group is investigating the use of RPM to
simplify the whole process A RFC is being written
04/20/23 23Annual briefing to ESDIS
Using NetCDF4 APIs to read HDF-EOS5
• Motivation: To allow the access of HDF-EOS5 files via
NetCDF4 APIs To ensure that HDF-EOS5 files still accessed by
the HDF-EOS5 library
04/20/23 24Annual briefing to ESDIS
Using NetCDF4 APIs to read HDF-EOS5
• Progress: Wrote an investigation report on this issue Discussed with Aura, netcdf and EOS developers Finalized a solution
o Augmented the EOS5 file to support dimensionso Relaxed creation order requirements of NetCDF4 to
access HDF-EOS5 data A prototyped augmentation tool is implemented
• Work to be done: Wait for NetCDF4 developer to release a snapshot to
support the relax of creation order Write a netCDF4 fortran example to read EOS5 files
04/20/23 25Annual briefing to ESDIS
HDF5 to jpeg tool prototype
• Request from HDF-EOS5 to support the conversion in HDF-EOS2 to HDF-EOS5 tool
• h52jpeg traverses an HDF5 file and saves each HDF5 image found to a jpeg file
• h52jpeg is only a prototype Not fully tested Not fully documented Not released with HDF5 library Source code is available only at a special branch
http://svn.hdfgroup.uiuc.edu/hdf5/branches/jpeg_converter/hl/tools/h52jpeg/
• Is there any request to make this tool fully supported?
04/20/23 26Annual briefing to ESDIS
Command line format conversion tools
• Request: In order of priority - netCDF, geotiff, grib and ascii
• Progress: This is a big task. We sent out a newsletter to ask if there are any
available tools; no useful responses received. We were implementing an HDF-EOS5 to netCDF4
conversion tool.
• Questions: Is this task still necessary? If yes, what’s the priority of this task?
04/20/23 27Annual briefing to ESDIS
EOS access example codes and tools
• Requests: Tools such as HEG, IDL , Matlab etc. Well-commented example codes
• Progress: Some experimental work has been done
• Plan: The work is re-scoped to be the focus of the new
year
04/20/23 28Annual briefing to ESDIS
Review dimension scale model
• Requests: Review dimension scale model and identify use
cases
• Plan: We will start investigating this after the briefing
• Any suggestions?
04/20/23 29Annual briefing to ESDIS
Thank You!
04/20/23 30Annual briefing to ESDIS
Acknowledgements
This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX08A077A.
Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.
04/20/23 31Annual briefing to ESDIS
Questions/comments?
04/20/23 32Annual briefing to ESDIS