hdf5 tools
DESCRIPTION
This tutorial is designed for anyone who needs to work with data stored in HDF5 files. It will cover functionality and useful features of the HDF5 utilities, which include h5dump, h5diff, h5repack, h5stat, h5copy, h5check and h5repart. The tutorial will also introduce recently changes and new features of the utilities. The HDFView is a visual tool for browsing and editing HDF4 and HDF5 files. Some basic features and new changes of HDFView will be presented. Details of recent development in HDF-Java products will be discussed in a separate presentation.TRANSCRIPT
www.hdfgroup.org
The HDF Group
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 1
HDF5 Tools
Peter Cao The HDF Group The 13th HDF and HDF-EOS Workshop November 3-5, 2009
www.hdfgroup.org
Outline
2
• An introduction to HDFView
• HDF5 command line tools
November 3-5, 2009 HDF/HDF-EOS Workshop XIII
www.hdfgroup.org
What is HDFView?
3
For more information and download, visitwww.hdfgroup.org/hdf-java-html
HDFView
HDF Application Browser/Editor Written in Java
November 3-5, 2009 HDF/HDF-EOS Workshop XIII
www.hdfgroup.org
HDFView Demo
• Browse objects in an HDF5 file
• Show file content
• Change file content
November 3-5, 2009 4HDF/HDF-EOS Workshop XIII
Advanced features and new development such as plugin, region references, full support of hdf5 1.8 will be in a separate talk.
HDFView is for general use. For specific purpose or more advanced features, you use other tools like MATLAB or IDL, or other earth science related tools such as HEG, NCL, or GrADS. At the same time, welcome ideas for features.
www.hdfgroup.org
HDFView 2.6
• Release planned in December, 2009
• Build with HDF5 1.8.4
• Fix bugs
• Add new features
• Add support for 64-bit mac.
November 3-5, 2009 5HDF/HDF-EOS Workshop XIII
Drop mac PowerPC
www.hdfgroup.org
HDF5 Command Line Tools
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 6
• Look or check file contenth5dump, h5diff, h5ls, h5check, etc.
• Change file content or create new fileh5repack, h5import, h5jam, h5copy, etc.
• Convert file contenth4toh5, h5toh4, gif2h5, h52gif, etc.
Only show you the basic features of some tools
www.hdfgroup.org
Where to Start?
November 3-5, 2009 7
1. Use “-h” option for help
2. Consult online help http://www.hdfgroup.org/hdf5tools.html
3. Email help desk [email protected]
HDF/HDF-EOS Workshop XIII
Tools are distributed with the library release and located under bin/
www.hdfgroup.org
h5dump
November 3-5, 2009 8
• ASCII text• XML • Binary
HDF/HDF-EOS Workshop XIII
Dumps contents of HDF5 file to stdout
www.hdfgroup.org
HDF5 "SDS.h5" { GROUP "/" { GROUP "Floats" { DATASET "FloatArray" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } } } DATASET "IntArray" { DATATYPE H5T_STD_I32LE DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) } } }}
h5dump: Object Headers
November 3-5, 2009 9HDF/HDF-EOS Workshop XIII
h5dump -H SDS.h5What’s in the file?
www.hdfgroup.org
h5dump: Dataset
November 3-5, 2009 10
HDF5 "SDS.h5" {
DATASET "/Floats/FloatArray" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) }
DATA {
(0,0): 0.01, 0.02, 0.03,
(1,0): 0.1, 0.2, 0.3,
(2,0): 1, 2, 3,
(3,0): 10, 20, 30
}}
}
h5dump -d /Floats/FloatArray SDS.h5
HDF/HDF-EOS Workshop XIII
What’s the content?
www.hdfgroup.org
h5diff
November 3-5, 2009 11
Showdifferencesbetween…
two objects or
two filesHDF/HDF-EOS Workshop XIII
www.hdfgroup.org
h5diff
November 3-5, 2009 12
Dataset: </IntArray>5 differences found
h5diff SDS.h5 SDS2.h5
HDF/HDF-EOS Workshop XIII
Any change in the data process?
www.hdfgroup.org
h5diff: Report Differences
Dataset: </IntArray> and </IntArray>position IntArray IntArray difference
------------------------------------------
[ 0 0 ] 0 10 10
[ 1 0 ] 10 100 90
[ 2 0 ] 20 200 180
[ 3 0 ] 30 300 270
[ 4 0 ] 40 400 360
5 differences found
h5diff -r SDS.h5 SDS2.h5 /IntArray
November 3-5, 2009 13HDF/HDF-EOS Workshop XIII
What are the changes?
www.hdfgroup.org
h5diff: Unsolved Issues
How to compare two files with different
• Structures
• Storage layouts
• Other properties
November 3-5, 2009 14HDF/HDF-EOS Workshop XIII
www.hdfgroup.org
h5repack
November 3-5, 2009 15HDF/HDF-EOS Workshop XIII
Copiesan entire file toa new file…
www.hdfgroup.org
h5repack: Change Data Layout
Size of a 2D byte array of 1024x1024?
Why is my file so big?
3MB
November 3-5, 2009 16HDF/HDF-EOS Workshop XIII
www.hdfgroup.org
h5repack: Change Data Layout
h5repack -i huge.h5 -o good.h5 -l CHUNK=64x1024
November 3-5, 2009 17HDF/HDF-EOS Workshop XIII
1MBSize of good.h5?
www.hdfgroup.org
h5repack: Compression Filter
Most of my data values are zeros and ones
Can I make it better?
www.hdfgroup.org
h5repack: Compression Filter
h5repack -i good.h5 -o better.h5 -f GZIP=6
November 3-5, 2009 19HDF/HDF-EOS Workshop XIII
6KBSize of better.h5?
Down from 3MB to 6KB, 500 times smaller!!!!
www.hdfgroup.org
h5import
November 3-5, 2009 20HDF/HDF-EOS Workshop XIII
What should I do?
1 GB
1 MB
www.hdfgroup.org
h5import
PATH work/First-set INPUT-CLASS TEXTFP RANK 3 DIMENSION-SIZES 5 2 4 OUTPUT-CLASS FP OUTPUT-SIZE 64 OUTPUT-ARCHITECTURE IEEE OUTPUT-BYTE-ORDER LE CHUNKED-DIMENSION-SIZES 2 2 2 MAXIMUM-DIMENSIONS 8 8 -1
GROUP "/" { GROUP "work" { DATASET "First-set" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H5S_UNLIMITED ) } DATA { (0,0,0): 1.01, 1.02, 1.03, 1.04, (0,1,0): 1.11, 1.12, 1.13, 1.14, (1,0,0): 1.21, 1.22, 1.23, 1.24, (1,1,0): 1.31, 1.32, 1.33, 1.34, (2,0,0): 1.41, 1.42, 1.43, 1.44, … } } }}}
h5import float5x4x2.txt -c First_set.conf -o First_set.h5
November 3-5, 2009 21HDF/HDF-EOS Workshop XIII
1.01, 1.02, 1.03, 1.041.11, 1.12, 1.13, 1.141.21, 1.22, 1.23, 1.241.31, 1.32, 1.33, 1.341.41, 1.42, 1.43, 1.44Use external datasets
www.hdfgroup.org
Only available in1.8, not in 1.6
h5stath5mkgrp
h5copy
h5check
November 3-5, 2009 22HDF/HDF-EOS Workshop XIII
www.hdfgroup.org
h5copy
November 3-5, 2009 23HDF/HDF-EOS Workshop XIII
Copiesobjectsto files…
www.hdfgroup.org
h5copy: dataset
/
i1
floatsintegers
64-biti2f32
f2f1
/
November 3-5, 2009 24HDF/HDF-EOS Workshop XIII
h5copy –i dset.h5 –s /floats/f32 –o dset2.h5 –d /f32
dset.h5 dset2.h5
f32
www.hdfgroup.org
H5copy: group
/
i1
floatsintegers
64-biti2 f32
f2f1
/
floats
64-bitf32
f2f1
November 3-5, 2009 25HDF/HDF-EOS Workshop XIII
h5copy –i dset.h5 –s /floats –o dset2.h5 –d /floats
dset.h5 dset2.h5
www.hdfgroup.org
h5mkgrp
/
info data
test1 test2
h5mkgrp test.h5 /info /data /data/test1 /data/test2
November 3-5, 2009 26HDF/HDF-EOS Workshop XIII
dset.h5
www.hdfgroup.org
h5stat
Prints statistics about HDF5 file• Header size• Raw data size• Attributes, datatypes, etc.
Helps • Troubleshoot overhead in HDF5 files• Choose specific object’s properties and storage strategies
27November 3-5, 2009 HDF/HDF-EOS Workshop XIII
www.hdfgroup.org
h5stat: Example
Filename: hdf5_test.h5
File information
# of unique groups: 4
# of unique datasets: 13
…
Object header size: (total/unused)
Groups: 288/32
Datasets: 6824/864
…
Dataset storage information:
Total raw data size: 1769009
…
November 3-5, 2009 28HDF/HDF-EOS Workshop XIII
>h5stat hdf5_test.h5
www.hdfgroup.org
Acknowledgements
This work was supported by cooperative agreement number NNX08AO77A from the National
Aeronautics and Space Administration (NASA).
Any opinions, findings, conclusions, or recommendations expressed in this material are
those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space
Administration.
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 29
www.hdfgroup.org
The HDF Group
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 30
Thank You!
Questions/comments?