hdf5: state of the union · cool recent application - imageworks’ field3d november 19, 2009 hdf5...

www.hdfgroup.org

The HDF Group

HDF5: State of the Union Quincey Koziol, Ruth Aydt {koziol,aydt}@hdfgroup.org

The HDF Group SC09 – HDF5 BOF November 19, 2009

November 19, 2009 HDF5 BOF - SC09 1

www.hdfgroup.org

HDF5 Technology Platform

•  HDF5 Abstract Data Model •  Defines the “building blocks” for data organization and

specification •  Files, Groups, Links, Datasets, Attributes, Datatypes,

Dataspaces

•  HDF5 Software •  Tools •  Language Interfaces •  HDF5 Library

•  HDF5 Binary File Format •  Bit-level organization of HDF5 file •  Defined by HDF5 File Format Specification

2 November 19, 2009 HDF5 BOF - SC09

www.hdfgroup.org

HDF5 API and Applications

3

Virtual File Layer

POSIX I/O

Split Files MPI I/O Custom

Internals Dataspace Selection

Datatype Conversion Filters Chunked

Storage Version

Compatibility and so on…

Language Interfaces

HDF5 Data Model Objects

Groups, Datasets, Attributes, … Tunable Properties Creation, Access, Transfer, ..

HD

F5 L

ibra

ry

…

…

HDF5 API

Storage

Domain Data Objects

EOS library

Applications YourBioApp aClimate Model

myApp

netCDF4 library

Sony Pict Field3d

BioHDF library

MATLAB©

November 19, 2009 HDF5 BOF - SC09

www.hdfgroup.org

Data challenges addressed by HDF

•  Our ability to organize complex collections of data

•  Efficient and scalable data storage and access •  A growing need to integrate a wide variety of

types of data •  The evolution of data technologies •  Long term preservation of data


www.hdfgroup.org

Areas of increased recent interest

•  Life sciences •  Gene sequencing •  Biomedical imaging

•  High performance computing (HPC) •  Microsoft products (HPC, .NET, others) •  Database integration •  Improvements

•  Concurrent access •  Improving parallel I/O performance •  Improving real-time write performance •  Improving high level language support

November 19, 2009 5 HDF5 BOF - SC09

www.hdfgroup.org

COMMUNICATIONS OF THE ACM – October 2009 November 19, 2009 HDF5 BOF - SC09 6

www.hdfgroup.org

Cool recent application -

Imageworks’ Field3D


Spiderman 3 The Polar Express

www.hdfgroup.org

Topics


What's up with The HDF Group?

Library Update

Tools update

HDF Java Products

Library development in the works

Other activities

www.hdfgroup.org

The HDF Group

•  Established in 1988 •  18 years at University of Illinois National Center for

Supercomputing Applications •  4 years as independent non-profit company, “The

HDF Group” •  The HDF Group owns HDF4 and HDF5

•  Basic HDF4 and HDF5 formats, libraries, and tools are open and free

•  Currently employ 25 FTEs


www.hdfgroup.org

The HDF Group

The HDF Group Mission

To ensure long-term accessibility of HDF data through sustainable development and support of HDF

technologies.


www.hdfgroup.org

Goals of The HDF Group

•  Maintain and evolve HDF for sponsors and communities that depend on it

•  Provide support to the HDF communities through consulting, training, tuning, development, research

•  Sustain the company for the long term to assure data access over time


www.hdfgroup.org

The HDF Group Services

•  Helpdesk and Mailing Lists •  Available to all users as a first level of support

•  Priority Support •  Rapid issue resolution and advice

•  Consulting •  Needs assessment, troubleshooting, design reviews, etc.

•  Training •  Tutorials and hands-on practical experience

•  Enterprise Support •  Coordinating HDF activities across departments

•  Special Projects •  Adapting customer applications to HDF •  New features and tools •  Research and Development


www.hdfgroup.org

Income Profile – past 12 months

13

Total income approximately $3.4 million

November 19, 2009 HDF5 BOF - SC09

www.hdfgroup.org

Topics



Library Update

Tools Update

HDF Java Products


Other activities

www.hdfgroup.org

HDF5 1.8.3 minor release (May 09)

•  New functions •  Improve flexibility when traversing external links •  Validate object identifier

•  Enabled data chunk cache properties to be set per dataset (per file in previous releases)

•  Forward/backward compatibility issues •  Modified library to be able to open files with

corrupt root group symbol table messages •  Also corrects corruption errors if found.


www.hdfgroup.org

HDF5 1.8.4 minor release (Nov 09)

•  Modified configure and make process to properly preserve user's CFLAGS and similar environment variables.

•  Corrected a problem where library would re-write the superblock in a file opened for R/W access, even when no changes were made to the file.


www.hdfgroup.org

HDF5 1.6 minor releases

•  1.6.9 May 09 •  Minor bug fixes •  Same tools improvements as in 1.8.3

•  1.6.10 Nov 09 •  Minor bug fixes •  Ability to embed library information in executable

binaries •  This is a last release of 1.6 series

•  announced in May 2009 – no response •  This is your last chance!


www.hdfgroup.org

Topics



Library Update

Tools update

HDF Java Products


Other activities

www.hdfgroup.org

Major Improvements for Existing Tools

•  H5dump additions •  Ability to show data pointed to by dataset region references. •  More options for dumping data into ASCII

•  Compatible with MS Excel •  Compatible with h5import

•  h5diff •  Some new flags

•  Report non-comparable objects •  Avoid NaN detection •  Option to use system epsilon to compare floating-point numbers

•  Treats two INFINITY values as equal •  Compares for strict equality first to improve performance •  Fixed segmentation fault problem on variable length strings.


www.hdfgroup.org

Major Improvements for Existing Tools

•  h5stat •  Fixed incorrect statistics on EOS big data files

with corrupted headers. •  h5repack

•  Added ability to preserve group creation order •  Fixed problem that 1.8 fails on a file created

with 1.6. •  When chunk size not specified, uses

heuristics to set chunk size


www.hdfgroup.org

Tool activities in the works

•  New tool -- h5tail •  Display new records appended to a dataset

•  Improved code quality and testing •  Tools library: general purpose APIs for tools

•  Tools library currently only for our developers •  Want to make it public so that people can use it in

their products


www.hdfgroup.org

Conversion Tools


•  HDF4 to HDF5 •  HDF5 to jpeg •  HDF5 to XML •  HDF5 to other formats?

Please send us your comments and requests regarding HDF5 conversion tools, such as

www.hdfgroup.org

Topics



Library Update

Tools update

HDF Java Products


Other activities

www.hdfgroup.org

HDF-Java 2.6 is on the way

•  Includes all HDF java products •  Java Wrapper API •  Java Object API •  HDFView

•  Adds new features, such as dataset region reference

•  Improves performance •  Release schedule

•  Beta 1: end of Nov. 09 •  Full release: end of Dec. 09


www.hdfgroup.org

Full support of HDF5 1.8.x in hdf-java

•  Full HDF5 1.8 support will be added to the release after version 2.6.

•  We are looking for input •  RFC:

http://www.hdfgroup.uiuc.edu/RFC/HDF5/hdf-java/ •  Java wrapper will be completed March 2010 •  Object API and HDFView update to come later


www.hdfgroup.org

Topics



Library Update

Tools update

HDF Java Products


Other activities

www.hdfgroup.org

Single-Writer/Multiple-Reader Access

•  Situation: A long-running process is modifying an HDF5 file and simultaneously other processes want to inspect data in the file.

•  Solution: Single-Writer/Multiple-Reader (SWMR) File Access. •  Allows simultaneous reading of HDF5 file while

the file is being modified by another process •  No inter-process coordination necessary


www.hdfgroup.org November 19, 2009 HDF5 BOF - SC09 28 28

Surviving a System Failure in HDF5

•  Problem: •  In the event of an application or system crash,

data in HDF5 files are susceptible to corruption •  Corruption can occur if structural metadata is

being written when the crash occurs

•  Initial Objective: •  Guarantee an HDF5 file with consistent

metadata can be reconstructed in the event of a crash

www.hdfgroup.org November 19, 2009 HDF5 BOF - SC09 29 29

Crash Survivability in HDF5

•  Approach: Metadata Journaling •  When an HDF5 file is opened with Metadata, a

companion journal file is created •  When an HDF5 function modifies metadata,

this modification is recorded in the journal file •  If the application crashes, a recovery program

can replay the journal by applying all metadata writes, ensuring that all metadata in the file is correct

www.hdfgroup.org

Metadata Journaling: Progress

•  Feature complete (but only works w/serial I/O) •  Beta released August 2009 •  Adding support for asynchronous I/O of journal

writes – Faster! •  Further development may support parallel I/O •  To be included in

1.10.0 release


www.hdfgroup.org

Improved Multi-Threaded Concurrency

•  Converting from “big lock” on code (entire library) to locks on internal library data structures

•  Will improve ability to have multiple threads performing HDF5 operations simultaneously

•  Working with Argonne MPICH team on “OpenPA” project - http://trac.mcs.anl.gov/projects/openpa


www.hdfgroup.org

Other Library Features

•  Saving space •  New Chunk Indexing Methods •  Store Partial Edge Chunks More Efficiently •  Persistent File Free Space tracking/recovery •  Allow a group’s link info to be compressed

•  Saving time •  Aggregate neighboring metadata for faster

metadata cache I/O


www.hdfgroup.org

New chunk indexing methods

Dataset type Index type Space improvements

Speed improvements

no unlimited dimensions, no I/O filters, no missing chunks

“implicit” no actual

chunk index

Same storage space as

contiguous dataset storage (no index)

Constant time lookups

Faster parallel I/O

no unlimited dimensions

“fixed sized” smaller chunk

index

Smaller index overhead

Constant time lookups

1 unlimited dimension

“extensible array”


Constant time lookups and

appends 2+ unlimited dimension

Improved B-tree*


Faster


www.hdfgroup.org November 19, 2009 HDF5 BOF - SC09 34

•  Project with NERSC to improve HDF5 performance on parallel applications

•  Up to 6x performance improvements on certain applications (so far)

Parallel I/O Improvements


•  Current work: •  Restructured library code to number of file

truncation operations •  Restructured library code to detect same “shape”

of selection in more cases, allowing optimized I/O path to be taken more often

•  Future work: •  Add high-level “tune for Lustre file system” API

call(s) •  Improvements to MPI-IO and MPI-POSIX VFDs

and library algorithms for faster/better use of MPI

Parallel I/O Improvements

www.hdfgroup.org

Topics



Library Update

Tools update

HDF Java Products


Other activities

www.hdfgroup.org

BIOHDF : TOWARD SCALABLE BIOINFORMATICS INFRASTRUCTURES

NIH STTR with Geospiza, Seattle WA

TM


www.hdfgroup.org

Next Generation DNA Sequencing

“Genome center in a mail room”

“Democratizing genomics”

“Changing the landscape”

“The beginning of the end for microarrays”

“Transforms today’s biology”


www.hdfgroup.org

BioHDF Project

•  Goal: Move bioinformatics problems from organizing and structuring data to asking questions and visualizing data •  Develop data models and tools to work with NGS data in HDF5 •  Create HDF5 domain-specific extensions and library modules to

support the unique aspects of NGS data BioHDF •  Integrate BioHDF technologies into Geospiza products

•  Deliver core BioHDF technologies to the community as open-source software


www.hdfgroup.org

The HDF Group

Performance evaluation of using SSHFS-FUSE to access

HDF5 files


www.hdfgroup.org

Using NFS

•  However, NFS requires the system admin. to mount the remote file system.

November 19, 2009

HDF5 BOF - SC09 41

remote system local system

NFS

www.hdfgroup.org

1. Downloading a whole file

•  What if only a small part is necessary from a huge file?

November 19, 2009

HDF5 BOF - SC09 42

local system

local filesystem

remote system

ftp, sftp, …

www.hdfgroup.org

2. Using FUSE+SSHFS

•  If permission is granted to access FUSE, general users can mount remote filesystems.

November 19, 2009

HDF5 BOF - SC09 43

remote system local system

SSHFS FUSE

www.hdfgroup.org

Results of experiment


sshfs/download&local File 1

Whole file 3.42

One dataset 0.23

One hyperslab 0.10

SSHFS consumed less time

SSHFS consumed more time

•  Elapsed time ratio •  - SSHFS / downloading a whole file and subsetting

www.hdfgroup.org

TRANSFORMING THE GEOCOMPUTATIONAL BATTLESPACE FRAMEWORK WITH HDF5

A Project with the Army Geospatial Center


www.hdfgroup.org

Data Challenges


Wide variety

Satellite Buckeye Culture

Large scale High efficiency

High res. Stream Accuracy Time

Military Decision Making Process


Concept Map : General HDFView


Plug-in: Concept Map

www.hdfgroup.org

The HDF Group

HDF-EOS library


www.hdfgroup.org

EOS support

•  HDF-EOS2 and HDF-EOS5 •  Automatic configuration with szip enabled/

disabled •  Now tested daily with HDF4 and HDF5

development code •  Updated the HDF-EOS website


www.hdfgroup.org

The HDF Group

HDF-EOS5/netCDF-4 Augmentation Tool

Accessing HDF-EOS5 files via netCDF-4 API


www.hdfgroup.org

The Main Challenge

•  NetCDF-4 model follows the HDF5 dimension scale model but HDF-EOS5 does not.

November 19, 2009

HDF5 BOF - SC09 52

GRIDS HDFEOS

CloudFractionAndPressure Data Fields

CloudFraction CloudPressure

No HDF5 dimension scales are associated with this variable

www.hdfgroup.org

Our Solution – Augmentation

•  Provide dimensions required by netCDF-4

November 19, 2009

HDF5 BOF - SC09 53

GRIDS HDFEOS

CloudFractionAndPressure Data Fields

CloudFraction[XDim][YDim] CloudPressure[XDim][YDim]

XDim

YDim

www.hdfgroup.org

Special values in HDF5

•  There are cases where a user may wish to specify more than one “special” value to describe non-standard data.

•  We provide several examples (C, Fortran, IDL) on how to store special values: •  http://www.hdfgroup.org/pubs/rfcs/


www.hdfgroup.org

The HDF Group

OPeNDAP


www.hdfgroup.org

OPeNDAP

•  HDF5-OPeNDAP handler •  Served OMI Swath data

•  HDF4-OPeNDAP handler •  Tested with some AIRS data and some MODIS

data


www.hdfgroup.org

Swath to Grid conversion Tool


•  Request from NASA GES DISC •  Convert Swath to Grid •  Support both HDF-EOS2 and TRMM data •  Still in the development

MODIS Swath

Converted Grid

www.hdfgroup.org

The HDF Group

Support for NPP/NPOESS by

The HDF Group


www.hdfgroup.org

Priorities for 2008-2009

•  Data accessibility and usability •  Developed library of high level APIs to support

NPP/NPOESS data management •  Modified h5dump to display region references •  Modified HDFView to view object and region

references and quality flags •  System maintenance •  User support


www.hdfgroup.org

HDF4 LAYOUT MAPS


www.hdfgroup.org

HDF4 Layout Map Project

•  Problem •  Long-term readability of HDF data depends

on long-term availability of software •  Proposed solution

•  Create a map of the layout of data objects in an HDF file, allowing a simple reader to be written to access the data


www.hdfgroup.org

Mapping a chunked SDS


HDF4 file layout

www.hdfgroup.org

The HDF Group

Thank You!

Questions & Comments?


hdf5: state of the union · cool recent application - imageworks’ field3d november 19, 2009 hdf5...

Documents