HDF5 Q4 Demo

Architecture

Friday, May 10, 2013 Friday Seminar 2

Full VOL Plugin Support

• Skeletal IOD– Artificial Data Read at server

• New Features/Functionality:– Attributes (H5A*)– Links (H5L*)– Objects (H5O*)

• Datatype Conversion:– Done at server, avoids extra buffer allocation at client– Examples (BE to LE, 16 to 32 bit, etc…)

• Non-contiguous memory selections:– Hyperslabs, point selections– Now supported with Mercury non-contiguous transfers

Asynchronous Execution• Nearly all HDF5 API routines are completely asynchronous now

– Dependencies built at the client, executed at the server– No wait at client for previous operations to complete to be able to forward a

child operation• Few API routines still cause partly asynchronous behavior

– Child operations need to wait for them to complete at the client.• A small number of routines have to be synchronous:

– Iterate/visit– H5Oopen

• Using event queue (EQ) objects now to manage requests instead of individual requests:– Very cumbersome to manage a request for every operation issued.– Event Queue can be though of as “bags” to wait/test a bunch of requests.– There is no effect on dependencies or influence on execution order for

requests in an EQ.– User can still pop off the last inserted request in EQ to manage it separately.

Unimplemented Functionality

• Deferred– Iterate/visit (Does not make sense without a storage backend –

Too much work to generate artificial metadata for them)– A few metadata query (H5*get_*) routines (same reason)– Variable Length Datatypes

• Probably or Definitely will not support:– External Links - probably– References (H5R*) – probably– H5*_by_index operations – probably– File Mounting – definitely– H5Oopen_by_addr – definitely– Some Query operations (H5Dget_offset) – definitely

Testing Status• Testing is improved:

– Verifying new VOL routines are shipped and call the correct IOD routines at server.

– Regression testing framework is setup (Cmake only now, autotools later).– Need to beef up regression testing, but won’t be able to do much until

we get something for a storage backend.• Temporary Native HDF5 backend:

– Call in to the native HDF5 storage format to verify routines and their parameters are shipped from the client and executed correctly at the server.

– Very limited:• 1 client & 1 server possible for this now• Not asynchronous

Data Integrity

• Support in our level of the stack.• Added a public routine for applications to

generate checksums: H5checksum().• Added properties and routines to set

properties for data integrity:• herr_t H5Pset_dxpl_checksum(hid_t dxpl_id, uint32_t

value);• herr_t H5Pset_dxpl_checksum_ptr(hid_t dxpl_id,

uint32_t *value);• herr_t H5Pset_dxpl_inject_corruption(hid_t dxpl_id,

hbool_t flag);

ACG Dynamic Data Structures

• Added 4 new routines and their FF version:

• herr_t H5DOappend(hid_t dataset_id, hid_t dxpl_id, unsigned axis, size_t extension, hid_t memtype, const void *buffer);

• herr_t H5DOsequence(hid_t dataset_id, hid_t dxpl_id, unsigned axis, hsize_t start, size_t sequence, hid_t memtype, void *buffer);

• herr_t H5DOset(hid_t dataset_id, hid_t dxpl_id, const hsize_t coord[], hid_t memtype, const void *buffer);

• herr_t H5DOget(hid_t dataset_id, hid_t dxpl_id, const hsize_t coord[],hid_t memtype, void *buffer);

H5DOappend()3x5 Dataset

1. H5DOappend(did, dxpl, 0, 3 , memtype, buf);2. H5DOappend(did, dxpl, 1, 2 , memtype, buf2);

After 1: 6x5 Dataset After 2: 6x7 Dataset

• H5DOsequence is very similar:• Read instead of write (no extension).• Accepts a starting position to read from.

ACG support for Next Quarter

• Support for VL Datatypes:– Each array element in the dataset corresponds to an IOD

BLOB object• Add an Append only hint to the Dataset Creation

property list:– Optimizes access and storage– Only 1 BLOB object needed– Each array element stores an offset into the BLOB object

• Map Objects:– KV stores at the HDF5 API level.