portable parallel i/o...datatype deﬁnitions can be shared among objects in an hdf ﬁle, providing...

Mitg

lied

derH

elm

holtz

-Gem

eins

chaf

t

Portable Parallel I/OHandling large datasets inheterogeneous parallel environments

May 21, 2014 Michael Stephan

Mitg

lied

derH

elm

holtz

-Gem

eins

chaf

t

Portable Parallel I/OPart I: HDF5

May 21, 2014 Michael Stephan

Learning Objectives

At the end of this lesson, you will be able toGet an idea about the HDF5 funktionality.Create a short example for HDF5 file I/O.Discuss the advantages and disadvantages of HDF5 fileI/O.UsersGuide (352 pages)ReferenceGuide (802 pages)

May 21, 2014 Michael Stephan Slide 3

Outline

Introduction

Introduction to HDF5 Programming Model and APIs

General Programming Paradigm

Parallele HDF5


Outline

IntroductionMotivationTerms and Definitions



Parallele HDF5


What is HDF5? I

Unique technology suite that makes possible themanagement of extremely large and complex datacollections


What is HDF5? IIThe HDF5 technology suite includes:

A versatile data model that can represent very complex dataobjects and a wide variety of metadata.A completely portable file format with no limit on the numberor size of data objects in the collection.A software library that runs on a range of computationalplatforms, from laptops to massively parallel systems, andimplements a high-level API with C, C++, Fortran 90, andJava interfaces.A rich set of integrated performance features that allow foraccess time and storage space optimizations.Tools and applications for managing, manipulating, viewing,and analyzing the data in the collection.The HDF5 data model, file format, API, library, and tools areopen and distributed without charge.


What is HDF5? IIIUnlimited size, extensibility, and portability

HDF5 does not limit the size of files or the size or number ofobjects in a file.The HDF5 format and library are extensible and designed toevolve gracefully to satisfy new demands.HDF5 functionality and data is portable across virtually allcomputing platforms and is distributed with C, C++, Java, andFortran90 programming interfaces.


What is HDF5? IVGeneral data model

HDF5 has a simple but versatile data model.The HDF5 data model supports complex data relationshipsand dependencies through its grouping and linkingmechanisms.HDF5 accommodates many common types of metadata andarbitrary user-defined metadata.


What is HDF5? VUnlimited variety of datatypes

HDF5 supports a rich set of pre-defined datatypes as well asthe creation of an unlimited variety of complex user-defineddatatypes.Datatype definitions can be shared among objects in an HDFfile, providing a powerful and efficient mechanism fordescribing data.Datatype definitions include information such as byte order(endian), size, and floating point representation, to fullydescribe how the data is stored, insuring portability to otherplatforms.


What is HDF5? VIFlexible, efficient I/O

HDF5, through its virtual file layer, offers extremely flexiblestorage and data transfer capabilities. Standard (Posix),Parallel, and Network I/O file drivers are provided with HDF5.Application developers can write additional file drivers toimplement customized data storage or transport capabilities.The parallel I/O driver for HDF5 reduces access times onparallel systems by reading/writing multiple data streamssimultaneously.


What is HDF5? VIIFlexible data storage

HDF5 employs various compression, extensibility, andchunking strategies to improve access, management, andstorage efficiency.HDF5 provides for external storage of raw data, allowing rawdata to be shared among HDF5 files and/or applications, andoften saving disk space.


What is HDF5? VIIIData transformation and complex subsetting

HDF5 enables datatype and spatial transformation during I/Ooperations.HDF5 data I/O functions can operate on selected subsets ofthe data, reducing transferred data volume and improvingaccess speed.


Who uses HDF5?

Applications that deal with big or complex dataOver 200 different types of apps2+ million product users world-wideAcademia, government agencies, industry


Outline

IntroductionMotivationTerms and Definitions



Parallele HDF5


An HDF5 “file” is a container...

...into which you can put your data objects Structures toorganize objects


HDF5 model

Groups – provide structure among objectsDatasets – where the primary data goes

Data arraysRich set of datatype optionsFlexible, efficient storage and I/O

Attributes, for metadataOther objects

Links (point to data in a file or in another HDF5 file)Datatypes (can be stored for complex structures and reusedby multiple datatsets)


HDF5 Dataset


HDF5 Dataspace

Two rolesDataspace contains spatial info about a dataset stored in afile

Rank and dimensionsPermanent part of dataset definition

Dataspace describes application’s data buffer and dataelements participating in I/O


HDF5 Datatype I

Datatype – how to interpret a data elementPermanent part of the dataset definitionTwo classes: atomic and compoundCan be stored in a file as an HDF5 object (HDF5 committeddatatype)Can be shared among different datasets


HDF5 Datatype IIHDF5 atomic types

normal integer and floatuser-definable (e.g., 13-bit integer)variable length types (e.g., strings)references to objects/dataset regionsenumeration - names mapped to integersarray


HDF5 Datatype IIIHDF5 compound types

Comparable to C structs (“records”)Members can be atomic or compound types


HDF5 dataset: array of records


Special storage options for dataset


HDF5 Attribute

Attribute – data of the form “name = value”, attached to anobject by applicationOperations similar to dataset operations, but

Not extendibleNo compression or partial I/O

Can be overwritten, deleted, added during the “life“ of adataset


HDF5 Group

A mechanism for organizing collections of related objectsEvery file starts with a root groupSimilar to UNIX directories

/ (root)/X/Y/X/temp

Can have attributes


Partial I/O

Move just part of a dataset


Layers – parallel example


Virtual I/O layer


Virtual I/O layer

A public API for writing I/O driversAllows HDF5 to interface to disk, the network, memory, ora user-defined device


Portability and Robustness

Runs almost anywhereLinux and UNIX workstationsWindows, Mac OS XBig ASC machines, Crays, VMS systemsTeraGrid and other clustersSource and binaries available fromhttp://www.hdfgroup.org/HDF5/release/index.html


Other Software

The HDF GroupHDFViewJava toolsCommand-line utilitiesWeb browser plug-inRegression and performance testing softwareParallel h5diff

3rd Party (IDL, MATLAB, Mathematica, PyTables, HDFExplorer, LabView)Communities (EOS, ASC, CGNS)Integration with other software (iRODS, OPeNDAP)


HDF5 software stack


Structure of HDF5 Library


Goals of HDF5 Library

Provide flexible API to support a wide range of operationson data.Support high performance access in serial and parallelcomputing environments.Be compatible with common data models andprogramming languages.

Because of these goals, the HDF5 API is rich and large


Operations Supported by the API

Create groups, datasets, attributes, linkagesCreate complex data typesAssign storage and I/O properties to objectsPerform complex subsetting during read/writeUse variety of I/O-”devices“ (parallel, remote, etc.)Transform data during I/OQuery about file and structure and propertiesQuery about object structure, content, properties


Characteristics of the HDF5 API

For flexibility, the API is extensive300+ functions

This can be daunting but there is hopeA few functions can do a lotStart simpleBuild up knowledge as more features are needed

Library functions are categorized by object type”H5Lite“ API supports basic capabilities


The General HDF5 API

Currently C, Fortran 90, Java, and C++ bindings.C routines begin with prefix H5?? is a character corresponding to the type of object thefunction acts on

Example APIs:H5D : Dataset interface e.g., H5DreadH5F : File interface e.g., H5FopenH5S : dataSpace interface e.g., H5Sclose


Compiling HDF5 Applications

HDF5 wrappers (not fully tested!)h5cc HDF5 C compiler command (similar to mpicc)h5fc HDF5 F90 compiler command (similar to mpif90)h5c++ HDF5 C++ compiler command

Makefile in /bgsys/local/hdf5/examples/jugene/



Properties of object are optionally definedCreation propertiesAccess property listsDefault values used if none are defined

Object is opened or createdObject is accessed, possibly many timesObject is closed


Order of Operations

An order is imposed on operations by argumentdependencies

A file must be opened before a dataset because the datasetopen call requires a file handle as an argument

Objects can be closed in any order


HDF5 Defined Types

For portability, the HDF5 library has its own defined types:hid t: object identifiers (native integer)hsize t: size used for dimensions (unsigned long or unsignedlong long)hssize t: for specifying coordinates and sometimes fordimensions (signed long or signed long long)herr t: function return valuehvl t: variable length datatype

For C, include hdf5.h in your HDF5 application.


HDF5 Information

http://www.hdfgroup.org/HDF5//usr/local/hdf5/examples/ (JUQUEEN / JUROPA)/bgsys/local/hdf5/examples/ (JUQUEEN)Google ;-)


Outline

Introduction



Parallele HDF5Overview of Parallel HDF5 Design


PHDF5 Requirements

Support MPI programmingPHDF5 files compatible with serial HDF5 files

Shareable between different serial or parallel platformsSingle file image to all processes

One file per process design is undesirableExpensive post processingNot usable by different number of processes

Standard parallel I/O interfaceMust be portable to different platforms


PHDF5 Implementation Layers


MPI-IO vs. HDF5 I

MPI-IO is an Input/Output API.It treats the data file as a “linear byte stream” and eachMPI application needs to provide its own file view and datarepresentations to interpret those bytes.All data stored are machine dependent except the“external32” representation.External32 is defined in Big Endianness

Little-endian machines have to do the data conversion in bothread or write operations.64bit sized data types may lose information.


MPI-IO vs. HDF5 IIHDF5 is a data management software.It stores the data and metadata according to the HDF5data format definition.

HDF5 file is self-described.Each machine can store the data in its own nativerepresentation for efficient I/O without loss of data precision.Any necessary data representation conversion is done by theHDF5 library automatically.


Programming Restrictions

Most PHDF5 APIs are collectivePHDF5 opens a parallel file with a communicator

Returns a file-handleFuture access to the file via the file-handleAll processes must participate in collective PHDF5 APIsDifferent files can be opened via different communicators


Examples of PHDF5 API

Examples of PHDF5 collective APIFile operations: H5Fcreate, H5Fopen, H5FcloseObjects creation: H5Dcreate, H5Dopen, H5DcloseObjects structure: H5Dextend (increase dimension sizes)

Array data transfer can be collective or independentDataset operations: H5Dwrite, H5DreadCollectiveness is indicated by function parameters, not byfunction names as in MPI API


What Does PHDF5 Support?

After a file is opened by the processes of a communicatorAll parts of file are accessible by all processesAll objects in the file are accessible by all processesMultiple processes may write to the same data arrayEach process may write to individual data array

C and F90 language interfaces


Programming model

HDF5 uses access template object (property list) to controlthe file access mechanismGeneral model to access HDF5 file in parallel:

Setup MPI-IO or POSIX access template(access property list)Open FileAccess DataClose File


Setup MPI-IO access template

Each process of the MPI communicator creates an accesstemplate and sets it up with MPI parallel accessinformation

C: herr t H5Pset fapl mpio(hid t plist id,

MPI Comm comm, MPI Info info);

F90: h5pset fapl mpio f(plist id, comm, info)

plist id is a file access property list identifier


Creating and Opening Dataset

All processes of the communicator open / close a datasetby a collective call

C: H5Dcreate or H5Dopen; H5DcloseF90: h5dcreate f or h5dopen f; h5dclose f

All processes of the communicator must extend anunlimited dimension dataset before writing to it

C: H5DextendF90: h5dextend f


Accessing a Dataset

All processes that have opened dataset may do collectiveI/OEach process may do independent and arbitrary numberof data I/O access calls

C: H5Dwrite and H5DreadF90: h5dwrite f and h5dread f


Dataset transfer property

Create and set dataset transfer propertyC: H5Pset dxpl mpio

H5FD MPIO COLLECTIVEH5FD MPIO INDEPENDENT (default)

F90: h5pset dxpl mpio fH5FD MPIO COLLECTIVE FH5FD MPIO INDEPENDENT F (default)

Access dataset with the defined transfer property


Writing and Reading Hyperslabs

Distributed memory model: data is split among processesPHDF5 uses HDF5 hyperslab modelEach process defines memory and file hyperslabsEach process executes partial write/read call

Collective callsIndependent calls

H5Sselect hyperslab(filespace,H5S SELECT SET,offset,

stride, count, block)


Writing dataset by rows


Writing dataset by columns


Writing dataset by pattern


Writing dataset by chunks


portable parallel i/o...datatype deﬁnitions can be shared among objects in an hdf ﬁle, providing...

Documents