Download - Integrating HDF5 with SRB
![Page 1: Integrating HDF5 with SRB](https://reader034.vdocuments.us/reader034/viewer/2022052413/559d3c351a28ab30078b468c/html5/thumbnails/1.jpg)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 1
Integrating HDF5 with SRB
Peter CaoThe HDF Group
November 29, 2006
Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration
![Page 2: Integrating HDF5 with SRB](https://reader034.vdocuments.us/reader034/viewer/2022052413/559d3c351a28ab30078b468c/html5/thumbnails/2.jpg)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD2
Imagine
10 GB
100 Frames x 10 GB
![Page 3: Integrating HDF5 with SRB](https://reader034.vdocuments.us/reader034/viewer/2022052413/559d3c351a28ab30078b468c/html5/thumbnails/3.jpg)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD3
Outline
• Introduction to the project• HDF5-SRB model• Server and client application• Demo if time permits
![Page 4: Integrating HDF5 with SRB](https://reader034.vdocuments.us/reader034/viewer/2022052413/559d3c351a28ab30078b468c/html5/thumbnails/4.jpg)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD4
Motivation
SRB HDF5
Distributed data system Indexing and searching Access control, etc
Large and diverse data High performance I/O Subsetting, etc
High performance distributed data system
![Page 5: Integrating HDF5 with SRB](https://reader034.vdocuments.us/reader034/viewer/2022052413/559d3c351a28ab30078b468c/html5/thumbnails/5.jpg)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD5
Goal
• Use SRB as middleware to transfer data between the server and client
• Use Object-level access for interactive and efficient access to parts of the file
A client/server system for object-level access to HDF5 files stored at SRB
server
![Page 6: Integrating HDF5 with SRB](https://reader034.vdocuments.us/reader034/viewer/2022052413/559d3c351a28ab30078b468c/html5/thumbnails/6.jpg)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD6
HDF5 Data Access on SRB
• Normal ways to access SRB:• Get the whole file: large files (100GB)• Use POSIX low level calls: low performance
• New way: • Implement proxy operations to access objects or parts of
objects in one request
![Page 7: Integrating HDF5 with SRB](https://reader034.vdocuments.us/reader034/viewer/2022052413/559d3c351a28ab30078b468c/html5/thumbnails/7.jpg)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD7
Example: Whole File Access
clientclient
Get the file
Transfer large file – slow!
HDF5HDF5
serverserverI need to see the eye of
Hurricane Bob!
HDF5HDF5
![Page 8: Integrating HDF5 with SRB](https://reader034.vdocuments.us/reader034/viewer/2022052413/559d3c351a28ab30078b468c/html5/thumbnails/8.jpg)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD8
Example: Object Level Access
clientclient
HDF5HDF5
Get me the eye of hurricane Bob
1 request, small transfer – fast!
serverserverI need to see the eye of
Hurricane Bob!
![Page 9: Integrating HDF5 with SRB](https://reader034.vdocuments.us/reader034/viewer/2022052413/559d3c351a28ab30078b468c/html5/thumbnails/9.jpg)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD9
HDF5/SRB Model
HDF5-SRB Module(pack/unpack messages)
clientclient serverserver
HDF5 Object(File, Group, Dataset,
Subset, Attribute)
SRB ServerSRB Server
HDF5HDF5
MCAT
HDF5 LibraryHDF5 Library
HDF5-SRB Module(pack/unpack messages)
HDF5 Object(File, Group, Dataset,
Subset, Attribute)
![Page 10: Integrating HDF5 with SRB](https://reader034.vdocuments.us/reader034/viewer/2022052413/559d3c351a28ab30078b468c/html5/thumbnails/10.jpg)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD10
New Data Objects and APIs
• A new set data objects• H5File, H5Group, H5Dataset, H5Datatype, etc• Encapsulated client requests and server results
• Enhanced SRB APIs• Pack/Unpack routines (exchange data between byte
stream and structure) to handle complicated struct – string, pointers, pointers to arrays, arrays of pointers, etc
• New srbGenProxyFunct (general Proxy Function) handles other types of request besides HDF5
![Page 11: Integrating HDF5 with SRB](https://reader034.vdocuments.us/reader034/viewer/2022052413/559d3c351a28ab30078b468c/html5/thumbnails/11.jpg)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD11
Running Server/Client
• A SRB server that supports HDF5• HDF5 library and other external libraries (SZIP, ZLIB)• A SRB version 3.4.0 or later from http://
www.sdsc.edu/srb/• Follow instruction on how to run SRB server from UG
packed with SRB source release or online at http://www.hdfgroup.org/hdf-srb-html/
• Any client application that implements HDF5-SRB Objects• No HDF5 library is required on the client• Example client application: HDFView 2.3 or above
![Page 12: Integrating HDF5 with SRB](https://reader034.vdocuments.us/reader034/viewer/2022052413/559d3c351a28ab30078b468c/html5/thumbnails/12.jpg)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD12
Example
![Page 13: Integrating HDF5 with SRB](https://reader034.vdocuments.us/reader034/viewer/2022052413/559d3c351a28ab30078b468c/html5/thumbnails/13.jpg)
02/18/14 HDF and HDF-EOS Workshop X, Landover, MD13
HDFView property file
.hdfview2.3srbaccount0.domain=demosrbaccount0.home=/A/home/srbAdmin.demosrbaccount0.host=kagiso.hdfgroup.uiuc.edusrbaccount0.password=srbDemo05srbaccount0.port=5544srbaccount0.resource=demoRescsrbaccount0.user=srbAdminsrbaccount1.domain=demosrbaccount1.home=/A/home/srbAdmin.demosrbaccount1.host=stanley.ncsa.uiuc.edusrbaccount1.password=srbdemo05srbaccount1.port=5544srbaccount1.resource=demoRescsrbaccount1.user=srbAdmin