![Page 1: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/1.jpg)
1ADIOS – http://www. adiosapi.org Jay Lofstead
Flexible IO and Integration for Scientific Codes
Through The Adaptable IO System (ADIOS)
Jay Lofstead (GT), Scott Klasky (ORNL), Karsten Schwan(GT), Norbert Podhorszki (ORNL), Chen Jin (ORNL)
June 23, 2008
![Page 2: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/2.jpg)
2ADIOS – http://www. adiosapi.org Jay Lofstead
Outline
• Motivation
• Design Goals
• Architecture
• Related Work
• Supported Features
• Examples
• Codes & Performance
• Future Work
• Collaborators
![Page 3: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/3.jpg)
3ADIOS – http://www. adiosapi.org Jay Lofstead
Motivation
• Multiple HPC architectures• Cray, IB-based clusters, BlueGene
• Many different APIs• MPI-IO, POSIX, HDF5, netCDF• GTC (fusion) has changed IO routines 8 times so
far based on moving platforms
• Different IO patterns• Restarts, analysis, diagnostics• Different combinations provide different levels of
I/O performance
![Page 4: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/4.jpg)
4ADIOS – http://www. adiosapi.org Jay Lofstead
Design Goals
• ADIOS Fortran and C based API almost as simple as standard POSIX IO
• External configuration to describe metadata and control IO settings
• Take advantage of existing IO techniques (no new native IO methods)
Fast, simple-to-write, efficient IO for multiple platforms without changing the source code
![Page 5: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/5.jpg)
5ADIOS – http://www. adiosapi.org Jay Lofstead
Design Goals
ADIOS Contributions:
1. Data Grouping: Multiple, independently controlled IO settings• diagnostics, restarts, analysis, viz
2. Optional data items: Different data from different processes• Single group write has header from a single proc and data from all
3. Data sizes change dynamically: Datasets vary in size• Run size changes data sizes; also AMR codes
4. Constructed output is special: Reused buffers must be handled properly
• stack temporaries...
5. IO memory is second to science: Buffer space for IO is strictly limited
• respect the memory needs of the scientific codes
![Page 6: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/6.jpg)
6ADIOS – http://www. adiosapi.org Jay Lofstead
Design Goals
Provide support for these future features:
• Provide integration opportunities for other systems• Workflow• In situ visualization• Other simulations
• Provenance collection
• Other data management tools• Science aware data compression
![Page 7: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/7.jpg)
7ADIOS – http://www. adiosapi.org Jay Lofstead
Architecture
• Thin API• XML file
• data groupings with annotation
• IO method selection• buffer sizes
• Common tools• Buffering• Scheduling
• Pluggable IO routines
ExternalMetadata(XML file)
Scientific Codes
ADIOS API
DART
LIVE/DataTap
MPI-IO
POSIX IO
HD
F-5
pnetCDF
Viz Engines
Others (plug-in)
buffering schedule feedback
![Page 8: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/8.jpg)
8ADIOS – http://www. adiosapi.org Jay Lofstead
Architecture
• Data groupings• logical groups of related items written at the same
time.• Not necessarily one group per writing event
• IO Methods• Choose what works best for each grouping• Vetted and/or written by experts for each
• POSIX (Wei-keng Lao Northwestern)• MPI-IO (Steve Hodson, ORNL)• MPI-IO Collective (Wei-keng Lao Northwestern)• NULL (Jay Lofstead GT)• Ga Tech DataTap Asynchronous (Hasan Abbasi GT)• others..
![Page 9: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/9.jpg)
9ADIOS – http://www. adiosapi.org Jay Lofstead
Related Work• Specialty APIs
• HDF-5 – complex API• Parallel netCDF – no structure
• File system aware middleware• MPI ADIO layer – File system connection,
complex API
• Parallel File systems• Lustre – Metadata server issues• PVFS2 – client complexity• LWFS – client complexity• GPFS, pNFS, Panasas
![Page 10: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/10.jpg)
10ADIOS – http://www. adiosapi.org Jay Lofstead
Supported Features
• Platforms tested• Cray CNL (ORNL Jaguar)• Cray Catamount (ORNL old-Jaguar and
SNL Redstorm)• Linux Infiniband (ORNL Ewok)
• IO Methods• MPI-IO independent, MPI-IO collective,
POSIX, NULL, Ga Tech DataTap asynchronous, Rutgers DART asynchronous
![Page 11: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/11.jpg)
11ADIOS – http://www. adiosapi.org Jay Lofstead
Examples
• Setup/Cleanup codecall adios_init (‘config.xml’)
...
call adios_finalize (my_rank)
• adios_init – reads the XML file (once for every process)
• adios_finalize – provide opportunity for cleanup
![Page 12: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/12.jpg)
12ADIOS – http://www. adiosapi.org Jay Lofstead
Examples
• IO codecall adios_open (handle, ‘filename’,
‘groupname’)
call adios_write (handle, ‘varname’, var)
ADIOS_WRITE(handle,var_name)
...
call adios_close (handle)
![Page 13: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/13.jpg)
13ADIOS – http://www. adiosapi.org Jay Lofstead
Examples
• IO codecall adios_open (handle, ‘filename’,
‘groupname’)
ADIOS_GWRITE (handle)
call adios_close (handle)
- ADIOS_GREAD coming soon
![Page 14: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/14.jpg)
14ADIOS – http://www. adiosapi.org Jay Lofstead
Examples
• IO hints (for asynchronous IO)• call adios_end_iteration ()
• pacing hints for asynchronous transport methods
• use in conjunction with ‘iterations’ attribute of method element in XML
• call adios_begin_calculation ()• hint for asynchronous IO that a low-IO phase is
starting
• call adios_end_calculation ()• hint for asynchronous IO that a low-IO phase is
ending
![Page 15: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/15.jpg)
15ADIOS – http://www. adiosapi.org Jay Lofstead
Examples• XML file contents (data elements)<adios-config host-language=“Fortran”><adios-group name=“restart” coordination-
communicator=“comm” coordination-var=“cvar”>
<var name=“elements” type=“integer” write=“no”/>
<var name=“data” type=“double” path=“/” dimensions=“elements”copy-on-write=“yes”/>
</adios-group></adios-config>
![Page 16: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/16.jpg)
16ADIOS – http://www. adiosapi.org Jay Lofstead
Examples• XML file contents (other)
<attribute name=“description” path=“/data” value=“simulation particle data”/>
<global-bounds dimensions=“..” offsets=“..”>
<var .../>
</global-bounds>
<mesh ..../>
<method method=“DataTap” iterations=“50” group=“restart”>params</method>
<method method=“visit” group=“restart”/>
<buffer size-MB=“100” allocate-time=“now”/>
![Page 17: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/17.jpg)
17ADIOS – http://www. adiosapi.org Jay Lofstead
Codes & Performance
• Codes• GTC (fusion), Chimera (supernova), GTS
(fusion), XGC0 (fusion), XGC1 (fusion), Flash (astrophysics), S3D (combustion)
• Performance Measurements• GTC: nearly 25 GB/sec• GTS: 12 GB/sec• Chimera: 6.5% reduction in wall clock time
![Page 18: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/18.jpg)
18ADIOS – http://www. adiosapi.org Jay Lofstead
Codes & Performance
![Page 19: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/19.jpg)
19ADIOS – http://www. adiosapi.org Jay Lofstead
Codes & Performance
Old GTC IO
New GT ADIOS/DataTap IO
12 GB/restart
GT
C O
n R
izzo
at
OR
NL
![Page 20: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/20.jpg)
20ADIOS – http://www. adiosapi.org Jay Lofstead
Codes & Performance
• June 7, 2008: 24 hour GTC run on Jaguar at ORNL• 93% of machine (28,672 cores)• MPI-OpenMP mixed model on quad-core
nodes (7168 MPI procs)• three interruptions total (simple node
failure) with 2 10+ hour runs• Wrote 56 TB of data at 20 GB/sec (21 TB
for post analysis)• IO overhead 3% of wall clock time• Mixed IO methods of synchronous MPI-IO
and POSIX IO configured in the XML file
![Page 21: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/21.jpg)
21ADIOS – http://www. adiosapi.org Jay Lofstead
Support Tools
• bpdump – similar to h5dump
• bp2h5 – convert output file to HDF5
• bp2nc – convert output to netCDF
![Page 22: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/22.jpg)
22ADIOS – http://www. adiosapi.org Jay Lofstead
Future Work
• Work on read functionality
• Index files
• Additional native transports• direct parallel HDF5 and parallel netCDF• asynchronous MPI• Visit• others as demanded
![Page 23: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/23.jpg)
23ADIOS – http://www. adiosapi.org Jay Lofstead
Future Work
• Offload to staging area for other processing
![Page 24: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/24.jpg)
24ADIOS – http://www. adiosapi.org Jay Lofstead
Collaborators
• Chen Jin, Scott Klasky, Stephen Hodson, James B. White III, Weikuan Yu (Oak Ridge National Laboratory)
• Hasan Abbasi, Greg Eisenhauer, Karsten Schwan, Matthew Wolf, Fang Zheng (Georgia Tech)
• Wei-keng Liao, Alok Choudhary, Northwestern University
• Manish Parashar, Ciprian Docan, Rutgers University.
• Ron Oldfield, Sandia Labs
![Page 25: Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS)](https://reader036.vdocuments.us/reader036/viewer/2022062309/56814bcb550346895db8a281/html5/thumbnails/25.jpg)
25ADIOS – http://www. adiosapi.org Jay Lofstead
References• Jay Lofstead, Scott Klasky, Karsten Schwan, Norbert
Podhorszki, Chen Jin. “Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS).” In CLADE 2008 at HPDC, Boston, MA, June 2008. ACM.
• Hasan Abbasi, Matthew Wolf, Karsten Schwan. “Live data workspace: A flexible, dynamic and extensible platform for petascale applications.” In Cluster Computing, Austin, TX, September 2007. IEEE International.
• Chen Jin, Scott Klasky, Stephen Hodson, Weikuan Yu (Oak Ridge National Laboratory) Jay Lofstead, Hasan Abbasi, Karsten Schwan, Matthew Wolf (Georgia Tech) Wei-keng Liao, Alok Choudhary(Northwestern University) Manish Parashar, Ciprian Docan, Rutgers University. Ron Oldfield (Sandia National Laboratories). “Adaptive IO System (ADIOS).” In CUG 2008. Helsinki, Findland, May 2008.
• Sarp Oral. NCCS Jaguar GTC 128 Nodes Test Results. http://users.nccs.gov/~oral/jagregtests/gtc128.html.