integrating adios into openfoam for disk i/oopenfoam.com/documentation/files/adios-201610.pdf ·...
Post on 29-Jun-2018
236 Views
Preview:
TRANSCRIPT
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
1www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
Integrating ADIOS into OpenFOAM for Disk I/OAn update and outlook
Karl Meredith, Mark Olesen, Norbert Podhorszki
FM Global, ESI-OpenCFD, Oak Ridge National Laboratory
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
2
What is the problem?
• I/O performance issues with larger parallel cases
• Too many (small) files.
Overwhelms filer.
Quota issues.
• Simple example:
9 fields, 240 processor= 2160 files per checkpoint.
With 1000, 2000 processors...?
• Add into OpenFOAM?
No centralized I/O, but the object-registry is often used.
Background
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
3
FM Global using ORNL resources
• Norbert:I/O bottleneck?Sounds familiar – let's try ADIOS.
• ADIOS = Adaptable Input/Output System
• Meta-layer for coordinated I/O from parallel simulations
• Has a lean 'BP' (binary packed) format.
Self-describing.
Contents entirely user-defined.
• Supports various parallel file-systems.
• Why not HDF5?
Better scaling, better throughput.
Simple API. Flexible content. Good traction.
Background
Already: ZFP supportUpcoming: ADIOS2 in C++11
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
4
Transport Methods
• MPI
All processes write to a single file.
Good for small scale (1000 processors).
• POSIX
Each process writes a file independently.
Scales well until the meta-data limits on file-system(several thousand processors).
Works well on Luster.
• MPI-Aggregate
Data are aggregated for a group of processes.
Works well for 10's thousands of processors.
• Others...
ADIOS
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
5
Transport Methods
• MPI
N procs -> 1 file
• POSIX
N procs -> N files
• MPI-Aggregate
N procs -> M files
M = number of aggregators
• Can write any number of time-steps into the file.
• For OpenFOAM, makes sense to write per time-step
Restart, removal of results etc.
ADIOS
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
6
How/where to start?
• Function Object
Minimally invasive.
Rapid prototyping.
Ensure all details work well.
• Guiding Idea
Alternative for some OpenFOAM I/O.
Don't replace everything.
• Concentrate on the big offenders:
Meshes, Volume fields, Lagrangian.
• Should be relatively simple to use.
Implementation
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
7
File Content / Layout Separated per time-step.
• ADIOS is very flexible.
Attributes are common for all processes.
Variables are data containers (byte, float,…)
• Internal Naming
Accommodate multi-region, multi-cloud, moving mesh ...
Enough description to document content, allow reuse ...
• Content
Meshes can use primitives (int, double).
Fields use raw byte content
▪ Parses identically to standard OpenFOAMbinary output stream.
Implementation
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
8
File Content Layout
• Lagrangian
Much trickier.
Various cloud / thermodynamics types.
Variable length records
▪ Collisions, multiple-species.
Implementation
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
9
General Adapter Layer
• Output
Stream through a counter.
Stream to an internal buffer.
Pass to ADIOS.
• Input
ADIOS -> data buffer.
Istream adapter on buffer -> OpenFOAM.
• Lagrangian
Encoding/decoding layer just added for downstream consumers.
Implementation
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
10
Output Behaviour
• Query object-registry for AUTO_WRITE
Meshes.
Volume/surface fields.
Clouds.
• Optionally
Specify additional fields.
Restrict to explicitly named fields.
Blacklist fields (eg, reduced output for post-pro)
• Output file contains the mesh, or time values for the last points / topology written.
Function Object Control
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
11
Restart Behaviour
• The boot-strapping problem.
• Use hybrid fast-forward approach.
Normal mechanisms for initial mesh, initial fields.
ADIOS data to jump forward in time.
Move points (currently w/o top change).
Update fields.
Recreate clouds.
Function Object Control
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
12
Using interFoam
• 20.3M cell VOF case
• Intel(R) Xeon(R) E5-2630 v3 @ 2.4 GHz
• NFS file system
• 24 processors + 240 processors
Demonstration Case
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
13
24 Processors Performance
• Test study
100 time steps.
Writing each time step.
• OpenFOAM
1080 s: no write
1647 s: write
• ADIOS transport
1251 s: null (no write)
1681 s: MPI
1670 s: POSIX
Demonstration Case
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
14
240 Processors Performance
• Test study
100 time steps.
Writing each time step.
• OpenFOAM
178 s: no write
3426 s: write
• ADIOS transport
226 s: null (no write)
1614 s: MPI
1318 s: POSIX
Demonstration Case
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
15
Output size and count
• Native output
2.2 GB
2160 files per checkpoint (9 x 240)
• ADIOS MPI
2.4 GB
1 bp file per checkpoint
• ADIOS POSIX
2.4 GB
240 bp files per checkpoint
Demonstration Case
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
16
Performance / Potential Development
• Performance advantages shown
Better parallel scaling.
I/O bottleneck removed == 3x wall-clock speedup (not many processes).
Further optimizations possible for much larger benefit.
• Checkpoint restart possible
Bootstrap using native OpenFOAMmechanisms.
• Demonstrated feasibility of ADIOS adapter.
Possible as a modular/incremental change.
• Function Object
Still represents a good path forward
• Still needs some work to improve handling
Interim Balance
www.esi-group.com
Copyright © ESI Group, 2016. All rights reserved.
17
Concrete General
• More Testing
Design, Performance, Scaling.
• Lagrangian
Finish handling more types.
More testing.
• Restart Improvements
Topology changes
• Code
Modularity / Structure
Decompose/reconstruct for ADIOS.
• Help support 3rd-party ADIOS/OpenFOAMreader.
Potential reuse of Takuya OSHIMA's vtk reader.
Slightly easier to parse than native files (no #include, regex etc)
• Many other topics
Ongoing / Future Work
top related