gabrielle allen*, thomas dramlitsch*, ian foster † , nicolas karonis ‡ ,
DESCRIPTION
Supporting Efficient Execution in Heterogeneous Distributed Computing Environments with Cactus and Globus. Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ , Matei Ripeanu # , Ed Seidel*, Brian Toonen †. * Max-Planck-Institut f ü r Gravitationsphysik - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/1.jpg)
Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster†, Nicolas Karonis‡,Matei Ripeanu#, Ed Seidel*, Brian Toonen†
* Max-Planck-Institut für Gravitationsphysik†Argonne National Labs
‡Northern Illinois University#University of Chicago
Supporting Efficient Execution in HeterogeneousDistributed Computing Environments with Cactus
and Globus
![Page 2: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/2.jpg)
• Large scale distributed computing what,why & recent experiments, results
• Short review of problems of executing codes in grid environments (networks, algorithms, infrastructure etc.)
• Introducing a framework for distributed computing how CACTUS, GLOBUS and MPICH-G2 together form a complete set of tools to for easy execution of codes in grid environments
• The status of distributed computing where we are, what we can do now
This talk is about
![Page 3: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/3.jpg)
Major Problems of Metacomputing
•Heterogeneitydifferent operating systems, different queue systems, different authentication schemes, different processors/processor speeds
•Networkswide area networks are getting faster every day, but are still orders of magnitude slower than intra-machine networks of supercomputers
•AlgorithmsMost parallel codes use communication schemes, processor distributions and algorithms which are written for single machine execution (i. e. unaware of the nature of a grid environment)
(see sc95,sc98,may 2001,now)
![Page 4: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/4.jpg)
Layered structure of the framework
GLOBUSBasic information about job, infrastructure, authentication, queues, resources, etc.
MPICH-G2 Distributed high-performanceimplementation of MPI
CACTUS Grid-aware parallelizing- and communication-algorithms
Application Numerical application, unaware of the grid
![Page 5: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/5.jpg)
First test: Distributed Teraflop Computing (DTF)
CPUs: 120 120 240 1020 = 1500
Gigabit-Ethernet-Connection (~100MB/s)
OC-12-Network(~2.5MB/s per stream)
NCSA SDSC
The code computed the evolution of gravitational waves, according to Einstein’s theory of general relativity.
The setup included all major problems: multiple sites/authentication, heterogenity, slow networks, different queue systems, MPI-implementations ...
![Page 6: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/6.jpg)
Communication internals: Ghostzones
![Page 7: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/7.jpg)
Communication internals: Ghostzones
![Page 8: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/8.jpg)
Communication internals: Ghostzones
![Page 9: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/9.jpg)
Communication internals: Ghostzones
![Page 10: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/10.jpg)
Communication internals: Ghostzones
In the DTF run we used a ghostzone size of 10
![Page 11: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/11.jpg)
DTF Setup
CPUs: 120 120 240 1020 = 1500
Gigabit-Ethernet-Connection (10 ghosts + compression)
OC-12-Network(10 ghosts + compression)
NCSA SDSC
Eficciency: 63% for 1500 CPU run and 88% for 1140 CPU run
Without ghostzone + compression: ~15%
![Page 12: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/12.jpg)
• Large scale distributed computing is possible with cactus,globus and mpich-g2
• Applying simple communication tricks improves efficiency a lot
• But: finding out best processor topology, where to compress, where to increase ghostsizes, how to loadbalance etc. goes far beyond what the user is willing to do
• configuration was not “fault-tolerant”
• Thus: we need a code which automatically and dynamically adapts itself to the given grid environment
And that’s what we have done
What we learnt from the DTF run
![Page 13: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/13.jpg)
Processor distributiony
x
0
3
2
1
![Page 14: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/14.jpg)
Processor distribution
y
x
0
3
2
1
![Page 15: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/15.jpg)
Load Balancing
0
3
2
1
![Page 16: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/16.jpg)
Load Balancing
0
3
2
1
![Page 17: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/17.jpg)
10
15
20
25
30
35
40
45
0 20 40 60 80 100 120
iteration
effic
ienc
y
4+4 processor transatlantic run
10
20
30
40
50
0 20 40 60 80 100
iteration
effic
ienc
y
8+8 processor NCSA+Washu
DTF run could be launched right away, with almost no preparation!
Runs here are “latest” physics-codes: many functions to synchronize , non-trivial data sets,non-communication-optimized algorithms on the application level
Adaptive runAdaptive run
Standard runStandard run
Adaptive Techniques
![Page 18: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/18.jpg)
5
15
25
35
0 20 40 60iteration
effic
ienc
y
128+128 run btw. NCSA and SDSC yesterday
Launched from a portal & gained efficiency improvements of factor 6 out of the box!!
256 processors run, using unoptimized and latest fortran codes.
![Page 19: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/19.jpg)
• Processor distribution/topologies are set up in a way that communication over the WAN is always minimal
• Loadbalancing: fully automatic
• Ghostzones and compression: dynamically adaptive during the run, and only where needed
• Now Fault-tolerant
Improvements btw. April 2001 and now
• To achieve all this, we consequently used globus (DUROC api)
![Page 20: Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster † , Nicolas Karonis ‡ ,](https://reader035.vdocuments.us/reader035/viewer/2022081603/56813590550346895d9cfc5e/html5/thumbnails/20.jpg)
Conclusion
• Executing codes in a metacomputing environment is becoming as easy as
executing codes on a single machine with CACTUS, GLOBUS and MPICH-G2
• A much higher efficiency is automatically achieved during the run through dynamical adaptation
• incredible improvements between SC95 and now
• Together with the usage of portals and resource brokers the user will be able to take full advantage of the grid