Michael Bender, SUNY Stony BrookDavid Bunde, Knox CollegeVitus Leung, Sandia National LaboratoriesKevin Pedretti, Sandia National LaboratoriesCynthia Phillips, Sandia National Laboratories
Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.
New Experimental Results in Communication-Aware Processor Allocation for Supercomputers
• Commodity-based supercomputers at Sandia National Laboratories (off-the-shelf components)
• Up to 2048 processors• Production computing environment
• Our Job: Improve parallel node allocation on Cplant to optimize performance.
Computational Plant (Cplant)
The Cplant System
• DEC alpha processors
• Myrinet interconnect (Sandia modified)
• MPI
• Different sizes/topologies: usually 2D or 3D grid with toroidal wraps – Ross = 2048 proc, 3D mesh
– Zermatt = 128-proc 2D mesh
– Alaska = ~600, heavily-augmented 2D mesh (cannibalized).
• Modified Linux OS (now public domain)
• Four processors/switch (compute, I/O, service nodes)
Scheduling Environment
• Users submit jobs to queue (online)
• Users specify number of processors and runtime estimate– If a job runs past this estimate by 5 min, it is killed
• No preemption, no migration, no multitasking (security)
• Actual runtime depends on set of processors allocated and placement of other jobs
Goals:
• User - minimum response time
• Bureaucracy (GAO) - high utilization
Scheduler/Allocator Association
Scheduler and allocator effect each others’ performance.
Scheduler Allocator
Performance dependencies
Scheduler/Allocator Dissociation
• Scheduler enforces policy– Management sets priorities for access, utilization policy
• Allocator can optimize performance
UserExecutable# processorsRequested time
Job:
Job
PBSScheduler
NodeAllocator
Cplant
queue
.
.
.
What’s a Good Allocation?
Objective: Allocate jobs to processors to minimize network contention processor locality.
• Especially important for commodity networks
Good allocationFor 2D mesh
Bad allocationFor 2D mesh
Quantitative Effect of Processor Locality
But, speed-up anomaly
= 2
faster than
= empty processor
Communication Hops on a 2D grid
• L1 distance = # hops (~ # switches) between 2 processors on grid
5
4
Allocation Problem
• Given n available points on grid (some unavailable)• Find a set of k available points with minimum average (or
total) L1 distance.• Example: green allocation: 3(2) + 3(1) = 9
EmpiricalCorrelation
Leung et al, 2002
Related support:Mache and Lo, 1996
Previous Work
• Various Work forcing a convex set– Insufficient processor utilization
• Mache, Lo, Windisch MC algorithm
• Krumke et al 2-approximation, NP-hard w/general metric
• Complexity open for grids
• Dispersion problem (max distance) linear time for fixed k (Fekete and Meijer)
Optimal Unconstrained Shape[Bender,Bender,Demaine,Fekete 2004]
Almost a circle but not quite.
Only .05 percent difference in area.
0.650 245 952 951
Previous Results (Bender et al 2005)
• 7/4-approximation (2 - in d dimensions)
• PTAS ((1+)-approximation in poly time for fixed )
• MC is a 4-approximation
• Linear-time exact dynamic program 1D
• O(n log n) time for k=3
• Simulations (performance on job streams)
1
2d
Experiments: Placement Algorithm MC
• Search in shell from minimum-size region of preferred shape.
• Weight processors by shells
• Return processor set with minimum weight.
Alternative: One-Dimensional Reduction
• Order processors so that
close in linear order close in physical processor graph
• Consider one-dimensional processor allocation– Bin packing (best fit, first fit, sum of squares)
– Pack jobs onto the line (or ring), allowing fragmentation
New System Red Storm
• 12,960 Dual-Core AMD Opteron 2.4Ghz
• 39.19 TB Memory, 340 TB disk
• 124 TF peak performance
• 3D Mesh
Impact
• Changed the node allocator on Cplant– 1D default allocator
– 2D algorithms implemented
– Carried over to Red Storm system software• 1D and 2D algorithms implemented
• Selectable at compilation
• R&D 100 winner (Leung, Bender, Bunde, Pedretti, Phillips 2006)
Red Storm Development Machine
1 Cray XT3/4 Cabinet
I/O node Compute node
Does Bandwidth Make a Difference?
Real time (seconds)
User time (seconds)
Sys time (seconds)
1/4 link bandwidth
15623.353 1012.302 50.298
Full bandwidth
6314.818 1010.752 50.003
• Yes!
Red Storm Development Machine
YZ S Curve
I/O node Compute node
Red Storm Development Machine
ZY S Curve
I/O node Compute node
Hilbert (Space-Filling) Curves
• For 2D and 3D grids• Previous applications
– I/O efficient and cache-oblivious computation– Compression (images)– Domain decomposition
Red Storm Development Machine
Zoltan Hilbert-Space-Filling Curve
I/O node Compute node
Red Storm Development Machine
Spliced Hilbert-Space-Filling Curve
I/O node Compute node
Results (Makespan in Seconds)
YZ ZY random Zoltan spliced
MC1x1 5807.1
SS 5830.6 7003.2 6610.1 6699.6 6021.1
FF 5868.6 7039.5 6639.6 6758.7 6052.3
BF 5826.2 7022.6 6631.9 6739.1 6023.4
simple 6102.4
• Consistent with simulations (Bender et al 2005)
Results (Makespan Normalized)
YZ ZY random Zoltan spliced
MC1x1 1
SS 1.0040 1.206 1.1383 1.1537 1.0369
FF 1.0106 1.2122 1.1434 1.1639 1.0422
BF 1.0033 1.2093 1.1420 1.1605 1.0372
simple 1.0509
Red Storm Development Machine
Is it I/O or interprocess communication?
I/O node Compute node
Results (Makespan Normalized)
YZ ZY random Zoltan spliced
BF 1 1.2053 1.1383 1.1567 1.0338
BF2 1 1.2398 1.176 1.1828 1.0443
• Not I/O
• Consistent with Cplant experiments (Leung et al 2002)
• Consistent with Pittsburgh Supercomputing Center experiments (Weisser et al 2006)
Experiments- Test Set
• All-to-All Communications
Job Size Number of Jobs
2 1820
5 660
15 620
20 660
• High communication, best-case for runtime improvements
• Small number of repetitions (3)
Questions
• What’s the right allocation for a stream (online)?
• Scheduling + Allocation
MPP
Jobs