reining in the outliers in map-reduce clusters using mantri ganesh ananthanarayanan, srikanth...
TRANSCRIPT
Reining in the Outliers in Map-Reduce Clusters using Mantri
Ganesh Ananthanarayanan, Srikanth Kandula, Albert Greenberg, Ion Stoica, Ganesh Ananthanarayanan, Srikanth Kandula, Albert Greenberg, Ion Stoica, Yi Lu, Bikas Saha, Edward HarrisYi Lu, Bikas Saha, Edward Harris@Microsoft@Microsoft
Presenter: Weiyue XuPresenter: Weiyue Xu
OSDI'10 Proceedings of the 9th USENIX conference on Operating systems OSDI'10 Proceedings of the 9th USENIX conference on Operating systems design and implementationdesign and implementation
Credits
• Modified version of – http://
www.cs.duke.edu/courses/fall10/cps296.2/lectures/
– research.microsoft.com/en-us/UM/people/srikanth/data/Combating%20Outliers%20in%20Map-Reduce.web.pptx
Outline
• Introduction
• Cause of the Outlier
• Mantri
• Evaluation
• Discussion and Conclusion
log(size of dataset)GB109
TB1012
PB1015
EB1018
log(size of cluster)
104
1
103
102
101
105
HPC,|| databases
mapreduce
MapReduce • Decouples customized data operations from mechanisms to scale• Widely used
• Cosmos (based on SVC’s Dryad) + Scope @ Bing• MapReduce @ Google• Hadoop inside Yahoo! and on Amazon’s Cloud (AWS)
e.g., the Internet, click logs, bio/genomic data
Local write
An Example
How it Works:
Goal Find frequent search queries to Bing
SELECT Query, COUNT(*) AS FreqFROM QueryTableHAVING Freq > X
What the user says:
Read Map Reduce
file block 0
job manager
task
task
tasktask
task
output block 0
output block 1
file block 1
file block 2
file block 3
assign work, get progress
Outliers slow down map-reduce jobs
Map.Read 22K
Map.Move 15K
Map 13K
Reduce 51K
Barrier
File System
We find that:
6
Tackling outliers, we can speed up jobs while using resources efficiently:• Quicker response improves productivity• Predictability supports SLAs• Better resource utilization
From a phase to a job
•A job may have many phases
•An outlier in an early phase has a cumulative effect
•Data loss may cause multi-phase recompute outliers
Delay due to a recompute readily cascades
Why outliers?
reduce
sort
Delay due to a recompute
map
8
Due to unavailable input, tasks have to be recomputed
Outlier
stragglers : Tasks that take 1.5 times the median task in that phaserecomputes : Tasks that are re-run because their output was lost
50% phases have > 10% stragglers and no recomputes10% of the stragglers take >10X longer
Frequency of Outliers
straggler straggler
At median, jobs slowed down by 35% due to outliers
Cost of Outliers
Previous solution
• The original MapReduce paper observed the problem and did not solve it in depth
• Current schemes (e.g. Hadoop, LATE) duplicate long-running tasks based on some metrics
• Drawbacks–Some may be unnecessary –Use extra resources–Placement may be a problem
What this Paper is About
• Identify fundamental causes of outliers
• Mantri: A Cause-, Resource-Aware mitigation scheme– Case by case analysis: takes distinct actions based
on cause– Considers opportunity cost of actions
• Results from a production deployment
Causes of Outliers
• Data Skew: data size varies across tasks in a phase.
Reduce task
Map output
uneven placement is typical in productionreduce tasks are placed at first available slots
• Crossrack Traffic
Causes of Outliers
Rack
•50% of recomputes happen on 5% of the machines
•Recompute increases resource usage
• Bad and Busy MachinesCauses of Outliers
•70% of cross track traffic is reduce traffic•Reduce reads from every map, tasks in a spot with slow network run slower•Tasks compete network among themselves
50% phases takes 62% longer to finish than ideal placement
Causes of Outliers
•Outliers cluster by time–Resource contention might be the cause
•Recomputes cluster by machines –Data loss may cause multiple recomputes
Mantri• Cause aware, and resource aware• Fix each problem with different strategies• Runtime = f (input, network, dataToProcess, ...)
Delay due to a recompute readily cascades
Why outliers?
reduce
sort
Delay due to a recompute
map
19
Mantri [Avoid Recomputations]
Mantri [Avoid Recomputations]• Idea: Replicate intermediate data, use copy if original
is unavailable• Challenge: What data to replicate?• Insight: Cost to recompute vs cost to replicate
M1
M2
tredo = r2(t2
+t1redo)
• Cost to recompute depends on data loss probabilities and time taken, and also recursively looks at prior phases.
• tredo > t rep Mantri preferentially acts on more costly inputs
20
t = predicted runtime of taskr = predicted probability of recomputation at machine
Reduce task
Map output
uneven placement is typical in productionreduce tasks are placed at first available slots
• Crossrack Traffic
Rack
Mantri [Network Aware Placement]
Mantri [Network Aware Placement]• Idea: Avoid hot-spots, keep traffic on a link proportiona
l to bandwidth• Challenges: Global co-ordination, congestion detection• Insights:
– Local control is a good approximation(each job balances its traffic)
– Link utilizations average out on the long task and are steady on the short task
If rack i has di map output need dui and dv
i for upload and download while bu
i and bvi bandwidths available
Place ai fraction of reduces such that: 22
• Data Skew - About 25% of outliers occur due to more dataToProcess (workload imbalance)
• Ignoring these is better than duplicating (state-of-the-art)
Mantri [Data Aware Task Ordering]
Mantri [Data Aware Task Ordering]• Problem: Workload imbalance causes tasks to straggle.• Idea: Restarting outliers that are lengthy is counter-
productive.• Insights:
– Theorem [Graham, 1969]– Scheduling tasks with longest processing time first is
at-most 33% worse than optimal schedule.
24
Builds an estimator T ~ function (dataToProcess) We schedule tasks in descending order of dataToProcess
Theorem [due to Graham, 1969] Doing so is no more than 33% worse than the optimal
Mantri [Resource Aware Restart]• Problem: 25% outliers remain, likely due to
contention@machine• Idea: Restart tasks elsewhere in the cluster asap• Challenge: restart or duplicate?
25
(a)
(b)
(c)
Running task
Potential restart (tnew)
nowtime
trem
√
××
Save time and resources iffP(c trem > (c+1) tnew) > δ
If pending work, duplicate iff save both time and resource Else, duplicate if expected savings are high
Continuously observe and kill wasteful copies
26
Summary• Reduce recomputation:
– preferentially replicate costly-to-recompute tasks• Poor network:
– each job locally avoids network hot-spots• DataToProcess:
– schedule in descending order of data size• Bad machines:
– quarantine persistently faulty machines• Others:
– restart or duplicate tasks, cognizant of resource cost.
Evaluation Methodology
• Mantri run on production clusters
• Baseline is results from Dryad
• Use trace-driven simulations to compare with other systems
Comparing Jobs in the Wild
w/ and w/o Mantri for one month of jobs in Bing production cluster
340 jobs that each repeated at least five times during May 25-28 (release) vs. Apr 1-30 (pre-release)
In Production, Restarts…
In Trace-Replay Simulations, Restarts…
CD
F %
clu
ster
res
ourc
es
CD
F %
clu
ster
res
ourc
es
Protecting Against Recomputes
CD
F %
clu
ster
res
ourc
es
Conclusion
• Outliers are a significant problem
• Happens due to many causes
• Mantri: cause and resource aware mitigation outperforms prior schemes
32
Discussion
• Mantri does case by case analysis for each cause, what if the causes are inter-dependent?
Questions or Comments?
Thanks!
Estimation of trem and tnew
• d: input data size
• dread: the amount read
Estimation of tnew
• processRate: estimated of all tasks in the phase• locationFactor: machine performance• d: input size