sparrow
DESCRIPTION
Sparrow. Kay Ousterhout, Patrick Wendell, Matei Zaharia , Ion Stoica. Distributed Low-Latency Spark Scheduling. Outline. The Spark scheduling bottleneck Sparrow’s fully distributed, fault-tolerant technique Sparrow’s near-optimal performance. Spark Today. User 1. Worker. Spark Context. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/1.jpg)
SparrowDistributed Low-Latency Spark SchedulingKay Ousterhout, Patrick Wendell, Matei Zaharia, Ion Stoica
![Page 2: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/2.jpg)
Outline
The Spark scheduling bottleneck
Sparrow’s fully distributed, fault-tolerant technique
Sparrow’s near-optimal performance
![Page 3: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/3.jpg)
Spark TodayWork
erWorkerWorkerWorkerWorker
Worker
Spark ContextUser
1User 2User 3
Query Compilation
StorageScheduling
![Page 4: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/4.jpg)
Spark TodayWork
erWorkerWorkerWorkerWorker
Worker
Spark ContextUser
1User 2User 3
Query Compilation
StorageScheduling
![Page 5: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/5.jpg)
Job Latencies Rapidly Decreasing
10 min.
10 sec.
100 ms
1 ms
2004: MapReducebatch job
2009: Hive
query
2010: Dremel Query
2012: Impala query 2010:
In-memory
Spark query
2013:Spark
streaming
![Page 6: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/6.jpg)
Job latencies rapidly decreasing
![Page 7: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/7.jpg)
Job latencies rapidly decreasing+
Spark deployments growing in size
Scheduling bottleneck!
![Page 8: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/8.jpg)
Spark scheduler throughput:1500 tasks / second
1 second 100100 ms 10
10 second 1000
Task DurationCluster size(# 16-core machines)
![Page 9: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/9.jpg)
Optimizing the Spark Scheduler
0.8: Monitoring code moved off critical path
0.8.1: Result deserialization moved off critical path
Future improvements may yield 2-3x higher throughput
![Page 10: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/10.jpg)
Is the scheduler the bottleneck in my cluster?
![Page 11: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/11.jpg)
WorkerWorkerWorkerWorkerWorker
Worker
Cluster Scheduler
Task launch
Task completion
![Page 12: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/12.jpg)
WorkerWorkerWorkerWorkerWorker
Worker
Cluster Scheduler
Task launch
Task completion
![Page 13: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/13.jpg)
WorkerWorkerWorkerWorkerWorker
Worker
Cluster Scheduler
Task launch
Task completion
Scheduler
delay
![Page 14: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/14.jpg)
![Page 15: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/15.jpg)
![Page 16: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/16.jpg)
Spark TodayWork
erWorkerWorkerWorkerWorker
Worker
Spark ContextUser
1User 2User 3
Query Compilation
StorageScheduling
![Page 17: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/17.jpg)
Future SparkWork
erWorkerWorkerWorkerWorker
Worker
User 1User 2User 3
SchedulerQuery
compilation
SchedulerQuery
compilation
SchedulerQuery
compilation
Benefits:High
throughputFault
tolerance
![Page 18: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/18.jpg)
Future SparkWork
erWorkerWorkerWorkerWorker
Worker
User 1User 2User 3
SchedulerQuery
compilation
SchedulerQuery
compilation
SchedulerQuery
compilation
Storage:
Tachyon
![Page 19: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/19.jpg)
Scheduling with SparrowWork
erWorkerWorkerWorkerWorker
Scheduler
Scheduler
Scheduler
SchedulerStage
Worker
![Page 20: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/20.jpg)
Stage
Batch SamplingWork
erWorkerWorkerWorkerWorker
Scheduler
Scheduler
Scheduler
Scheduler
Worker
Place m tasks on the least loaded of 2m workers
4 probes (d =
2)
![Page 21: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/21.jpg)
Queue length poor predictor of wait timeWork
erWorker
80 ms155
ms
530 ms
Poor performance on heterogeneous workloads
![Page 22: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/22.jpg)
Stage
Late Binding
WorkerWorkerWorkerWorkerWorker
Scheduler
Scheduler
SchedulerScheduler
Worker
Place m tasks on the least loaded of dm workers
4 probes (d =
2)
![Page 23: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/23.jpg)
Late Binding
Scheduler
Scheduler
SchedulerScheduler
Place m tasks on the least loaded of dm workers
4 probes (d =
2)
WorkerWorkerWorkerWorkerWorker
Worker
Stage
![Page 24: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/24.jpg)
Late Binding
Scheduler
Scheduler
SchedulerScheduler
Place m tasks on the least loaded of dm workers
Worker
requests
task
WorkerWorkerWorkerWorkerWorker
Worker
Stage
![Page 25: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/25.jpg)
What about constraints?
![Page 26: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/26.jpg)
Stage
Per-Task Constraints
Scheduler
Scheduler
Scheduler
Scheduler
WorkerWorkerWorkerWorkerWorker
Worker
Probe separately for each task
![Page 27: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/27.jpg)
Technique Recap
Scheduler
Scheduler
Scheduler
SchedulerBatch
sampling+
Late binding+
Constraints
WorkerWorkerWorkerWorkerWorker
Worker
![Page 28: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/28.jpg)
How well does Sparrow perform?
![Page 29: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/29.jpg)
How does Sparrow compare to Spark’s native scheduler?
100 16-core EC2 nodes, 10 tasks/job, 10 schedulers, 80% load
![Page 30: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/30.jpg)
TPC-H Queries: BackgroundTPC-H: Common benchmark for
analytics workloads
Sparrow
Spark
Shark: SQL execution engine
![Page 31: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/31.jpg)
TPC-H Queries
100 16-core EC2 nodes, 10 schedulers, 80% load
95
75
2550
Percentiles
5
Within 12% of ideal
Median queuing delay of 9ms
![Page 32: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/32.jpg)
Policy Enforcement
WorkerHigh PriorityLow Priority Worker
User A (75%)User B (25%)
Fair SharesServe queues using
weighted fair queuing
PrioritiesServe queues based on strict priorities
![Page 33: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/33.jpg)
Weighted Fair Sharing
![Page 34: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/34.jpg)
Fault Tolerance
Scheduler 1
Scheduler 2
Spark Client 1 ✗Spark
Client 2
Timeout: 100msFailover: 5ms
Re-launch queries: 15ms
![Page 35: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/35.jpg)
Making Sparrow feature-complete
Interfacing with UI
Delay scheduling
Speculation
![Page 36: Sparrow](https://reader035.vdocuments.us/reader035/viewer/2022062410/568161ea550346895dd21a62/html5/thumbnails/36.jpg)
(2) Distributed,
fault-tolerant scheduling
with Sparrow www.github.com/radlab/sparrow
Scheduler
Scheduler
Scheduler
Scheduler
WorkerWorkerWorkerWorkerWorker
Worker
(1) Diagnosing a
Spark scheduling bottleneck