cs 744: big data...
TRANSCRIPT
![Page 1: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/1.jpg)
CS 744: Big Data Systems
Shivaram Venkataraman Fall 2018
![Page 2: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/2.jpg)
ADMINISTRIVIA
- Assignment 2, Midterm grades this week - Course Projects: round 2 meetings next Friday - Next Tuesday: Guest speaker for first part
![Page 3: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/3.jpg)
WHAT WE KNOW SO FAR
![Page 4: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/4.jpg)
CONTINUOUS OPERATOR MODEL
Long-lived operators
Distributed Checkpoints High overhead for Fault
Recover
Naiad Task
Control Message Driver
Network Transfer
Mutable State
Stragglers ?
![Page 5: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/5.jpg)
GOALS
1. Scalability to hundreds of nodes
2. Minimal cost beyond base processing (no replication) 3. Second-scale latency 4. Second-scale recovery from faults and stragglers
![Page 6: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/6.jpg)
DISCRETIZED STREAMS
![Page 7: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/7.jpg)
DISCRETIZED STREAMS (DSTREAMS)
Approach - Use short, stateless, deterministic tasks - Store state across tasks as in-memory RDDs - Fine-grained tasks à Parallel recovery / speculation
Model
- Chunk inputs into a number of micro-batches - Processed via parallel operations (i.e., map, reduce, groupBy etc.)
- Save intermediate state as RDD / write output to external systems
![Page 8: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/8.jpg)
COMPUTATION MODEL: MICRO-BATCHES
Task
Control Message Driver
SHUFFLE
Network Transfer
Micro-Batch
![Page 9: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/9.jpg)
EXAMPLE pageViews=readStream(http://...,"1s")ones=pageViews.map(event=>(event.url,1))counts=ones.runningReduce((a,b)=>a+b)
![Page 10: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/10.jpg)
ARCHITECHTURE
![Page 11: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/11.jpg)
DSTREAM API
Output operations save output to external database / filesystem
Transformations
Stateless: map, reduce, groupBy, join Stateful: window(“5s”) à RDDs with data in [0,5), [1,6), [2,7) reduceByWindow(“5s”, (a, b) => a + b) à incremental aggregation
![Page 12: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/12.jpg)
ASSOCIATIVE, INVERTIBLE
Add previous 5 each time
Subtract previous and add current
![Page 13: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/13.jpg)
OTHER ASPECTS
Tracking State: streams of (Key, Event) à (Key, State) - Initialize: Create a State from the first event - Update: Return new State given, old state and event - Timeout for dropping old states.
Unifying batch and stream
- Join DStream with static RDD - Attach console and query existing RDDs - Shared codebase, functions etc.
events.track((key,ev)=>1,(key,st,ev)=>ev==Exit?null:1,"30s”)
![Page 14: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/14.jpg)
SYSTEM IMPLEMENTATION
![Page 15: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/15.jpg)
OPTIMIZATIONS
Network Communication Rewrote Spark’s data plane to use asynchronous I/O
Timestep Pipelining
No barrier across timesteps unless needed Tasks from the next timestep scheduled before current finishes
Checkpointing
Async I/O, as RDDs are immutable Forget lineage after checkpoint
![Page 16: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/16.jpg)
FAULT TOLERANCE: PARALLEL RECOVERY
Worker failure - Need to recompute state RDDs stored on worker - Re-execute tasks running on the worker
Strategy - Run all independent recovery tasks in parallel - Parallelism from partitions in timestep and across timesteps
![Page 17: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/17.jpg)
EXAMPLE pageViews=readStream(http://...,"1s")ones=pageViews.map(event=>(event.url,1))counts=ones.runningReduce((a,b)=>a+b)
![Page 18: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/18.jpg)
FAULT TOLERANCE
Straggler Mitigation Use speculative execution Task runs more than 1.4x longer than median task à straggler
Master Recovery
- At each timestep, write out graph of DStreams and Scala function objects - Workers connect to a new master and report their RDD partitions - Note: No problem if a given RDD is computed twice (determinism).
![Page 19: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/19.jpg)
DISCUSSION/SHORTCOMINGS
Expressiveness - Current API requires users to “think” in micro-batches
Setting batch interval - Manual tuning. Higher batch à better throughput but worse latency
Memory usage - LRU cache stores state RDDs in memory
![Page 20: CS 744: Big Data Systemspages.cs.wisc.edu/~shivaram/cs744-slides/cs744-spark-streaming-v2.pdfCONTINUOUS OPERATOR MODEL Long-lived operators Distributed Checkpoints High overhead for](https://reader033.vdocuments.us/reader033/viewer/2022050411/5f885958c88bbe7cb0214006/html5/thumbnails/20.jpg)
SUMMARY
Micro-batches: New approach to stream processing Higher latency for fault tolerance, straggler mitigation Unifying batch, streaming analytics