![Page 1: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/1.jpg)
Confidential + ProprietaryConfidential + Proprietary
Distributed Systems in Practice, in TheoryAysylu GreenbergMarch 8th, 2016
![Page 2: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/2.jpg)
Confidential + Proprietary
![Page 3: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/3.jpg)
Confidential + Proprietary
![Page 4: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/4.jpg)
Confidential + Proprietary
Aysylu Greenberg
@aysylu22
![Page 5: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/5.jpg)
Confidential + Proprietary
Towards Distributed Build System
![Page 6: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/6.jpg)
Confidential + Proprietary
Towards Distributed Build System
![Page 7: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/7.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 8: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/8.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 9: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/9.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 10: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/10.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 11: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/11.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 12: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/12.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 13: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/13.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 14: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/14.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 15: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/15.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 16: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/16.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 17: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/17.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 18: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/18.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 19: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/19.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 20: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/20.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 21: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/21.jpg)
Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System
![Page 22: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/22.jpg)
Confidential + Proprietary
![Page 23: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/23.jpg)
Confidential + Proprietary
Papers We Love SF
![Page 24: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/24.jpg)
Confidential + Proprietary
Aysylu Greenberg
@aysylu22
![Page 25: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/25.jpg)
Confidential + Proprietarymeetup.com/Papers-We-Love-London
![Page 26: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/26.jpg)
Confidential + Proprietary
Today
● Staged Event-Driven Architecture
![Page 27: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/27.jpg)
Confidential + Proprietary
Today
● Staged Event-Driven Architecture● Leases
![Page 28: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/28.jpg)
Confidential + Proprietary
Today
● Staged Event-Driven Architecture● Leases● Inaccurate Computations
![Page 29: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/29.jpg)
Confidential + ProprietaryConfidential + Proprietary
Computer Science ResearchInDistributed Systems Industry
![Page 30: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/30.jpg)
Confidential + Proprietary
Operating systems research
![Page 31: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/31.jpg)
Confidential + Proprietary
Operating systems research
![Page 32: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/32.jpg)
Confidential + Proprietary
Operating systems research
Concurrency
![Page 33: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/33.jpg)
Confidential + Proprietary
Operating systems research
Concurrency
Concurrency primitives: mutex & semaphore
![Page 34: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/34.jpg)
Confidential + Proprietary
Operating systems research
Concurrency
Concurrency primitives: mutex & semaphore
Processes execute at different speeds
![Page 35: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/35.jpg)
Confidential + Proprietary
Time in distributed systems
![Page 36: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/36.jpg)
Confidential + Proprietary
Time in distributed systems
![Page 37: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/37.jpg)
Confidential + Proprietary
Time in distributed systems
Pipelining
![Page 38: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/38.jpg)
Confidential + Proprietary
Time in distributed systems
Pipelining
![Page 39: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/39.jpg)
Confidential + Proprietary
Internet
![Page 40: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/40.jpg)
Confidential + Proprietary
Internet
Distributed consensus
![Page 41: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/41.jpg)
Confidential + Proprietary
Internet
Distributed consensus
![Page 42: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/42.jpg)
Confidential + Proprietary
Internet
Distributed consensus
![Page 43: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/43.jpg)
Confidential + Proprietary
Internet
Distributed consensus
Paxos
![Page 44: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/44.jpg)
Confidential + Proprietary
Reconsider large systems
![Page 45: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/45.jpg)
Confidential + Proprietary
Reconsider large systems
Platform as a service
![Page 46: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/46.jpg)
Confidential + Proprietary
CS Research is Timeless
Inform decisions
Mitigate technical risk
![Page 47: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/47.jpg)
Confidential + ProprietaryConfidential + Proprietary
Staged Event Driven Architecture&Deep Pipelines
2001
![Page 48: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/48.jpg)
Confidential + Proprietary
Hardware Pipelines
![Page 49: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/49.jpg)
Confidential + Proprietary
Data Pipelines
https://en.wikipedia.org/wiki/Graphics_pipeline
![Page 50: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/50.jpg)
Confidential + Proprietary
Search Indexing Pipelines
![Page 51: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/51.jpg)
Confidential + Proprietary
Search Indexing Pipelines
![Page 52: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/52.jpg)
Confidential + Proprietary
Search Indexing Pipelines
![Page 53: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/53.jpg)
Confidential + Proprietary
Search Indexing Pipelines
![Page 54: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/54.jpg)
Confidential + Proprietary
Search Indexing Pipelines
![Page 55: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/55.jpg)
Confidential + Proprietary
Search Indexing Pipelines
![Page 56: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/56.jpg)
Confidential + Proprietary
Search Indexing Pipelines
![Page 57: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/57.jpg)
Confidential + Proprietary
Search Indexing Pipelines
![Page 58: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/58.jpg)
Confidential + Proprietary
Search Indexing Pipelines
![Page 59: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/59.jpg)
Confidential + Proprietary
Search Indexing Pipelines
![Page 60: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/60.jpg)
Confidential + Proprietary
Search Indexing Pipelines
![Page 61: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/61.jpg)
Confidential + Proprietary
![Page 62: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/62.jpg)
Confidential + Proprietary
Staged Event Driven Architecture
![Page 63: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/63.jpg)
Confidential + Proprietary
Single-machine pipeline
generalizes to distributed pipelines
Staged Event Driven Architecture
![Page 64: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/64.jpg)
Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers
![Page 65: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/65.jpg)
Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers
+-
![Page 66: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/66.jpg)
Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers○ automatic tuning
+-
![Page 67: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/67.jpg)
Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers○ automatic tuning
■ thread pool sizing+
-
![Page 68: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/68.jpg)
Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers○ automatic tuning
■ thread pool sizing■ event batching +
-
![Page 69: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/69.jpg)
Confidential + Proprietary
+-
Staged Event Driven Architecture
● Dynamic resource controllers○ automatic tuning
■ thread pool sizing■ event batching
○ load shedding via queue
![Page 70: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/70.jpg)
Confidential + Proprietary
+-
Staged Event Driven Architecture
● Dynamic resource controllers○ automatic tuning
■ thread pool sizing■ event batching
○ load shedding via queue■ backpressure
![Page 71: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/71.jpg)
Confidential + Proprietary
+-
Staged Event Driven Architecture
● Dynamic resource controllers○ automatic tuning
■ thread pool sizing■ event batching
○ load shedding via queue■ backpressure■ threshold, filter, reorder, aggregate
![Page 72: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/72.jpg)
Confidential + ProprietaryConfidential + Proprietary
Leasesas Heart Beat in Distributed Systems 1989
![Page 73: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/73.jpg)
Confidential + Proprietary
![Page 74: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/74.jpg)
Confidential + Proprietary
Leases
● Distributed locking
![Page 75: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/75.jpg)
Confidential + Proprietary
Leases
● Distributed locking● Lease term tradeoffs
○ short
![Page 76: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/76.jpg)
Confidential + Proprietary
Leases
● Distributed locking● Lease term tradeoffs
○ short■ delay from client and server failures
minimized■ reduced false write-sharing
![Page 77: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/77.jpg)
Confidential + Proprietary
Leases
● Distributed locking● Lease term tradeoffs
○ short vs long
![Page 78: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/78.jpg)
Confidential + Proprietary
Leases
● Distributed locking● Lease term tradeoffs
○ short vs long■ more efficient for frequently accessed
data■ minimized lease extension overhead on
server and client
![Page 79: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/79.jpg)
Confidential + Proprietary
Leases
● Distributed locking● Lease term tradeoffs
○ short vs long● Use of leases in modern applications
○ Leader election TTL (in etcd)
![Page 80: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/80.jpg)
Confidential + Proprietary
Leases
● Distributed locking● Lease term tradeoffs
○ short vs long● Use of leases in modern applications
○ Leader election TTL (in etcd)○ Liveness detection
![Page 81: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/81.jpg)
Confidential + Proprietary
![Page 82: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/82.jpg)
Confidential + Proprietary
Do the build, s'il vous plaît!
Build System
Dita Roque-Gourary
![Page 83: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/83.jpg)
Confidential + Proprietary
Do the build, s'il vous plaît!
Build System
OK/D'Accord!
Dita Roque-Gourary
![Page 84: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/84.jpg)
Confidential + Proprietary
Do the build, s'il vous plaît!
Build System
OK/D'Accord!
Waiting for the results, merci.
Dita Roque-Gourary
![Page 85: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/85.jpg)
Confidential + Proprietary
Do the build, s'il vous plaît!
Build System
OK/D'Accord!
Waiting for the results, merci.
Build is in progress
Dita Roque-Gourary
![Page 86: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/86.jpg)
Confidential + Proprietary
Do the build, s'il vous plaît!
Build System
OK/D'Accord!
Waiting for the results, merci.
Build is in progress
Dita Roque-Gourary
Waiting for the results, merci.
![Page 87: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/87.jpg)
Confidential + Proprietary
Do the build, s'il vous plaît!
Build System
OK/D'Accord!
Waiting for the results, merci.
Build is in progress
Build is finished/fini.Dita Roque-Gourary
Waiting for the results, merci.
![Page 88: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/88.jpg)
Confidential + Proprietary
Leases in Build System
![Page 89: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/89.jpg)
Confidential + ProprietaryConfidential + Proprietary
Inaccurate Computations&Serving Search Results
![Page 90: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/90.jpg)
Confidential + Proprietary
From Accurate to"Good Enough"
![Page 91: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/91.jpg)
Confidential + Proprietary
![Page 92: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/92.jpg)
Confidential + Proprietary
![Page 93: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/93.jpg)
Confidential + Proprietary
![Page 94: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/94.jpg)
Confidential + Proprietary
![Page 95: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/95.jpg)
Confidential + Proprietary
![Page 96: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/96.jpg)
Confidential + Proprietary
Inaccuracy for Resilience
1. Task decomposition
![Page 97: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/97.jpg)
Confidential + Proprietary
Inaccuracy for Resilience
1. Task decomposition2. Baseline for correctness
![Page 98: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/98.jpg)
Confidential + Proprietary
Inaccuracy for Resilience
1. Task decomposition2. Baseline for correctness3. Criticality Testing
![Page 99: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/99.jpg)
Confidential + Proprietary
Inaccuracy for Resilience
1. Task decomposition2. Baseline for correctness3. Criticality Testing4. Distortion and timing models
![Page 100: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/100.jpg)
Confidential + ProprietaryJeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010
![Page 101: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/101.jpg)
Confidential + ProprietaryJeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010
![Page 102: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/102.jpg)
Confidential + Proprietary
![Page 103: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/103.jpg)
Confidential + Proprietary
![Page 104: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/104.jpg)
Confidential + Proprietary
Robust & scaleable pipelines
![Page 105: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/105.jpg)
Confidential + Proprietary
Robust & scaleable pipelinesLeases for sharing & heartbeat
![Page 106: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/106.jpg)
Confidential + Proprietary
Robust & scaleable pipelinesLeases for sharing & heartbeat
Trade off inaccuracy for resilience & performance
![Page 107: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/107.jpg)
Confidential + Proprietary
Robust, scaleable pipelinesLeases for sharing & heartbeat
Trade off inaccurate for resilient
CS research is timeless:
use it to mitigate risk
![Page 108: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/108.jpg)
Confidential + Proprietary
ThanksInes SombraDavid GreenbergAlex HutchesonScott ZawalskiKaran Parikh
![Page 109: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/109.jpg)
Confidential + Proprietary
References● T. Wurthinger, C. Wimmer et al. "One VM to Rule Them All"● M. Rinard "Probabilistic Accuracy Bounds for Fault-Tolerant
Computations that Discard Tasks"● F. Corbato, M. Daggett, R. Daley "An Experimental Time-
Sharing System"● E. Dijkstra "Cooperating Sequential Processes"● L. Lamport "Time, Clocks, and the Ordering of Events in a
Distributed System"
![Page 110: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/110.jpg)
Confidential + Proprietary
References● B. Oki, B. Liskov "Viewstamped Replication: A New Primary Copy
Method to Support Highly-Available Distributed Systems"● L. Lamport "The Part-Time Parliament"● M. Welsh, D. Culler, E. Brewer "SEDA: An Architecture for Well-
Conditioned, Scalable Internet Services"● C. Gray, D. Cheriton "Leases: An Efficient Fault-Tolerant
Mechanism for Distributed File Cache Consistency"● S. Agarwal, B. Mozafari et al. "BlinkDB: Queries with Bounded
Errors and Bounded Response Times on Very Large Data"
![Page 111: March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary Confidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March](https://reader035.vdocuments.us/reader035/viewer/2022070823/5f29ad1816bd21798b479bd2/html5/thumbnails/111.jpg)
Confidential + Proprietary
Should I read papers?
YES