1 the netflix data platform – a recipe for high business impact kurt brown, director, data...
TRANSCRIPT
![Page 1: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/1.jpg)
1
The Netflix Data Platform – A Recipe for High Business ImpactKurt Brown, Director, Data Platform
![Page 2: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/2.jpg)
Agenda
• Architecture
• Cloud
• OSS
• Philosophy
![Page 3: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/3.jpg)
Architecture
![Page 4: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/4.jpg)
![Page 5: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/5.jpg)
S3
![Page 6: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/6.jpg)
~5 PB
S3
![Page 7: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/7.jpg)
![Page 8: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/8.jpg)
![Page 9: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/9.jpg)
![Page 10: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/10.jpg)
vs.
![Page 11: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/11.jpg)
![Page 12: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/12.jpg)
![Page 13: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/13.jpg)
![Page 14: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/14.jpg)
![Page 15: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/15.jpg)
![Page 16: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/16.jpg)
![Page 17: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/17.jpg)
![Page 18: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/18.jpg)
![Page 19: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/19.jpg)
![Page 20: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/20.jpg)
Sting
![Page 21: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/21.jpg)
![Page 22: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/22.jpg)
S3
Sting
Data Platform
![Page 23: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/23.jpg)
Tez
/ Shark
![Page 24: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/24.jpg)
?
S3
Data Platform
TezYarn
???
Sting
![Page 25: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/25.jpg)
Cloud
![Page 26: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/26.jpg)
Yes
![Page 27: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/27.jpg)
OSS
![Page 28: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/28.jpg)
![Page 29: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/29.jpg)
Cost
![Page 30: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/30.jpg)
Cost
![Page 31: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/31.jpg)
Flexibility
• Growth
• Spikes
• Self-service
![Page 32: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/32.jpg)
Services
• SQS, EMR, S3,…
• Available
• Focus!
![Page 33: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/33.jpg)
Redundancy / Global Presence
• Multi-DC
• International
![Page 34: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/34.jpg)
Cloud Native
• Build for failure
• Swing systems
• Excess capacity
![Page 35: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/35.jpg)
Right for everyone?
• Are you Google or Facebook?
• Netflix
• Undifferentiated
• Opportunity Cost
![Page 36: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/36.jpg)
Cost?
![Page 37: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/37.jpg)
OSS
![Page 38: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/38.jpg)
General OSS
• Pooled goodness
• Manage own destiny
• Vendor relationships
• Conflicting agendas
![Page 39: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/39.jpg)
Netflix OSS
• Meet Netflix needs
• Shared direction
• Recruiting & Retention
• Code quality
• Contributions
![Page 40: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/40.jpg)
Data Platform OSS
• Genie
• Lipstick
• Aegisthus
• S3mper
• Suro
• Future: Inviso
![Page 41: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/41.jpg)
Should you open source?
• We’re all using it
• You get value
• Others would get value
• Will invest some time
![Page 42: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/42.jpg)
Philosophy
![Page 43: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/43.jpg)
Netflix Expense Policy
“Act in Netflix’s Best Interest”
![Page 44: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/44.jpg)
Development & Deployment Flow
BestPractices Components
ConsultationTools
Automation
Clean-up
![Page 45: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/45.jpg)
DBAs?
![Page 46: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/46.jpg)
QA Team?
![Page 47: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/47.jpg)
Testing
![Page 48: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/48.jpg)
Accept things WILL break
![Page 49: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/49.jpg)
Safety Nets
![Page 50: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/50.jpg)
Give folks what they need
![Page 51: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/51.jpg)
Eliminate Process
• Status Reports
• Tickets
• Documentation
![Page 52: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/52.jpg)
Tell vs. Ask
![Page 53: 1 The Netflix Data Platform – A Recipe for High Business Impact Kurt Brown, Director, Data Platform](https://reader034.vdocuments.us/reader034/viewer/2022051621/56649c7d5503460f94931d3a/html5/thumbnails/53.jpg)
Does it really matter?