![Page 1: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/1.jpg)
(one way to think of)
Operational Insight Roy Rapoport
@royrapoport or [email protected] March 30, 2017
![Page 2: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/2.jpg)
Insight Engineering @ Netflix
• 17 + 3 people
• Two groups
• Largest Single Cloud Spend Item
Real-Time Operational Insight
![Page 3: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/3.jpg)
What does that mean?
![Page 4: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/4.jpg)
What does that mean?
![Page 5: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/5.jpg)
What does that mean?
![Page 6: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/6.jpg)
What does that mean?
![Page 7: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/7.jpg)
What does it all mean?
![Page 8: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/8.jpg)
![Page 9: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/9.jpg)
John Boyd
![Page 10: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/10.jpg)
Observe
Orient
Decide
Act OODA
“This approach favors agility over raw power in dealing with human opponents in any endeavor” - Wikipedia
![Page 11: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/11.jpg)
This Is What We Do
![Page 12: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/12.jpg)
Observe
Orient
Decide
Act OODA
![Page 13: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/13.jpg)
Observe
Orient
Decide
Act OODA
Telemetry platforms
![Page 14: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/14.jpg)
Observe
Orient
Decide
Act OODA
Telemetry platforms
Data Viz Graphs
Dashboards
![Page 15: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/15.jpg)
Observe
Orient
Decide
Act OODA
Telemetry platforms
Data Viz Graphs
Dashboards
Alerting Decision-Output Systems
![Page 16: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/16.jpg)
Observe
Orient
Decide
Act OODA
Telemetry platforms
Data Viz Graphs
Dashboards
Alerting Decision-Output Systems
![Page 17: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/17.jpg)
Observe
Orient
Decide
Act OODA
Telemetry platforms
Data Viz Graphs
Dashboards
Alerting Decision-Output Systems
Remediation
![Page 18: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/18.jpg)
Observe
Orient
Decide
Act OODA
Telemetry platforms
Data Viz Graphs
Dashboards
Alerting Decision-Output Systems
Remediation
![Page 19: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/19.jpg)
OODA* AT
![Page 20: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/20.jpg)
Observe• Atlas
• Time Series Database
• Up to ~3B time series
• 6H, 3D, 18D tiers
• Indefinite Retention
• Custom
• Expensive
![Page 21: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/21.jpg)
Observe
• Chronos
• What the Hell Changed?
• Change Reporting System
• Entirely Automated
• ~18 months*
• REST on top of ElasticSearch
![Page 22: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/22.jpg)
![Page 23: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/23.jpg)
Observe
• Salp
• Based on Dapper
• Distributed Trace
• Dependency Discovery
![Page 24: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/24.jpg)
Orient• Already Showed Some
• Lumen: Extensible Dashboards
• JSON Construction
• Surprising Use Cases
• Easily Extensible
• Data Sources
• Visualizations
![Page 25: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/25.jpg)
Decide
• Alerting: Simple
![Page 26: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/26.jpg)
Decide
• Alerting: Simple
• Alerting: Not Simple
sps_all,nf.region,(,us-east-1,),:in,name,COUNTER-playback-action-APIPlaybackStartAction,:eq, :and,isSupplementalVideo,false,:eq,:and,operation,onComplete,:eq,:and,device.operationalName, streaming_stick,,:eq,:and,:sum,:set,sps_1h_offset,sps_all,:get,1h,:offset,:set,entering_trough,sps_1h_ offset,:get,0.95,:mul,sps_all,:get,:gt,:set,smoothed,sps_all,:get,10,0.1,0.02,:des,:set,low_volume, smoothed,:get,-0.005,:mul,0.1,:add,:set,mid_volume,smoothed,:get,-0.00125,:mul,0.1,:add,:set,base, 0.06,:set,min_pct,1,smoothed,:get,20,:lt,low_volume,:get,:mul,smoothed,:get,80,:lt,mid_volume,:get, :mul,:add,entering_trough,:get,0.05,:mul,:add,base,:get,:add,:sub,10,0.1,0.02,:des,:set,sps_all,: get,min_pct,:get,smoothed,:get,:mul,:lt,10,:rolling-count,5,:ge,$device.operationalName Alert Trigger, :legend
![Page 27: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/27.jpg)
Decide
• Alerting: Simple
• Alerting: Not Simple
• Kepler: Outlier Detection
![Page 28: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/28.jpg)
Decide
• Alerting: Simple
• Alerting: Not Simple
• Kepler: Outlier Detection
![Page 29: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/29.jpg)
Decide
• Alerting: Simple
• Alerting: Not Simple
• Kepler: Outlier Detection
• Automated Canary Analysis
![Page 30: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/30.jpg)
Decide• Alerting: Simple
• Alerting: Not Simple
• Kepler: Outlier Detection
• Automated Canary Analysis
• Active:
• UVAD
• MVAD
![Page 31: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/31.jpg)
Act
![Page 32: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/32.jpg)
Act
![Page 33: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/33.jpg)
Act
![Page 34: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/34.jpg)
Observe
Orient
Decide
Act OODA
Telemetry platforms
Data Viz Graphs
Dashboards
Alerting Outlier Detection
Anomaly Detection Canary Analysis
Remediation
![Page 35: A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix](https://reader034.vdocuments.us/reader034/viewer/2022042707/58ec1a391a28ab4c508b4753/html5/thumbnails/35.jpg)
?