Download - Machine Learning at Netflix Scale
![Page 1: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/1.jpg)
![Page 2: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/2.jpg)
Machine Learning At Netflix Scale
Aish Fenton Manager - Research Engineering @aishfenton
![Page 3: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/3.jpg)
Everything is a recommendation
![Page 4: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/4.jpg)
4
![Page 5: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/5.jpg)
Top Picks for Aish
![Page 6: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/6.jpg)
Movies based on books
![Page 7: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/7.jpg)
Because you watched Bob’s Burgers
![Page 8: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/8.jpg)
![Page 9: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/9.jpg)
Rank based on your taste
Ran
k ba
sed
on y
our
tast
e
![Page 10: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/10.jpg)
75% of plays come from homepage
![Page 11: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/11.jpg)
Back Story…
![Page 12: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/12.jpg)
![Page 13: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/13.jpg)
Proxy question: ▪ Accuracy in predicted rating ▪ Improve by 10% = $1million!
What we were interested in: ▪ High quality recommendations
predicted
actual
![Page 14: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/14.jpg)
SVD RBMs
Top two results still used in production!
![Page 15: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/15.jpg)
>
![Page 16: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/16.jpg)
2006 2013
![Page 17: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/17.jpg)
• > 44M members
• > 40 countries
• > 5B hours in Q3 2013
• Log 100B events/day
• 31.62% of peak US downstream traffic
![Page 18: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/18.jpg)
Data and Models
![Page 19: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/19.jpg)
▪ > 40M subscribers ▪ Ratings: ~5M/day ▪ Searches: >3M/day ▪ Plays: > 50M/day ▪ Streamed hours: o 5B hours in Q3 2013
Geo Info
Time
Impressions
Device Info
Metadata
Social
Ratings
Demographics
Member Behavior
Plays
![Page 20: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/20.jpg)
Aish House of Cards
Latent User Vector
Latent Item Vector
![Page 21: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/21.jpg)
3.53
RU
M
u1 u2 u3
m1 !m2!m3
House of Cards
Aish Aish
House of Cards
![Page 22: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/22.jpg)
![Page 23: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/23.jpg)
Mean Rating My Bias
Movie Bias
Interaction
![Page 24: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/24.jpg)
Mean Rating My Bias
Movie Bias
Interaction
3.55 = 2.50 + -1.5 + 1.2 + pq
My rating for House of Cards
![Page 25: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/25.jpg)
R3.53
U
M
u1 u2 u3
m1 !m2!m3
House of Cards
Aish
2.35
1.34
Time
T
t1 t2 t3 Time
![Page 26: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/26.jpg)
▪ Matrix/Tensor Factorization ▪ Regression models (Logistic, Linear, Elastic nets) ▪ Factorization Machines ▪ Restricted Boltzmann Machines ▪ Markov Chains & other graph models ▪ Clustering / Topic Models ▪ Neural Networks ▪ Association Rules ▪ GBDT/RF ▪ …
![Page 27: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/27.jpg)
Popularity
+ Ratings
+ More Features & Optimized Models
0% 50%
100%
150%
200%
250%
300%
Improvement Over Baseline
![Page 28: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/28.jpg)
Anatomy of a Machine Learning
Platform
![Page 29: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/29.jpg)
Problem
Data
Experiment Offline
Produce Model
Test / Metrics
![Page 30: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/30.jpg)
Near-line
Online
UI Clients
Event Distribution
Online Algs
Model Trainer
Pre-compute
AB Test Metrics
API Layer
Monitoring
Offline
Hadoop / Data Warehouse
Experimentation Platform
S3 / HDFS
Offline MetricsQuery Tools
Models
Models
![Page 31: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/31.jpg)
Near-line
Online
UI Clients
Event Distribution
Online Algs
Model Trainer
Pre-compute
AB Test Metrics
API Layer
Monitoring
Offline
Hadoop / Data Warehouse
Experimentation Platform
S3 / HDFS
Offline MetricsQuery Tools
Models
Models
![Page 32: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/32.jpg)
▪ App Logs ▪ User Actions
▪ Ratings ▪ Plays ▪ Queue Adds
▪ Algo Actions ▪ Impressions (Presentation Bias)
▪ Context ▪ Device Info ▪ User Demographics ▪ Social ▪ Time
▪ …
Many different types of data…
![Page 33: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/33.jpg)
![Page 34: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/34.jpg)
![Page 35: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/35.jpg)
Near-line
Online
UI Clients
Event Distribution
Online Algs
Model Trainer
Pre-compute
AB Test Metrics
API Layer
Monitoring
Offline
Hadoop / Data Warehouse
Experimentation Platform
S3 / HDFS
Offline MetricsQuery Tools
Models
Models
Embedded
Embedded
![Page 36: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/36.jpg)
Weights
Real-time popularity of movie
![Page 37: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/37.jpg)
Example: Neural Network Training
![Page 38: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/38.jpg)
![Page 39: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/39.jpg)
θ
Input OutputHidden Layer
![Page 40: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/40.jpg)
Input OutputHidden Layers
![Page 41: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/41.jpg)
Neural Network Training
1,536 cores
G2 Instances $0.60 p/h
![Page 42: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/42.jpg)
But… things can go astray
![Page 43: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/43.jpg)
![Page 44: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/44.jpg)
![Page 45: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/45.jpg)
Near-line
Online
UI Clients
Event Distribution
Online Algs
Model Trainer
Pre-compute
AB Test Metrics
API Layer
Monitoring
Offline
Hadoop / Data Warehouse
Experimentation Platform
S3 / HDFS
Offline MetricsQuery Tools
Models
Models
![Page 46: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/46.jpg)
RU
MPre-compute
u1 u2 u3Online
![Page 47: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/47.jpg)
Near-line
Online
UI Clients
Event Distribution
Online Algs
Model Trainer
Pre-compute
AB Test Metrics
API Layer
Monitoring
Offline
Hadoop / Data Warehouse
Experimentation Platform
S3 / HDFS
Offline MetricsQuery Tools
Models
Models
Aish played HoC
Publish new model for Aish
![Page 48: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/48.jpg)
Aish Fenton @aishfenton https://www.linkedin.com/profile/view?id=47917219