machine learning at netflix scale
DESCRIPTION
Netflix is the world’s leading Internet television network with over 48 million members in more than 40 countries enjoying more than one billion hours of TV shows and movies per month, including original series. Netflix uses machine learning to deliver a personalized experience to each one of our 48 million users. In this talk you will hear about the machine learning algorithms that power almost every part of the Netflix experience, including some of our recent work on distributed Neural Networks on AWS GPUs. You will also get an insight into the innovation approach that includes offline experimentation and online AB testing. Finally, you will learn about the system architectures that enable all of this at a Netflix scale.TRANSCRIPT
![Page 1: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/1.jpg)
![Page 2: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/2.jpg)
Machine Learning At Netflix Scale
Aish Fenton Manager - Research Engineering @aishfenton
![Page 3: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/3.jpg)
Everything is a recommendation
![Page 4: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/4.jpg)
4
![Page 5: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/5.jpg)
Top Picks for Aish
![Page 6: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/6.jpg)
Movies based on books
![Page 7: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/7.jpg)
Because you watched Bob’s Burgers
![Page 8: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/8.jpg)
![Page 9: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/9.jpg)
Rank based on your taste
Ran
k ba
sed
on y
our
tast
e
![Page 10: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/10.jpg)
75% of plays come from homepage
![Page 11: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/11.jpg)
Back Story…
![Page 12: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/12.jpg)
![Page 13: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/13.jpg)
Proxy question: ▪ Accuracy in predicted rating ▪ Improve by 10% = $1million!
What we were interested in: ▪ High quality recommendations
predicted
actual
![Page 14: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/14.jpg)
SVD RBMs
Top two results still used in production!
![Page 15: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/15.jpg)
>
![Page 16: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/16.jpg)
2006 2013
![Page 17: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/17.jpg)
• > 44M members
• > 40 countries
• > 5B hours in Q3 2013
• Log 100B events/day
• 31.62% of peak US downstream traffic
![Page 18: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/18.jpg)
Data and Models
![Page 19: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/19.jpg)
▪ > 40M subscribers ▪ Ratings: ~5M/day ▪ Searches: >3M/day ▪ Plays: > 50M/day ▪ Streamed hours: o 5B hours in Q3 2013
Geo Info
Time
Impressions
Device Info
Metadata
Social
Ratings
Demographics
Member Behavior
Plays
![Page 20: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/20.jpg)
Aish House of Cards
Latent User Vector
Latent Item Vector
![Page 21: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/21.jpg)
3.53
RU
M
u1 u2 u3
m1 !m2!m3
House of Cards
Aish Aish
House of Cards
![Page 22: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/22.jpg)
![Page 23: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/23.jpg)
Mean Rating My Bias
Movie Bias
Interaction
![Page 24: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/24.jpg)
Mean Rating My Bias
Movie Bias
Interaction
3.55 = 2.50 + -1.5 + 1.2 + pq
My rating for House of Cards
![Page 25: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/25.jpg)
R3.53
U
M
u1 u2 u3
m1 !m2!m3
House of Cards
Aish
2.35
1.34
Time
T
t1 t2 t3 Time
![Page 26: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/26.jpg)
▪ Matrix/Tensor Factorization ▪ Regression models (Logistic, Linear, Elastic nets) ▪ Factorization Machines ▪ Restricted Boltzmann Machines ▪ Markov Chains & other graph models ▪ Clustering / Topic Models ▪ Neural Networks ▪ Association Rules ▪ GBDT/RF ▪ …
![Page 27: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/27.jpg)
Popularity
+ Ratings
+ More Features & Optimized Models
0% 50%
100%
150%
200%
250%
300%
Improvement Over Baseline
![Page 28: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/28.jpg)
Anatomy of a Machine Learning
Platform
![Page 29: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/29.jpg)
Problem
Data
Experiment Offline
Produce Model
Test / Metrics
![Page 30: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/30.jpg)
Near-line
Online
UI Clients
Event Distribution
Online Algs
Model Trainer
Pre-compute
AB Test Metrics
API Layer
Monitoring
Offline
Hadoop / Data Warehouse
Experimentation Platform
S3 / HDFS
Offline MetricsQuery Tools
Models
Models
![Page 31: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/31.jpg)
Near-line
Online
UI Clients
Event Distribution
Online Algs
Model Trainer
Pre-compute
AB Test Metrics
API Layer
Monitoring
Offline
Hadoop / Data Warehouse
Experimentation Platform
S3 / HDFS
Offline MetricsQuery Tools
Models
Models
![Page 32: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/32.jpg)
▪ App Logs ▪ User Actions
▪ Ratings ▪ Plays ▪ Queue Adds
▪ Algo Actions ▪ Impressions (Presentation Bias)
▪ Context ▪ Device Info ▪ User Demographics ▪ Social ▪ Time
▪ …
Many different types of data…
![Page 33: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/33.jpg)
![Page 34: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/34.jpg)
![Page 35: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/35.jpg)
Near-line
Online
UI Clients
Event Distribution
Online Algs
Model Trainer
Pre-compute
AB Test Metrics
API Layer
Monitoring
Offline
Hadoop / Data Warehouse
Experimentation Platform
S3 / HDFS
Offline MetricsQuery Tools
Models
Models
Embedded
Embedded
![Page 36: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/36.jpg)
Weights
Real-time popularity of movie
![Page 37: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/37.jpg)
Example: Neural Network Training
![Page 38: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/38.jpg)
![Page 39: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/39.jpg)
θ
Input OutputHidden Layer
![Page 40: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/40.jpg)
Input OutputHidden Layers
![Page 41: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/41.jpg)
Neural Network Training
1,536 cores
G2 Instances $0.60 p/h
![Page 42: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/42.jpg)
But… things can go astray
![Page 43: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/43.jpg)
![Page 44: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/44.jpg)
![Page 45: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/45.jpg)
Near-line
Online
UI Clients
Event Distribution
Online Algs
Model Trainer
Pre-compute
AB Test Metrics
API Layer
Monitoring
Offline
Hadoop / Data Warehouse
Experimentation Platform
S3 / HDFS
Offline MetricsQuery Tools
Models
Models
![Page 46: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/46.jpg)
RU
MPre-compute
u1 u2 u3Online
![Page 47: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/47.jpg)
Near-line
Online
UI Clients
Event Distribution
Online Algs
Model Trainer
Pre-compute
AB Test Metrics
API Layer
Monitoring
Offline
Hadoop / Data Warehouse
Experimentation Platform
S3 / HDFS
Offline MetricsQuery Tools
Models
Models
Aish played HoC
Publish new model for Aish
![Page 48: Machine Learning at Netflix Scale](https://reader035.vdocuments.us/reader035/viewer/2022062404/554a0764b4c905507a8b55ac/html5/thumbnails/48.jpg)
Aish Fenton @aishfenton https://www.linkedin.com/profile/view?id=47917219