api world 2013 - transforming the netflix api
DESCRIPTION
The Netflix API has undergone a transformation since its inception in 2008. It has transitioned from being a public API with a generic RESTful interface to a platform for creating highly optimized, device-centric APIs that are critical to delivering the Netflix streaming experience on over 1000 different device types. This talk covers the design principles that shaped the transformation of the API as well as the technology that powers it, enabling rapid user experience iteration and bringing Netflix streaming to almost 38 million subscribers around the world.TRANSCRIPT
![Page 1: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/1.jpg)
Transforming the Netflix API
Ben Schmaus, NetflixOctober 2013, API World
[email protected] || @schmaus
![Page 2: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/2.jpg)
Streaming TV Shows & Movies Globally
![Page 3: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/3.jpg)
> 1000 Devices
![Page 4: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/4.jpg)
1/3 ofInternet at peak
![Page 5: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/5.jpg)
ExclusiveContent
![Page 6: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/6.jpg)
Almost 38 million subscribers in over
40 countries
![Page 7: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/7.jpg)
PersonalizationEngine User Info Movie
Metadata Ratings SimilarMovies
InstantQueue
A/B TestEngine
API
![Page 8: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/8.jpg)
PersonalizationEngine User Info Movie
Metadata Ratings SimilarMovies
InstantQueue
A/B TestEngine
APIEnable UX Innovation
Insulate from Failure
![Page 9: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/9.jpg)
It wasn’t always this way
![Page 10: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/10.jpg)
FosterInnovation
fromOutside
![Page 11: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/11.jpg)
REST for Easy Interop, Integration
RESTful API
![Page 12: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/12.jpg)
REST for Easy Interop, Integration
RESTful API
Model resources like users, movies, series, ratings, etc
![Page 13: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/13.jpg)
REST for Easy Interop, Integration
RESTful API
/catalog/titles/catalog/titles/movies/catalog/titles/series
/catalog/titles/series/episodes
![Page 14: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/14.jpg)
REST for Easy Interop, Integration
RESTful API
/users/lists/users/title_states
/users/ratings
![Page 15: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/15.jpg)
EnterStreaming
Devices
![Page 16: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/16.jpg)
![Page 17: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/17.jpg)
Lots of devices, lots of variety across platforms
![Page 18: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/18.jpg)
![Page 19: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/19.jpg)
External apps
![Page 20: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/20.jpg)
1 /apps/{app_id}/config1 /users/{user_id}1 /users/{user_id}/queues/instant/available1 /users/{user_id}/lists26 /catalog/titles?...
![Page 21: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/21.jpg)
Doesn’t include JS, CSS, images, etc.
1 /apps/{app_id}/config1 /users/{user_id}1 /users/{user_id}/queues/instant/available1 /users/{user_id}/lists26 /catalog/titles?...
![Page 22: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/22.jpg)
Device variance: interaction models
![Page 23: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/23.jpg)
Device variance: interaction models & form factors
![Page 24: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/24.jpg)
A/B Tests Add to the Challenge
![Page 25: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/25.jpg)
“Bolt on” functionality
expand=@queue_item@title,@box_art_exists,@short_synopsis,@directors,@cast,@episodes,@formats,@format_availability,@subtitle_languages,@languages_and_audio,@maturity_ratings
![Page 26: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/26.jpg)
Resource-based API wasn’t scaling
![Page 27: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/27.jpg)
Model Resources/catalog, /users
Model Experiences/tv/home
![Page 28: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/28.jpg)
Reduce network chattiness
Support device optimizations
Enable fast dev iterations
![Page 29: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/29.jpg)
GET/users/{user_id}/lists
apiGateway.getLists(userId)
![Page 30: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/30.jpg)
![Page 31: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/31.jpg)
Discrete HTTP requests pay WAN tax repeatedly
![Page 32: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/32.jpg)
![Page 33: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/33.jpg)
Single, optimized request; pay WAN tax once
![Page 34: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/34.jpg)
Single, optimized request; pay WAN tax once
Client data assembly logic pushed to server
![Page 35: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/35.jpg)
Add server-side scripting capability
Enable independent dev and device optimization
![Page 36: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/36.jpg)
![Page 37: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/37.jpg)
API
Internet /tv/home
![Page 38: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/38.jpg)
API
/tv/home
Client Server
Internet
![Page 39: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/39.jpg)
API
Application
Client Server
Internet /tv/home
![Page 40: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/40.jpg)
API
Client Server
Internet
UI Teams
Application
/tv/home
![Page 41: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/41.jpg)
API
API Team
Client Server
Internet
UI Teams
Application
/tv/home
![Page 42: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/42.jpg)
APIUI
Teams
API Team
Mid-tierServices
Client Server
Internet
Application
/tv/home
![Page 43: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/43.jpg)
RxJava Hystrix
API Team
Java Service Layer
Mid-tierServices
UI Teams
Client Server
Internet
Application
/tv/home
![Page 44: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/44.jpg)
ELB ZuulMid-tier Services
ScriptableBackend
ScriptableBackend
+
API Layer
![Page 45: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/45.jpg)
https://github.com/Netflix/Hystrixresilience patterns for distributed systems
https://github.com/Netflix/RxJavaasync, event-based programming
https://github.com/Netflix/zuuldynamic request routing, realtime monitoring, resiliency
![Page 46: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/46.jpg)
Need well-appointed toolkit
![Page 47: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/47.jpg)
Cassandra
upload script.groovyfor /script
![Page 48: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/48.jpg)
Cassandra
Cassandra
activate /script
upload script.groovyfor /script
![Page 49: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/49.jpg)
Cassandra
Cassandra
activate /script
CassandraScriptableBackend
load /script
upload script.groovyfor /script
![Page 50: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/50.jpg)
Cassandra
Cassandra
activate /script
CassandraScriptableBackend
load /script
upload script.groovyfor /script
ScriptableBackend
DeviceClient
GET/script
![Page 51: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/51.jpg)
![Page 52: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/52.jpg)
![Page 53: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/53.jpg)
Deployment & Ops
![Page 54: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/54.jpg)
Historical & realtime metrics, sort realtime by error/request rate
![Page 55: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/55.jpg)
Distributed grep + tail
2013-05-09.20:38:54 MX 200 us-east-1c i-1824cb73 i-1c61b77f prod NFPS3-001-8G50FJCX... 288404769389848058 90ms api-global.netflix.com GET /tvui/release/470/plus/pathEvaluator -amazon.ami-id: ami-502eb039amazon.availability-zone: us-east-1camazon.instance-id: i-1824cb73amazon.instance-type: m2.2xlargeamazon.local-ipv4: 10.6.213.112amazon.public-hostname: ec2-54-243-4-69.compute-1.amazonaws.comamazon.public-ipv4: 54.243.4.69cookie_esn: NFPS3-001-8G50FJCX...country: MXcurrentTime: 1368131934468duration-millis: 90esn: NFPS3-001-8G50FJCX...geo.city: CIUDADOBREGON...
$ ./simple_stream.py -f -q 'e["country"]=="MX" && e["esn"]==~/NFPS3.*/' -r us
![Page 56: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/56.jpg)
Go for haystack handing you the needle
![Page 57: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/57.jpg)
Or at least be able to make smaller haystacks
![Page 58: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/58.jpg)
![Page 59: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/59.jpg)
Run 1% of your traffic on the new code and see how it does
![Page 60: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/60.jpg)
API ami-123 API ami-456
2xx4xx5xx
latencybusy threads
load...
![Page 61: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/61.jpg)
Confidence score for each AMI based on comparison of 1000+ metrics
![Page 62: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/62.jpg)
Scannable visualization of metric space
More important
Less important
![Page 63: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/63.jpg)
Your basic red/black push
![Page 64: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/64.jpg)
![Page 65: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/65.jpg)
![Page 66: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/66.jpg)
![Page 67: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/67.jpg)
![Page 68: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/68.jpg)
Doing red/black by hand for multiple clusters across multiple regions is
not fun
![Page 69: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/69.jpg)
Automate multi-cluster/region pushes
![Page 70: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/70.jpg)
Automate multi-cluster/region pushes
Don't forget to automate
rollbacks, too!
![Page 71: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/71.jpg)
$Who, $What, $Where, $When
e.g., "bschmaus, ami-123, Sandbox Canary, 2013-05-06 19:05"
Latest prod change in chat topic
![Page 72: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/72.jpg)
Quickly see status of all clusters in a region
![Page 73: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/73.jpg)
API for APIs
Control network interactions
Device optimization
Development agility
Automation & insights
![Page 74: API World 2013 - Transforming the Netflix API](https://reader033.vdocuments.us/reader033/viewer/2022061223/54c6a91f4a7959aa4b8b4571/html5/thumbnails/74.jpg)
Continuously experiment to make hard things easier