building our own cdn
DESCRIPTION
TRANSCRIPT
![Page 1: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/1.jpg)
Building our own CDN
How to serve millions of viewers efficiently
Zoltán Németh
![Page 2: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/2.jpg)
at a glance Live streaming
Free broadcastingSoftware as a Service
Big events• Sony Playstation• Nintendo• Lady Gaga• Festivals
Citizen journalism• Rescue of Chilean miners• Earthquake in Japan• Revolutions in Egypt, Syria• Protests in Ukraine
• Obama campaign• Climate Reality• Football• Justin Bieber
Concurrent viewership peaks at almost 2 million
![Page 3: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/3.jpg)
Streaming Definitions
Broadcaster, viewer
Media
Buffering
History
FMS, RTMP
Edge
UHS, HTTP
![Page 4: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/4.jpg)
How we use the network
Live streaming properties
Continuous connection vs
chunk download
Big bandwidth
Stability critical
Large audience
Global delivery
Providers performance
Mobile networks
![Page 5: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/5.jpg)
First approach: CDN
Pros
Scaling, flexibility
Bigger capacity
Cons
Costs
Origin requests
Limitations
Regional and provider
differences
Capacity race conditions
![Page 6: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/6.jpg)
Multiple CDNs Global coverage solved
Costs high
Unified configuration problem
How to select which one to use
![Page 7: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/7.jpg)
Own capacity Why
Cost-effectiveness
Keep CDNs for peaks
Transit lines
Peering
Billing model
95% / Flat rate
![Page 8: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/8.jpg)
How to utilize own network
Basic decision logic
Peering first
Transit second
CDNs last
Cache layer
Varnish
Monitoring
POPs, Edge clusters
![Page 9: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/9.jpg)
Viewer side switching - QoS
List all providers
In case of problem – switch
Challenges:
How to detect a problem
Sync
Broadcaster side problems
![Page 10: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/10.jpg)
Server side prediction
Initial guess for QoS
Minimize useless switches (UX)
Cost optimization
How?
Viewers report metrics
Server side processing and
aggregation
![Page 11: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/11.jpg)
UMS Ustream Media Server
Continuous connection to all
clients
Real-time push-based updates
First use: viewer number
display
Channel status poll
Viewer authentication and
authorization
Stream information
Java, kernel tweaks
![Page 12: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/12.jpg)
UMS Frontend layer
Up to 150.000 clients /
machine
3 IP addresses per machine
1.5 billion outgoing messages
per machine per day (10
billion for the live cluster)
Aggregation / logic layer
Current: 400.000 clients per
media limit
Future: distributed
![Page 13: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/13.jpg)
The data source: Streamstat
Viewer reports
UMS process
Log
Historic data processing
From log
Hadoop + Tableau
Realtime processing in UMS
Chandra
{"appId":1,"appVersion":1,"application":"channel","as":"AS3462 Data Communication Business Group","brandId":"1","city":"Taipei","clientIP":"220.142.6.108","clientId":1421873992,"country":"TW","createTime":1399223703493,"eventType":"STAT_UPDATE","mediaId":"17913321","rpin":"rpin.07680805512286006","rsid":"fvwfnxzx:y1wowjfr","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProtocol":"http","size":471654,"streamChunkHash":"1108464946","streamChunkId":"1399223036","streamName":"live_1","streamNetworkProvider":"ntt","streamProtocol":"uhs","time":405,"url":"http://sjc-ucdn03.ntt.tcdn.ustream.tv/sjc-uhs22/streams/httpflv/ustreamVideo/17913321/streams/live_1_1399223036_1108464946.flv"}]}},"swfUrl":"http://static-cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931444,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://www.ustream.tv/channel/thdudf4"}
{"appId":1,"appVersion":1,"application":"channel","as":"AS15377 ISP Fregat Ltd.","brandId":"1","city":"Dnepropetrovsk","clientIP":"46.98.73.39","clientId":1188395153,"country":"UA","createTime":1399224734039,"eventType":"STAT_UPDATE","mediaId":"13166013","rpin":"rpin.23043611550692003","rsid":"0qx26uw2:671wyhbv","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"akamai","networkProtocol":"http","size":214689,"streamChunkHash":"225119325","streamChunkId":"1399223580","streamName":"live_1","streamProtocol":"uhs","time":241,"url":"http://uhs-akamai.ustream.tv/ams/ams-uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223580_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":180117,"streamChunkHash":"225119325","streamChunkId":"1399223581","streamName":"live_1","streamProtocol":"uhs","time":259,"url":"http://uhs-akamai.ustream.tv/ams/ams-uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223581_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":198214,"streamChunkHash":"225119325","streamChunkId":"1399223582","streamName":"live_1","streamProtocol":"uhs","time":164,"url":"http://uhs-akamai.ustream.tv/ams/ams-uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223582_225119325.flv"},{"cdnProvider":"level3","networkProtocol":"http","size":192575,"streamChunkHash":"225119325","streamChunkId":"1399223579","streamName":"live_1","streamProtocol":"uhs","time":6044,"url":"http://uhs-level3.ustream.tv/ams/ams-uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223579_225119325.flv"}]},"common":{"pvdname":"uhs_akamai"}},"swfUrl":"http://static-cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 11,6,602,180","webUrl":"http://www.ustream.tv/channel/new-odessa"}
{"appId":1,"appVersion":1,"application":"channel","as":"AS5615 Koninklijke KPN N.V.","brandId":"1","city":"Eindhoven","clientIP":"82.170.189.202","clientId":469374750,"country":"NL","createTime":1399224671784,"eventType":"STAT_UPDATE","mediaId":"17926037","rpin":"rpin.5570720779755052","rsid":"4uv3mjhk:naytle8n","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProtocol":"http","size":343420,"streamChunkHash":"118845438","streamChunkId":"1399206465","streamName":"live_1","streamNetworkProvider":"ntt","streamProtocol":"uhs","time":2083,"url":"http://sjc-ucdn05.ntt.tcdn.ustream.tv/ams-uhs03/streams/httpflv/ustreamVideo/17926037/streams/live_1_1399206465_118845438.flv"}]}},"swfUrl":"http://static-cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://goodcast.tv/e/5.html"}
![Page 14: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/14.jpg)
Streamstat reports
Types of reports
Playing
Buffering
Bandwidth
Errors, QoS events
Verticals
Geo
AS number
Media
![Page 15: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/15.jpg)
Chandra Real-time data aggregation layer
Replaceable data store
Redis
Simple logic: based on increments
Easily scalable
![Page 16: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/16.jpg)
Chandra demo
![Page 17: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/17.jpg)
Decisions Ways of routing viewers
Initial setup
Updates
Stream priority
Stream module
Ways of decision
Rule-based
Dynamic
![Page 18: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/18.jpg)
Stream module generation flow
Filter the Stream Priority list
Dedications / whitelists
![Page 19: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/19.jpg)
Stream module generation flow
Filter the Stream Priority list
Dedications / whitelists
![Page 20: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/20.jpg)
Stream module generation flowRule blocking UCDN
![Page 21: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/21.jpg)
Routing on own network
Provider resolution service
Reject to CDN if full
Capacity monitoring
Lines
Clusters
![Page 22: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/22.jpg)
What we have now
3 big CDN providers
~5 smaller providers
UMS clusters
Core in SJC
POPs: NRT, AMS, BUD
Exchanges
QoS and static rules
![Page 23: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/23.jpg)
![Page 24: Building our own CDN](https://reader033.vdocuments.us/reader033/viewer/2022061103/53f927128d7f7253318b4afa/html5/thumbnails/24.jpg)
Future plans Growth
Dynamic rules
QoS and MBR
UMS Elastic Logic
Open sourcing
Framework
Chandra