2016 streaming media west - how to improve video quality of service with an effective ab testing...
TRANSCRIPT
How to improve quality of service with player-side iterations and an effective AB testing workflow
Streaming Media West – Track DTuesday, November 1, 2016 2:45 to 3:30 pm
● Creators of peer-accelerated video delivery for large-scale OTT
● Pioneers in hybrid video delivery systems to accompany exponential growth
● Experts in HTML5 video through integrations into open-source & proprietary players
Infinite scale, limitless delivery.
Streamroot: Who are we?
I. How to build an AB Testing workflowWhen and what to test, analytics pipeline, AB testing setup
II. Dailymotion use case: ABR iterations for quick, dramatic QoS improvements
ABR iterations to hls.js to cut rebuffering
III. Streamroot use case: using effective AB testing to improve our productStreamroot workflow and example
Presentation Outline
Infinite scale, limitless delivery.
Infinite scale, limitless delivery.
AB Testing is Data-Driven
Development.…here applied to Media engines & QoS.
I. Building an effective AB testing workflow
Infinite scale, limitless delivery.
I. Building an effective AB testing workflow
AB testing makes sense when:
You can’t test locally: distributed and unmanaged networks (ABR, P2P).
Test subjective hypothesis, magic numbers.
You deploy a completely new feature that breaks your old logic.
Infinite scale, limitless delivery.
I. Building an effective AB testing workflow
Before starting:
Define your metrics
Determine an appropriate sample size and testing timeframe
Define your variables
Define your populations
Infinite scale, limitless delivery.
For testing, you need:
The right data
A DB to store it
A BI/analytics tool
I. Building an effective AB testing workflow
Infinite scale, limitless delivery.
I. How to build: Data pipelineGeneral workflow
Infinite scale, limitless delivery.
Streamroot workflow
I. How to build: Data PipelineGeneral workflow
Many options based on your requirements: open-source, proprietary, Cloud...
Infinite scale, limitless delivery.
I. How to build: Data PipelineWhat to chose from
Message broker / stream processor
Storage + possible query/visualization
BI/viz tools Advanced analytics
Kafka (Kinesis) Druid Caravel R
RabbitMQ InfluxDB Grafana Matlab
ActiveMQ Cassandra (DynamoDB) Tableau
PostGresSQL QlikView
ElasticSearch Kibana
Infinite scale, limitless delivery.
I. How to build: AB Testing workflowGeneral setup
Client AB testing methods:
1) Dynamic Config Injection2) Different builds with a reverse proxy
Infinite scale, limitless delivery.
I. How to build: AB Testing workflowGeneral setup
Infinite scale, limitless delivery.
I. How to build: AB Testing workflowConfig injection
Infinite scale, limitless delivery.
I. How to build: AB Testing workflowConfig injection
Infinite scale, limitless delivery.
I. How to build: AB Testing workflowDynamic config injection
PROs
Simple & easy splitsSeparate - fast to deploy & rollbackScalable
CONs
Limited scope Complications in codeNeed to handle scaling & high availability
WARNING
Async implementation!
Infinite scale, limitless delivery.
I. How to build: AB Testing workflowDifferent builds with reverse proxy
Infinite scale, limitless delivery.
PROs
Serve very different (major) versions of playerPerfect for rolling out new features smoothly (10% then 30% then...)
CONs
Good CI/CD necessary to roll out new versions quicklyHeavy stack to maintain and scale
WARNING
Make sure the file request is not slower!Single point of failure!
I. How to build: AB Testing workflowDifferent builds with reverse proxy
1. AB test when it makes sense.
2. Choose the right metrics.
3. Build a pipeline with tools that are adapted to your use case.
4. Use the right method depending on what you’re testing.
Infinite scale, limitless delivery.
I. How to build: AB Testing workflowTips & sum up
Infinite scale, limitless delivery.
II. DailymotionSome numbers: world’s 2nd largest video hosting platform
100 million video views each day 63%
Desktop
31%Mobile
8%Tablet
1%SmartTV
69%HTML5
31%Flash
Infinite scale, limitless delivery.
II. DailymotionABR iterations for quick, dramatic QoS improvements
I should have rehearsed
this more…When’s lunch?
Adaptive bitrate algorithms can have a huge impact on QoS.
They are easy to tweak, but improvements depend on your specific use case.
One size does not fit all.
Infinite scale, limitless delivery.
II. DailymotionABR iterations for quick, dramatic QoS improvements
Infinite scale, limitless delivery.
II. DailymotionImproving hls.js’s ABR logic
Infinite scale, limitless delivery.
II. DailymotionImproving hls.js’s ABR logic
Introduce history parameter to bandwidth estimation.
Infinite scale, limitless delivery.
II. DailymotionImproving hls.js’s ABR logic
Average out the values.
https://github.com/dailymotion/hls.js/blob/master/src/controller/abr-controller.js
EWMA:
Infinite scale, limitless delivery.
II. DailymotionImproving hls.js’s ABR logic
Shaka Player:
ABR now based on two bandwidth moving averages:
• Fast → adapt down quickly
• Slow → adapt up more slowly
Bandwidth estimation = min(fast, slow)
Infinite scale, limitless delivery.
II. DailymotionImproving hls.js’s ABR logic
Control over media engine
Control over events
QoS events
AB testing & customizationABR algos, config, etc.
Infinite scale, limitless delivery.
CDNDirector
Analytics aggregation and visualization
II. DailymotionAB testing workflow: total control over their stack
67.4% native
70.3%hls.js
78.6%flashls
Infinite scale, limitless delivery.
II. DailymotionComparing the results - live
Rebuffering, percentile rank
native flashls hls.js,s=9,f=3 hls.js,s=0,f=0 hls.js,s=6,f=1
Infinite scale, limitless delivery.
II. DailymotionComparing the results - live
Rebuffering, percentile rank
hls.js,s=0, f=0
hls.js, s=9, f=3
Infinite scale, limitless delivery.
II. DailymotionNumber of level switches - live
With the right workflow, big improvement in less than 2 weeks of testing.
Numbers are use-case specific (segment length, network, etc.).
Lots of other options to test → a continuous effort to improve QoS!
Infinite scale, limitless delivery.
II. DailymotionConclusions
Infinite scale, limitless delivery.
III. StreamrootPeer-accelerated delivery
...We have a very distributed network!
AB testing on:
- matching devices to get the best connections- Transfer algorithms and protocol improvements- QoS impact and improvements
Infinite scale, limitless delivery.
III. StreamrootWhy are we using AB testing
Infinite scale, limitless delivery.
III. StreamrootThe pipeline
GET build-a.js
Pipeline
AB testing File Distribution
AB Test on AskSegment
Infinite scale, limitless delivery.
III. StreamrootThe test case
Infinite scale, limitless delivery.
III. StreamrootResults
AskSegment Timeout Samples P2P %
Longer 15,772,542 67.41%
Shorter 16,152,624 67.43%
No effect?
Infinite scale, limitless delivery.
III. StreamrootResults
AskSegment Timeout Type Samples P2P %
Longer Live 9,004,140 66.43%
Shorter Live 9,340,504 68.02%
Longer VOD 6,768,403 68.72%
Shorter VOD 6,812,120 66.61%
Live: +1.59% for shorter timeoutVOD: +2.11% for longer timeout
- Understand and know your data
- Ab testing is sometimes the only way to prove your hypothesis
- You will be wrong sometimes: Make sure you can rollback quickly!!
- The most difficult is to start! Once it is set up: be relentless!
Continuous AB testing FTW
Infinite scale, limitless delivery.
III. Streamroot Takeaways
Infinite scale, limitless delivery.
Questions?
Nikolay Rodionov, Co-Founder and CPO, [email protected]
Erica Beavers, Head of Partnerships, [email protected]
Infinite scale, limitless delivery.
Contact us!