2016 streaming media west - how to improve video quality of service with an effective ab testing...

How to improve quality of service with player-side iterations and an effective AB testing workflow

Streaming Media West – Track DTuesday, November 1, 2016 2:45 to 3:30 pm

● Creators of peer-accelerated video delivery for large-scale OTT

● Pioneers in hybrid video delivery systems to accompany exponential growth

● Experts in HTML5 video through integrations into open-source & proprietary players

Infinite scale, limitless delivery.

Streamroot: Who are we?

I. How to build an AB Testing workflowWhen and what to test, analytics pipeline, AB testing setup

II. Dailymotion use case: ABR iterations for quick, dramatic QoS improvements

ABR iterations to hls.js to cut rebuffering

III. Streamroot use case: using effective AB testing to improve our productStreamroot workflow and example

Presentation Outline

Infinite scale, limitless delivery.

Infinite scale, limitless delivery.

AB Testing is Data-Driven

Development.…here applied to Media engines & QoS.

I. Building an effective AB testing workflow

Infinite scale, limitless delivery.

I. Building an effective AB testing workflow

AB testing makes sense when:

You can’t test locally: distributed and unmanaged networks (ABR, P2P).

Test subjective hypothesis, magic numbers.

You deploy a completely new feature that breaks your old logic.

Infinite scale, limitless delivery.

I. Building an effective AB testing workflow

Before starting:

Define your metrics

Determine an appropriate sample size and testing timeframe

Define your variables

Define your populations

Infinite scale, limitless delivery.

For testing, you need:

The right data

A DB to store it

A BI/analytics tool

I. Building an effective AB testing workflow

Infinite scale, limitless delivery.

I. How to build: Data pipelineGeneral workflow

Infinite scale, limitless delivery.

Streamroot workflow

I. How to build: Data PipelineGeneral workflow

Many options based on your requirements: open-source, proprietary, Cloud...

Infinite scale, limitless delivery.

I. How to build: Data PipelineWhat to chose from

Message broker / stream processor

Storage + possible query/visualization

BI/viz tools Advanced analytics

Kafka (Kinesis) Druid Caravel R

RabbitMQ InfluxDB Grafana Matlab

ActiveMQ Cassandra (DynamoDB) Tableau

PostGresSQL QlikView

ElasticSearch Kibana

Infinite scale, limitless delivery.

I. How to build: AB Testing workflowGeneral setup

Client AB testing methods:

1) Dynamic Config Injection2) Different builds with a reverse proxy

Infinite scale, limitless delivery.

I. How to build: AB Testing workflowGeneral setup

Infinite scale, limitless delivery.

I. How to build: AB Testing workflowConfig injection

Infinite scale, limitless delivery.

I. How to build: AB Testing workflowConfig injection

Infinite scale, limitless delivery.

I. How to build: AB Testing workflowDynamic config injection

PROs

Simple & easy splitsSeparate - fast to deploy & rollbackScalable

CONs

Limited scope Complications in codeNeed to handle scaling & high availability

WARNING

Async implementation!

Infinite scale, limitless delivery.

I. How to build: AB Testing workflowDifferent builds with reverse proxy

Infinite scale, limitless delivery.

PROs

Serve very different (major) versions of playerPerfect for rolling out new features smoothly (10% then 30% then...)

CONs

Good CI/CD necessary to roll out new versions quicklyHeavy stack to maintain and scale

WARNING

Make sure the file request is not slower!Single point of failure!

I. How to build: AB Testing workflowDifferent builds with reverse proxy

1. AB test when it makes sense.

2. Choose the right metrics.

3. Build a pipeline with tools that are adapted to your use case.

4. Use the right method depending on what you’re testing.

Infinite scale, limitless delivery.

I. How to build: AB Testing workflowTips & sum up

Infinite scale, limitless delivery.

II. DailymotionSome numbers: world’s 2nd largest video hosting platform

100 million video views each day 63%

Desktop

31%Mobile

8%Tablet

1%SmartTV

69%HTML5

31%Flash

Infinite scale, limitless delivery.

II. DailymotionABR iterations for quick, dramatic QoS improvements

I should have rehearsed

this more…When’s lunch?

Adaptive bitrate algorithms can have a huge impact on QoS.

They are easy to tweak, but improvements depend on your specific use case.

One size does not fit all.

Infinite scale, limitless delivery.

II. DailymotionABR iterations for quick, dramatic QoS improvements

Infinite scale, limitless delivery.

II. DailymotionImproving hls.js’s ABR logic

Infinite scale, limitless delivery.

II. DailymotionImproving hls.js’s ABR logic

Introduce history parameter to bandwidth estimation.

Infinite scale, limitless delivery.

II. DailymotionImproving hls.js’s ABR logic

Average out the values.

https://github.com/dailymotion/hls.js/blob/master/src/controller/abr-controller.js

EWMA:

Infinite scale, limitless delivery.

II. DailymotionImproving hls.js’s ABR logic

Shaka Player:

ABR now based on two bandwidth moving averages:

• Fast → adapt down quickly

• Slow → adapt up more slowly

Bandwidth estimation = min(fast, slow)

Infinite scale, limitless delivery.

II. DailymotionImproving hls.js’s ABR logic

Control over media engine

Control over events

QoS events

AB testing & customizationABR algos, config, etc.

Infinite scale, limitless delivery.

CDNDirector

Analytics aggregation and visualization

II. DailymotionAB testing workflow: total control over their stack

67.4% native

70.3%hls.js

78.6%flashls

Infinite scale, limitless delivery.

II. DailymotionComparing the results - live

Rebuffering, percentile rank

native flashls hls.js,s=9,f=3 hls.js,s=0,f=0 hls.js,s=6,f=1

Infinite scale, limitless delivery.

II. DailymotionComparing the results - live

Rebuffering, percentile rank

hls.js,s=0, f=0

hls.js, s=9, f=3

Infinite scale, limitless delivery.

II. DailymotionNumber of level switches - live

With the right workflow, big improvement in less than 2 weeks of testing.

Numbers are use-case specific (segment length, network, etc.).

Lots of other options to test → a continuous effort to improve QoS!

Infinite scale, limitless delivery.

II. DailymotionConclusions

Infinite scale, limitless delivery.

III. StreamrootPeer-accelerated delivery

...We have a very distributed network!

AB testing on:

- matching devices to get the best connections- Transfer algorithms and protocol improvements- QoS impact and improvements

Infinite scale, limitless delivery.

III. StreamrootWhy are we using AB testing

Infinite scale, limitless delivery.

III. StreamrootThe pipeline

GET build-a.js

Pipeline

AB testing File Distribution

AB Test on AskSegment

Infinite scale, limitless delivery.

III. StreamrootThe test case

Infinite scale, limitless delivery.

III. StreamrootResults

AskSegment Timeout Samples P2P %

Longer 15,772,542 67.41%

Shorter 16,152,624 67.43%

No effect?

Infinite scale, limitless delivery.

III. StreamrootResults

AskSegment Timeout Type Samples P2P %

Longer Live 9,004,140 66.43%

Shorter Live 9,340,504 68.02%

Longer VOD 6,768,403 68.72%

Shorter VOD 6,812,120 66.61%

Live: +1.59% for shorter timeoutVOD: +2.11% for longer timeout

- Understand and know your data

- Ab testing is sometimes the only way to prove your hypothesis

- You will be wrong sometimes: Make sure you can rollback quickly!!

- The most difficult is to start! Once it is set up: be relentless!

Continuous AB testing FTW

Infinite scale, limitless delivery.

III. Streamroot Takeaways

Infinite scale, limitless delivery.

Questions?

Nikolay Rodionov, Co-Founder and CPO, nikolay@streamroot.io

Erica Beavers, Head of Partnerships, erica@streamroot.io

Infinite scale, limitless delivery.

Contact us!