zt: it's not the cost, it's the quality
TRANSCRIPT
It’s Not the Cost, It’s the Quality!
Ion Stoica
Conviva Networks and UC Berkeley
1
A Brief History
! Fall, 2006: Started Conviva with Hui Zhang (CMU)
! Initial goal: use p2p technologies to reduce distribution costs and improve the scale
! Slowly, realized our customers (content premium producers & aggregators) value more quality than cost
! Today: maximize distribution quality, distribution management, and provide real-time analytics
2
Where is the Data Coming From?
! Content Providers and Aggregators
! CDNs
3
4
Trends
Trends: CDN Pricing
! CDN pricing has decreased x1.5-2 every year over the last 5 year
5
0
5
10
15
20
25
30
35
40
45
2006 2007 2008 2009 2010
cent
s/G
B
Trends: Streaming Rate for Premium Content ! Average streaming rate has increased 20-40%
every year
6
0
200
400
600
800
1000
1200
1400
1600
2006 2007 2008 2009 2010
Kbp
s
Trends: Per-hour Streaming Cost ! Per-hour streaming cost has decreased 15-35%
every year
7
0
1
2
3
4
5
6
7
8
9
10
2006 2007 2008 2009 2010
cent
s/ho
ur
HTTP Chunking
! Trend accelerated by switching from proprietary streaming technologies (e.g., Adobe’s FMS) to HTTP Chunking: Move Networks (2005)
Apple (2008)
Microsoft (2008/2009)
Adobe (2010, 2nd half)
8
How Does HTTP Chunking Work?
9
Viewer
CDN1
Viewer Viewer
CDN2
ISP A ISP B
… origin
http cache
HTTP Chunking Advantages
! Chunks: immutable, relative large objects (hundreds of KB) Great for caching
! Leverage existing HTTP infrastructure CDNs ISP deployed caches Enterprise http proxies
! Low cost and high scale
10
What Does this Mean?
! Ad supported premium content CPM (cost per thousand of ad impressions) for
premium content has reached: $20-$40 One ad covers one hour of streaming!
! Paid content $0.99 episode, distribution cost < 3%
! Subscription based premium content Distribution, usually a few percents of total cost It costs $1.6 per month to stream content to an user
watching 2 hours per day
! Production & rights costs dominate 11
12
Quality Matters
Quality Matters
! Better quality Increase viewing time more ad opportunities Increase retention rate Protect brand
! Quality Join time Buffering ratio Rendering quality Streaming rate
13
Analysis
! Load: Four channels of a premier video-on demand (VoD)
content producer Four days Number of sessions (views): 1,176,049
A large live event: ~250,000 concurrent viewers
! Metrics Content length distribution Viewer Hour Loss (VHL): number of viewer hours lost
due to quality issues
VoD Object Length Distribution
15
0
50
100
150
200
250
300
10
40
70
100
130
160
190
220
250
280
310
340
370
400
430
460
490
520
550
580
610
640
680
750
880
1240
13
30
2540
25
70
2600
26
40
4040
54
10
5550
Object Length
Short clips: [2min, 3min] Medium clips:
[9min, 11min] Full Episodes: [42min, 45min]
Num
ber o
f Obj
ects
Quality Metrics
! Buffering Quality (BQ): PlayingTime/(PlayingTime + BufferingTime)
Rendering Quality (RQ): RenderingRate/EncoderRate
Good session BQ > 95% RQ > 60%
Analysis Underestimates Quality Impact
! For most analysis use BQ only RQ only a small part of quality issues due to low
bit rate (500-700Kbps)
! Ignore connection failures
Short Clip (2-3min) Analysis
18
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50
Aver
age
Play
ing
Tim
e (M
inut
es)
Quality
Short Clip (2-3min) Analysis
19
0
50000
100000
150000
200000
250000
300000
98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50
Num
ber
of S
essi
ons
Quality
Viewer Hour Gain
! D : Average duration of sessions with high quality (0.98 <= quality < 1)
! Dq : Average duration of sessions with quality = q
! Nq : Number of sessions with quality = q
! Viewer hour gain for sessions with quality q Nq x (D – Dq)
! Total viewer hour gain ∑q Nq x (D – Dq)
20
Viewer hour loss for 1-2 minute clips:
Medium Clip (9-11min) Analysis
21
0
1
2
3
4
5
6
7
8
98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50
Aver
age
Play
ing
Tim
e (M
inut
es)
Quality
Viewer hour loss for 9-11min clips:
Full Episodes (42-45min) Analysis
22
0
5
10
15
20
25
98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50
Aver
age
Play
ing
Tim
e (M
inut
es)
Quality
• Viewer hour loss for episodes: • Viewer hour loss for all content:
Large Scale Live Event
23
0
5
10
15
20
25
30
35
98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50
Aver
age
Play
ing
Tim
e (M
inut
es)
Quality Viewer hour loss:
Large Scale Live Event: Engagement Funnel
24
1 0.99
0.8624
0.7364 0.6602
0.5509 0.4357
0.3639
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
All 0-0 0-1 1-3 3-5 5-10 10-20 20-30
Frac
tion
of
Sesi
ons
Session Duration (Mins)
0<=Q<=0.5
0.5<=Q<=0.75
0.75<=Q<=0.9
0.9 <= Q <= 0.95
0.95<=Q<=0.98
0.98<=Q<=1.0
Quality = 1.0
Continuing
Half people leave due to quality issues
Another Case Study: Live Event
25
Total sessions 151,980
Unique viewers 73,942
Sessions per viewer 1.9
Total viewer hours 58,436
0
5000
10000
15000
20000
7:00
PM
7:
15 P
M
7:30
PM
7:
45 P
M
8:00
PM
8:
15 P
M
8:30
PM
8:
45 P
M
9:00
PM
9:
15 P
M
9:30
PM
9:
45 P
M
10:0
0 PM
10
:15
PM
10:3
0 PM
10
:45
PM
11:0
0 PM
11
:15
PM
11:3
0 PM
11
:45
PM
12:0
0 AM
12
:15
AM
12:3
0 AM
12
:45
AM
1:00
AM
Peak Concurrent Views Quality Engagement
Total views 151,980 25 minutes
Failed views 13,815 (9%) 0 minutes
Quality impacted views
21,584 (14%) 16 minutes
Good views 116,581 (77%) 27 minutes
Unique viewers 75,328 48 minutes
Failed viewers 1,386 (2%) 0 minutes
Quality impacted viewers
14,309 (19%) 30 minutes
Good viewers 59,633 (79%) 51 minutes
Total viewer hours 58,436 hours
Lost viewer hours 5,134 hours (9%)
Viewer with poor quality watch 41% less minutes!
Does High Bit Rate Video Help?
! Comparing Engagement of low and high bitrates Viewers watch longer on average on 1500Kbps
0
5
10
15
20
25
30
35
min
utes
Average Play Time (minutes)
500 kbps
1500 kbps
10-15% increase
Summary
! Quality impact: BQ can impact viewer engagement by up to 40% Higher bit-rates can increase viewer engagement by
up to 15%
! Engagement loss due to quality issues: between 4 and 30% Even a 4% improvement, may offset distribution
costs Ignore other quality issues, like connectivity and
media failures
27
28
Root Cause Analysis
Viewers vs. Buffering Quality
29
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
0
50000
100000
150000
200000
250000
0:00
:00
0:15
:00
0:30
:00
0:45
:00
1:00
:00
1:15
:00
1:30
:00
1:45
:00
2:00
:00
2:15
:00
2:30
:00
2:45
:00
3:00
:00
3:15
:00
3:30
:00
3:45
:00
4:00
:00
4:15
:00
4:30
:00
4:45
:00
5:00
:00
5:15
:00
5:30
:00
5:45
:00
6:00
:00
6:15
:00
Viewers Average Buffering Quality %
Light Period Heavy Period
What causes quality issues?
Root Cause Analysis
! Root cause a quality issue to: Viewer machine (CPU) Last mile + ISP (Autonomous System Number) CDN
! Note: Cannot differentiate between edge and core ISPs Use only passive measurements, no IP traceroute
Viewer
Last Mile +
ISP
CDN Peering ISP
Metrics and Definitions
! Quality metrics Buffering quality (BQ)
playing time/(playing time + buffering time) Rendering quality (RQ)
rendering frame rate/encoded frame rate
! Session classification: Good: (BQ >= 95%) AND (RQ >= 60%) Low BQ: (BQ < 95%) Low RQ: (BQ >= 95%) AND (RQ < 60%)
Methodology: Root Causing Viewer Machine
! CPU likely to be the issue when: Rendering quality low Buffering quality high
! Conclude CPU is the issue when session’s RQ < 60% BQ > 95%
Good Low BQ Low RQ
Quality Issues: Light Period
74%
21.5%
4% (CPU issue) Network/CDN issues
Good Low BQ Low RQ
Quality Issues: High Period
62%
6.2% (CPU issue)
31.5%
Network/CDN issues
Explaining Buffering Issues
! Assume buffering quality issues are either due to: CDN, or ISP
! Recall: a session has buffering quality issues if BQ < 95%
Methodology: Root Causing CDN (1/2)
! Viewers connected to same ASN but using two CDNs
! Intuition: if quality experienced by CDN 1 viewers is significantly lower than of CDN 2 viewers for same ASN, CDN 1 has quality issues
Last Mile +
ISP
CDN 1 Peering ISP
CDN 2 Peering
ISP
Methodology: Root Causing CDN (2/2)
! Select all ASNs who have more than 50 sessions for each CDN
• If difference between quality of viewers in CDN1 and CDN2 for same ASN is > 10% Lower quality CDN is root cause at current time
Methodology: Root Causing ASN/ISP
! Two CDNs: Conclude ASN A has quality issues if ASN A’s viewers
connected to either CDN1 or CDN2 experience “bad quality”
Average quality of viewers connected to other ASNs higher
! One CDN ASN A’s viewers connected to CDN have much lower
quality than the average quality of viewers connected to CDN
Buffering Quality: Light Period
Unqualified CDN ISP Uknown
46% 33%
16%
5%
Buffering Quality: Heavy Period
40
Unqualified CDN ISP Uknown
36.7%
39.7%
15.4%
8.2%
Some Findings
! Most of ASNs who had quality issues were enterprise ASNs Expected given that the large scale event was during
the workday One ASN had 44% buffering quality!
! No CDN was uniformly bad (see next)
CDN Comparison
! Quantify quality difference between CDNs
! Methodology: 1. Select all ASNs which have more than 50 sessions
on both CDNs 2. Compute average quality for CDN1 and CDN2
viewers per ASN 3. Order ASNs by difference in quality between CDN1
and CDN2
Internet delivery is more variable than realized…
! Content Delivery Networks all have problems sometime
! Even in the same viewer session the best quality changed many times during the event
CDN 1 was best
CDNs were even
CDN 2 was best
Summary 24
-38%
of
Tota
l Ses
sion
s ha
ve Q
ualit
y Is
sues
Quality Issues Classification
Solution
CDN (7-12% of total sessions)
Resource switching
End-Host CPU (4-6% of total sessions)
Bit-rate switching
ISP (2-3% of total sessions)
Localize traffic, bit-rate switching
Unqualified (9-11% of total sessions)
Mitigated by above
Unknown (1-4% of total sessions)
N/A
Conclusions
! At least for premium content Reducing cost is important, but… … improving quality is even more important
! P2P can play an important role Localize traffic Highly robust to source failures
! Great opportunity Adobe has announced full p2p support for Flash
Player 10.1 No need for client download!
45