presentation
TRANSCRIPT
Video Quality Assessment and Video Quality Assessment and Comparative Evaluation of Comparative Evaluation of
Peer-to-Peer Video Streaming Systems Peer-to-Peer Video Streaming Systems
Video Quality Assessment and Video Quality Assessment and Comparative Evaluation of Comparative Evaluation of
Peer-to-Peer Video Streaming Systems Peer-to-Peer Video Streaming Systems
Aditya MavlankarAditya Mavlankar Pierpaolo Baccichet Pierpaolo Baccichet
Bernd GirodBernd Girod
Stanford UniversityStanford UniversityStanford CA, USAStanford CA, USA
Sachin AgarwalSachin AgarwalJatinder Pal SinghJatinder Pal Singh
Deutsche Telekom A.G., Deutsche Telekom A.G., LaboratoriesLaboratories
Berlin, GermanyBerlin, Germany
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 2
Outline
Introduction to P2P live video streaming Prior work on system performance assessment Test-bed setup Performance of tested systems
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 3
P2P Live Video Streaming
Extension of P2P file-sharing Low-cost and scalable delivery mechanism Several deployed commercial implementations today Increasing content / channels available
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 4
Related Work on Performance Assessment
Networking related metrics, e.g. bandwidth usage, packet loss, continuity index, etc.– CoolStreaming [Zhang et al., 2005]: PlanetLab– PPLive [Hei et al., 2006]: packet sniffing and crawling– SopCast [Sentinelli et al., 2007]: “watching”, PlanetLab– . . .
No video PSNR results No repeatable test conditions– Network conditions– Encoded video characteristics– Peer behavior
No fair head-to-head comparison of different systems
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 5
Test-Bed Setup
TU Munich, Germany (3)[Emulated HS Broadband]
Stanford, CA (22)[Emulated HS Broadband]
Test centerBerlin, Germany
Server 1, 2
Berlin, Germany (15)[Emulated HS Broadband]
128 X 2192 X 2576 X 21024 X 2
3072 X 3
576 X 161024 X 52048 X 1
576 X 91024 X 52048 X 1
52 Mbps
ISP
DatacenterErfurt, Germany
Internet
Berlin, Germany (8)[Real HS Broadband]
PLR, delay, jitter and bandwidth measured for representative real connections and emulated using NISTNet traffic shaper
PLR, delay, jitter and bandwidth measured for representative real connections and emulated using NISTNet traffic shaper
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 6
Encoded Video Stream
La Dolce Vita (Fellini, 1960) 24 fps, 352x240 pixels H.264/AVC video codec, 400 kbit/sec CBR bitstream,
42 dB PSNR I B B P B B P B B P . . . (I frame every second) H.264 bitstream wrapped in Microsoft ASF container, if
required by tested system Last frame error concealment
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 7
Peer Churn Model
30-minute simulation run During each 6-minute time-slot– Peer on with probability 0.9– Peer off with probability 0.1– Peer can switch off for the rest of the run with
probability 0.05 During last 5 minutes, peer off with probability 0.5
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 8
Representative Results
Tested systems– System A: Tree-based, push approach– System B: Mesh-based, data-driven or pull approach
Emulation runs– Run 1: with traffic shaping (using NISTNet)– Run 2: without traffic shaping
Same realization of peer On-Off model for all runs
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 9
0 10 20 30 40 50 6010
15
20
25
30
35
40
45
Pre-roll delay [sec]
Avg
. PS
NR
acr
oss
all
clie
nts
[dB
]
Tree, w/ traffic shapingTree, w/o traffic shapingMesh, w/ traffic shapingMesh, w/o traffic shaping
Pre-Roll Delay
about 30 sec enough for System A (tree-based)
about 60 sec enough for System B (mesh-based)
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 10
PSNR Drop (w/ traffic shaping)
0 20 400
1
2
3
4
5
6
Client ID
Avg
. dro
p in
PS
NR
[dB
]
0 20 400
1
2
3
4
5
6
Client ID
Avg
. dro
p in
PS
NR
[dB
]
System A (tree-based) System B (mesh-based)
¼X sdhsdj sdf hdsf j sdf j dsf sdf sdf kj sf j sdkj f skd¼
32
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 11
0 20 400
1
2
3
4
5
6
Client ID
Avg
. dro
p in
PS
NR
[dB
]
0 20 400
1
2
3
4
5
6
Client ID
Avg
. dro
p in
PS
NR
[dB
]
PSNR Drop (w/o traffic shaping)
System A (tree-based) System B (mesh-based)32
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 12
Statistics of Frame Freezes
Frames frozen (as percentage of total frames to be displayed)
Average no. of distinct frame-freeze events per client in 30 min.
System A (tree-based) System B (mesh-based)
Run 1 4.2% 2.0%
Run 2 3.0% 0.2%
System A (tree-based) System B (mesh-based)
Run 1 64 23
Run 2 40 2
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 13
Statistics of Frame Freezes (cont.)
0 20 40 60 80 100 1200.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Length of frame freeze [number of frames]
cdf
System A, Run 1System A, Run 2System B, Run 1System B, Run 2
Long frame freezes more likely with System B (mesh-based)
System A (tree-based) employs content-aware prioritization
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 14
No. of Peers Failing to Decode a Frame
0 10 20 300
10
20
30
40
Time [minutes]
No
. of p
ee
rs fa
ilin
g to
de
cod
e th
e fr
am
e
0 10 20 300
10
20
30
40
Time [minutes]
No
. of p
ee
rs fa
ilin
g to
de
cod
e th
e fr
am
eSystem A (tree-based), Run 1 System B (mesh-based), Run 1
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 15
Redundancy (bytes received in excess of required video stream bytes)– System A (tree-based): 6% in both runs– System B (mesh-based): 35% and 20% in Runs 1 (w/
traffic shaping) and 2 (w/o traffic shaping) respectively For both Systems, peer receives on average less than 10%
of its data directly from the server; slightly more for Run 2 of System B
System A (tree-based): Sustained downloads from lower number of parent peers
Redundancy, Server Load and Parent-Peer Analysis
Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 16
Summary
Proposed methodology allows measuring video PSNR, buffering time, frame-freeze statistics, peers failing to decode a frame, etc. beyond network usage, packet loss, etc.
Test conditions chosen by analyzing real-world conditions and experiments are repeatable
Tested three commercial-grade P2P video streaming systems Room for improvement in current systems:
– Long buffering time (10s of seconds)– Display freezes for more than 100 frames
Tested tree-based system outperforms mesh-based system:– Redundancy– Buffering time
Thank you!Thank you!http://www.stanford.edu/~maditya/publication.htmlhttp://www.stanford.edu/~maditya/publication.html
Related:Related:
[Agarwal, [Agarwal, et alet al., TRIDENTCOM 2008]., TRIDENTCOM 2008]