measuring quality of experience for successful iptv deployments dr. stefan winkler
TRANSCRIPT
Measuring Quality of Experience for Successful IPTV Deployments
Dr. Stefan Winkler
Outline
• Challenges– Digital Video Quality Issues– Traditional Measurements (QoS) vs.
Quality of Experience (QoE)
• Possible Solutions– QoE Measurement Approaches– End-to-end QoE Management
• Conclusions
Digital Video Challenges
Demanding traffic profiles High bandwidth streams
High traffic volumes
Live, VOD
Network effects Video impacted heavily with
minor network impairments
Multi-vendor network complicates diagnosis / troubleshooting
High end-user expectations Defined with decades of history
Grow rapidly with HD
Low tolerance for poor quality
New architectures Sensitive video processing
devices create possibility for various impairment sources
Ad-insertion, middleware
Service quality degradations
Difficult diagnosis, troubleshooting
Rising management and
OPEX costs
Higher customer churn
What Drives End-Users
Source: MRG 2007 IPTV Video Quality Survey, available at http://qoe.symmetricom.com
Service Providers View
Source: MRG 2007 IPTV Video Quality Survey, available at http://qoe.symmetricom.com
Service Providers’ View
7Source: MRG 2007 IPTV Video Quality Survey, available at http://qoe.symmetricom.com
Sources of Video Issues
Consider all elements for true end-to-end solution
Compression Artifacts
Original MPEG-2 H.264
PSNR vs. QoE
Same amount of distortion (PSNR) – different perceived quality
Understand & model human vision system
QoS vs. QoE
• Quality of Service– Network-centric– Delay, packet loss, jitter – Transmission quality– Content agnostic
• Quality of Experience– Content impairments – Blockiness, Jerkiness, …– End-user quality– Application driven
QoS QoE
Same network impairmentsPacket Loss: 1%
Delay: 10msJitter: 50us
Bandwidth: 500kbps
Different perceived quality!
QoS vs. QoE
MDI vs. QoE
• Media Delivery Index (MDI)• MDI consists of two metrics:
– Delay Factor (DF)– Media Loss Rate (MLR)
• MDI limitations:– MDI assumes constant bit rate (CBR) traffic– MDI does not consider video payload or content– MDI values are not intuitive– MDI doesn’t correlate with video quality
MDI vs. QoE
0
1
2
3
4
5
V-F
AC
TO
R
0
500
1000
1500
2000
2500
3000
Med
ia L
oss
Jitter1 to 50ms
Packet Drop1 in 500 periodic
Packet Drop1 in 500 poisson
Packet Drop1 in 500 uniform
Packet Drop1 in 500 and Jitter
Duplicate Packets1 in 10
Duplicate Packets1 in 500
Reordering Packets1 in 500
Med
ia L
oss
MO
S
QoS/QoE Cycle
Desired QoE
Perceived QoE
Targeted QoS
DeliveredQoS
End-userService
provider
Alignment gap
Perception gap
Value gap Execution gap
Adapted from ITU-T Rec. G.1000 and COM12–C185–E
Outline
• Challenges– Digital Video Quality Issues– Traditional Measurements (QoS) vs.
Quality of Experience (QoE)
• Possible Solutions– QoE Measurement Approaches– End-to-end QoE Management
• Conclusions
Full-Reference Approach
• Comparison of individual video frames• Offline analysis (capture is required) – lab applications• High detail and accuracy• Alignment procedure
Sender Receiver
Full reference information
Full Ref. QualityMeasurement
No-Reference Approach
• Non-intrusive, in-service measurement• Real-time monitoring applications• No alignment required
Sender Receiver
No-Ref. QualityMeasurement
• Monitoring applications• Correlation of content and network impairments• Encrypted environments
Sender Receiver
Reduced Ref.Measurement
Feature Extraction
Feature Extraction
Reduced-Reference Approach
Content & Network Metrics
20
"Vision is the most highly developed of the human senses, so people are even more sensitive to flaws in video images than, say, the sound of a telephone conversation.”
Ken Wirt, Cisco Vice President Consumer Marketing, Jan 2008
(Correlation Engine)
• Contrast perception– Visibility of different patterns– Frequency dependencies
• Masking effects– Interaction of content and impairments– Texture, edges, luminance– Spatial and temporal masking
• Color perception
Spatial frequency [cpd]Temporal frequency [Hz]
Sen
sitiv
ity
Vision Modeling
Masker contrast
Visibility threshold
Targ
et
contr
ast
Maskingcurve
Thresholdwithoutmasker
End-to-end QoE
Deep Content Analysis(pixel by pixel)
Source content and encoder / transcoder validation
Human Vision System Model
VideoQualityReports
Content Impairments:• Blockiness, blur• Jerkiness• Freeze/black frame• Noise, Color
Network Impairments:• Loss• Delay• Jitter• Bandwidth
Content Stream Analysis:• PES inspection• PCR jitter etc.
Deep Content Analysis (bitstream)
Detect content impairmentsDeep inspection to associate content to timestamps (eg: TS1 = I-Frame) Network (header or stream) Analysis
Detect QoS issuesContent analysis where possible (unencrypted)Inspection of QoS to associate timestamps to impairments (eg: TS1 = Packet Loss)
Q-Advisor
Correlation Engine
TS1 = I-Frame
TS1 = Packet Loss
Packet Loss -> I-Frame
IPTV QoE Management
25
Issue Possible Causes
Blockiness Encoder Transcoder Network Loss
Blur Camera (focus) Encoder Transcoder STB (bad filtering)
Freeze Frame, Jerkiness Encoder (dropped frames)
Network loss Bad synchronization
Black Screen, Blue Screen
No Video Signal (source)
Ads not inserted Major network loss
Color Encoder Camera Transcoder
Video Noise (analog noise)
Camera STB
Noise (digital) Encoder Transcoder
Audio Microphone Encoder (bad mono stereo encoding
Encoder (lip sync) STB
1.0
1. Understand the Service Is there an issue?Does it matter?
1. Understand the Service Is there an issue?Does it matter?
2. Understand the ProblemWhat does the customer see?What is the exact cause?
2. Understand the ProblemWhat does the customer see?What is the exact cause?
3. Understand the SolutionWhat is the impairment source?
3. Understand the SolutionWhat is the impairment source?
1
2
3
4
5
Very Annoying
Annoying
Slightly Annoying
Perceptible
Imperceptible
Conclusions
• QoE is application-driven– Measure both network and content impairments
• QoE is user-oriented– Measure how end-user perceives service issues
• End-to-end quality measurement– Cover different impairment sources– Identify problem causes
Stefan [email protected]
Company:qoe.symmetricom.com
Further Reading:stefan.winkler.net/book.html
Contact Info