July 2006
Royce Fernald - Intel Corporation
Slide 1
doc.: IEEE 802.11-06/0965r1
Submission
Video Performance
Notice: This document has been prepared to assist IEEE 802.11. It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein.
Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE 802.11.
Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures <http:// ieee802.org/guides/bylaws/sb-bylaws.pdf>, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair <[email protected]> as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE 802.11 Working Group. If you have questions, contact the IEEE Patent Committee Administrator at <[email protected]>.
Name Company Address Phone Email Royce Fernald Intel 5200 Elam Young Pkwy
MS:HF3-96 Hillsboro, OR 97124
503-696-4318 [email protected]
Uriel Lemberger
Intel Matam Industrial Park PO Box 1659 Haifa 31015 Israel
+972-4-865-5701
Alexander Tolpin Intel Matam Industrial Park PO Box 1659 Haifa 31015 Israel
+972-4-865-5430
Philip Corriveau Intel 5200 Elam Young Pkwy MS:HF3-96 Hillsboro, OR 97124
503-696-1837 [email protected]
Fahd Pirzada Dell One Dell Way Round Rock, TX 78682
512-338-4400 [email protected]
Pratik Mehta Dell One Dell Way Round Rock, TX 78682
512-338-4400 [email protected]
Fanny Mlinarsky Azimuth 31 Nagog Park, Acton, MA 01720
978-268-9205 [email protected]
Dalton Victor Broadcom 190 Mathilda Place Sunnyvale, CA 94025
408-922-5824 [email protected]
Mark Kobayhashi Broadcom 190 Mathilda Place Sunnyvale, CA 94025
408-543-3318 [email protected]
Authors:
July 2006
Royce Fernald - Intel Corporation
Slide 2
doc.: IEEE 802.11-06/0965r1
Submission
Abstract
This presentation defines the primary metrics for video performance as well as key secondary metrics
This presentation corresponds to the draft text submission 802.11-06/0964r1
July 2006
Royce Fernald - Intel Corporation
Slide 3
doc.: IEEE 802.11-06/0965r1
Submission
Agenda
Background– Proposal Framework– Test Output– Video Delivery and Video Quality– Measuring Video Delivery and Video Quality– The Media Delivery Index– Video Delivery, Video Quality and Wireless Performance– References
• Test Description (doc.: 802.11-06/0964r1)– Introduction and Purpose– Resource Requirements– Test Setup– Video Content Preparation and Analysis– Calibration– Baseline Configuration– Modifiers– Measurement Procedures– Reported Results– Sample Data
• Motion
July 2006
Royce Fernald - Intel Corporation
Slide 4
doc.: IEEE 802.11-06/0965r1
Submission
Proposal Framework
• Usage case: Video streaming applications• Test Environments:
– Conducted Environment– OTA Environments
• Primary Metrics– Video Delivery
• Video Delivery Error Rate (VDER)
– Video Quality• Perceived Video Quality Mean Opinion Score (MOS)
• Secondary Metrics– Media Delivery Index (MDI)
• Media Loss Rate (MLR)• Delay Factor (DF)
July 2006
Royce Fernald - Intel Corporation
Slide 5
doc.: IEEE 802.11-06/0965r1
Submission
Test Output
• Video Delivery and Video Quality metrics are the primary output of the test
• MDI quantifies the impact of the network on the test outcome– Helps isolate root causes of poor performance
July 2006
Royce Fernald - Intel Corporation
Slide 6
doc.: IEEE 802.11-06/0965r1
Submission
Video Delivery and Video Quality
• There are two complementary ways to characterize video performance – video delivery and video quality
• Video delivery metrics quantify the impact of dropped, repeated or out-of-sequence video frames– Video delivery errors are caused by large-scale packet losses or
delay on the link
• Video quality metrics quantify the impact of video artifacts on end user perception– Video artifacts are caused by packet loss – Different codecs / content types translate levels of packet loss to
different levels of video quality
July 2006
Royce Fernald - Intel Corporation
Slide 7
doc.: IEEE 802.11-06/0965r1
Submission
Measuring Video Delivery and Video Quality
• Video Delivery Measurements– Video delivery measurements are made with video playback analysis
applications (VPAA)– VPAAs provide quantitative metrics that reflect the performance of the
underlying transport medium– A correlation can be established between video delivery measurements
and MOS of end user perception
• Video Quality Measurements– Video quality analysis applications (VQAA) output mean opinion scores
(MOS) that represent the expected end user perception of video quality– VQAAs are based on psychovisual models (such as VQM) which are
designed to characterize the quality of a perceived video clip relative to a source file
July 2006
Royce Fernald - Intel Corporation
Slide 8
doc.: IEEE 802.11-06/0965r1
Submission
The Media Delivery Index
• MDI defines media-weighted secondary metrics that quantify the impact of wireless link performance on observed video quality
• The media loss rate (MLR) reflects the portion of the video stream that was undelivered due to packet loss– The MLR is the packet loss rate expressed as a percentage of the video
stream– High MLR corresponds to video delivery errors (client buffer starvation)
or noticeable display artifacts from undelivered packets
• The delay factor (DF) quantifies the amount of buffering required to maintain an uninterrupted media stream– The DF is the required client buffer depth divided by the bitrate of the
video stream– High DF corresponds to non-fluid playback and loss of interactivity for
real-time applications such as video conferencing
July 2006
Royce Fernald - Intel Corporation
Slide 9
doc.: IEEE 802.11-06/0965r1
Submission
Video Delivery, Video Quality and Wireless Performance – Best Effort Traffic (i.e. UDP/RTP)
July 2006
Royce Fernald - Intel Corporation
Slide 10
doc.: IEEE 802.11-06/0965r1
Submission
Video Delivery, Video Quality and Wireless Performance – Guaranteed Traffic (i.e. TCP)
July 2006
Royce Fernald - Intel Corporation
Slide 11
doc.: IEEE 802.11-06/0965r1
Submission
References
IEEE 802.11-06/0964r1 “Video Performance”
IEEE 802.11-06/0651r1 “Video Delivery versus Attenuation in a Conducted Environment”
IEEE 802.11-06/0321r0 “Video Gross Error Detector Video over Wireless Methodology”
IEEE 802.11-06/0144r1 “Video over Wireless Methodology”
IEEE 802.11-05/1194r0, “Video Testing Methodology”
IEEE 802.11-05/0887r0, “Video Testing Strategy”
July 2006
Royce Fernald - Intel Corporation
Slide 12
doc.: IEEE 802.11-06/0965r1
Submission
Agenda
Background– Proposal Framework– Test Output– Video Delivery and Video Quality– Measuring Video Delivery and Video Quality– The Media Delivery Index– Video Delivery, Video Quality and Wireless Performance– References
Test Description (doc.: 802.11-06/0964r1)– Introduction and Purpose– Resource Requirements– Test Setup– Video Content Preparation and Analysis– Calibration– Baseline Configuration– Modifiers– Measurement Procedures– Reported Results– Sample Data
• Motion
July 2006
Royce Fernald - Intel Corporation
Slide 13
doc.: IEEE 802.11-06/0965r1
Submission
Introduction and Purpose
• This test measures the video streaming performance of the link between a DUT and a WLCP– Video traffic is streamed to the DUT over the RF interface
– The DUT display data is captured and analyzed for video delivery errors and video quality defects
– Traffic measured by the 802.11 traffic analyzer is used to calculate the MDI (media loss rate and delay factor)
July 2006
Royce Fernald - Intel Corporation
Slide 14
doc.: IEEE 802.11-06/0965r1
Submission
Hardware Resource Requirements
• The basic test configuration is defined in section 5
• In addition, the following items are required:– A traffic generator capable of generating video traffic
• The video traffic generator can be a general purpose PC or dedicated device that streams video content to the DUT
– Video capture device, a PC or dedicated device capable of connecting to the DUT display port and saving video data to non-volatile storage
– The DUT may also act as the video capture device if it has the ability to save or process media streams in real-time (not progressive download)
July 2006
Royce Fernald - Intel Corporation
Slide 15
doc.: IEEE 802.11-06/0965r1
Submission
Software Resource Requirements
• A video playback analysis application capable of instrumenting video content for the detection of dropped, repeated and out-of-sequence video frames
• A video quality analysis application capable of analyzing video frames for image defects and rendering a quality score based on a perceptual model of the video format under test
• A video encoder application for encoding test media in the required format
• A video capture application for capturing display data from the DUT
July 2006
Royce Fernald - Intel Corporation
Slide 16
doc.: IEEE 802.11-06/0965r1
Submission
Test Setup
July 2006
Royce Fernald - Intel Corporation
Slide 17
doc.: IEEE 802.11-06/0965r1
Submission
Video Content Preparation
• Video content must be instrumented by the video playback analysis application before it can be used for this test
• Instrumentation ensures content-independent VDER results
• Uncompressed source content is first instrumented and then encoded in the desired test format before being loaded on the video traffic generator
• Preparation can be performed in real-time if supported by the video traffic generator
July 2006
Royce Fernald - Intel Corporation
Slide 18
doc.: IEEE 802.11-06/0965r1
Submission
Video Content Preparation Procedure
July 2006
Royce Fernald - Intel Corporation
Slide 19
doc.: IEEE 802.11-06/0965r1
Submission
Video Content Analysis
• After passing through the RF data connection, video content is captured from the DUT display and saved to a file (typically uncompressed)
• The instrumented video capture file is examined for video delivery error and video quality defects by the video playback analysis application and the video quality analysis application
• Analysis can be performed in real-time if supported by the video capture device
July 2006
Royce Fernald - Intel Corporation
Slide 20
doc.: IEEE 802.11-06/0965r1
Submission
Video Content Analysis Procedure
July 2006
Royce Fernald - Intel Corporation
Slide 21
doc.: IEEE 802.11-06/0965r1
Submission
Calibration
• Video performance tests require that the DUT support the video format, bitrate, network transport and streaming protocol used by the video traffic generator
• Video format compatibility must be verified before the test begins
• Video performance measurements can be affected by the capture device, so capture device performance must be verified with local video playback from the DUT
• If there are too many errors detected during calibration, the capture device should not be used for video performance tests
July 2006
Royce Fernald - Intel Corporation
Slide 22
doc.: IEEE 802.11-06/0965r1
Submission
Baseline Configuration
• Maximum transmit power setting
• QoS enabled with video traffic at high priority (no background traffic)
• No security
• No power management
• Video traffic configured to model desired application (i.e. 8 Mbps MPEG2 for DVD-quality video)
• Default DUT buffer size
• No content encryption
July 2006
Royce Fernald - Intel Corporation
Slide 23
doc.: IEEE 802.11-06/0965r1
Submission
Modifiers
• QoS disabled
• QoS enabled with background traffic at low priority
• MAC security
• Power management enabled, different profiles
• Alternate video bitrates and/or network transports
• DUT buffer size
• Video content encryption
July 2006
Royce Fernald - Intel Corporation
Slide 24
doc.: IEEE 802.11-06/0965r1
Submission
Measurement Procedure
• Load video content on the video traffic generator• Associate DUT with WLCP
Repeat the following steps for each attenuation or range increment:
• Start the video capture operation on the capture device and then start the video stream to the DUT
• After the specified test duration, stop the capture operation and then stop the video stream
• Analyze the resulting video capture file with the video playback analysis application and the video quality analysis application to measure the video delivery error rate and the video quality score
• Calculate MDI using metrics from the 802.11 traffic analyzer
July 2006
Royce Fernald - Intel Corporation
Slide 25
doc.: IEEE 802.11-06/0965r1
Submission
Reported Results
• The reported results should include the following items– Calibrated VDER, including the capture device baseline
– Video Quality MOS and DUT MOS upper bound
– MDI MLR and DF
– Detailed information about the test configuration including the video format, network transport and video capture device
Attenuation (dB) or Range (m)
Video Delivery Error Rate (Errors/minute)
Video Quality (MOS)
MDI – MLR MDI – DF
Sample Report:
July 2006
Royce Fernald - Intel Corporation
Slide 26
doc.: IEEE 802.11-06/0965r1
Submission
Sample Data
Compares video streaming performance of wireless adapters “A” and “B”
• Conducted environment
• Standard definition video (720x480 pixels)
• MPEG2 transport stream– 12 Mbps MPEG2 video
– Constant bitrate
– No audio
• UDP video transport
July 2006
Royce Fernald - Intel Corporation
Slide 27
doc.: IEEE 802.11-06/0965r1
Submission
Sample Data – Adapter A
July 2006
Royce Fernald - Intel Corporation
Slide 28
doc.: IEEE 802.11-06/0965r1
Submission
Sample Data – Adapter B
July 2006
Royce Fernald - Intel Corporation
Slide 29
doc.: IEEE 802.11-06/0965r1
Submission
Agenda
Background– Proposal Framework– Test Output– Video Delivery and Video Quality– Measuring Video Delivery and Video Quality– The Media Delivery Index– Video Delivery, Video Quality and Wireless Performance– References
Test Description (doc.: 802.11-06/0964r1)– Introduction and Purpose– Resource Requirements– Test Setup– Video Content Preparation and Analysis– Calibration– Baseline Configuration– Modifiers– Measurement Procedures– Reported Results– Sample Data
Motion
July 2006
Royce Fernald - Intel Corporation
Slide 30
doc.: IEEE 802.11-06/0965r1
Submission
Motion
Move to instruct the editor to incorporate the contents of document 802.11-06/0964r1 into the P802.11.2 draft.
Technical (75%)
Y:
N:
A: