applications of objective quality assessment … class standards agenda introduction scenarios of...

World Class Standards

Applications of objective quality assessment models in IPTV

Applications of objectives quality assessment model s in IPTV (Akira Takahashi)

Akira TakahashiNTT Service Integration Labs.

© ETSI 2007. All rights reservedEFFECTS OF TRANSMISSION PERFORMANCE ON MULTIMEDIA QoS


Agenda

� Introduction

� Scenarios of QoE evaluation

� Classification and application of objective QoE mod els

Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 2

� Classification and application of objective QoE mod els� Media-layer models� Parametric packet-layer models� Bitstream models� Parametric planning models� Hybrid models

� Concluding remarks and future work


What is QoE?

� The overall acceptability of an application or service, as perceived subjectively by the end user.

NOTES1 Quality of Experience includes the complete end-to-end system effects

(client, terminal, network, services infrastructure, etc).2 Overall acceptability may be influenced by user expectations and context.2 Overall acceptability may be influenced by user expectations and context.

~From ITU-T Recommendation P.10/G.100~



Factors affecting QoE of IPTV services

4

Vide

o

Audi

o

Text

Gra

phic

s

Con

trol f

unct

ion

EPG

Met

adat

a

Brow

ser

Acce

ssib

ility

Prici

ng

Supp

ort

QoE of IPTV services

Applications of objectives quality assessment model s in IPTV (Akira Takahashi)


ApplianceTelephone

ＰＣ

QoS &QoS &QoS &QoS & QoEQoEQoEQoE

ＴＶ

Storage serverIP transmitter

Homenetwork

Core/accessnetwork

Transmissionperformance

QoSQoSQoSQoS

ＳＴＢ

ＴＶ

Decoder/PLC

performance

Display QualitySourcequality

QoEQoEQoEQoE

Encoder

Coding quality

5Applications of objectives quality assessment model s in IPTV (Akira Takahashi)


Why QoE evaluation?

� Recent survey indicates that….� 84% of IPTV providers think video quality monitorin g is critical or

very important� 77% say video quality is a main reason for customer churn� 78% say video quality problems are a main reason fo r high volumes

of customer service calls


of customer service calls~From http://telephonyonline.com/iptv/news/iptv-video-qua lity-0219/ ~

� There surely is a market need for QoE evaluation me thods/tools!


Quality monitoring as well as quality design

� Conventional telephone services over PSTN are relat ively stable after a call is connected. � In PSTN services, service providers established tel ephone systems

with static quality design , � then, monitor/manage the “availability” and “functiona lity”

� IP networks (core/access/home -NWs) and IP -based terminals are


� IP networks (core/access/home -NWs) and IP -based terminals are time-variant systems.� In IPTV services, media quality varies depending on coding

performance at H/E, traffic conditions in networks, and terminal performance


What to measure?

� Pre-transmission domain• Encode quality• System failure (blackout, frame freezing etc.)

� Network domain• Packet loss• Delay• Delay• Delay jitter

� Post-transmission domain• Error recovery performance such as FEC (Forward Err or Correction)• Decode quality (including PLC: Packet Loss Concealm ent)• D/A conversion performance• Color management performance• Display performance



When to measure?

� QoE measurement is necessary in the following phase s:

� Before the service is developed:• Individual quality factors (coding distortion etc.)• Quality planning


� After the service is developed, but before the service is delivered:• Off-service quality check

� After the service is delivered:• In-service quality monitoring at user end• In-service network mid-point monitoring• On-site quality check (on claim)


Where to measure?

� Draft ITU-T Recommendation G.1081 （（（（IPTV-PMP））））:PrePre--transmission qualitytransmission quality Network qualityNetwork quality

PostPost--transmission transmission


PostPost--transmission transmission qualityquality


How to measure?

� Subjective quality assessment is time consuming and expensive.

� It cannot be applied to on-site and/or real-time measu rement , which is essential in quality monitoring.


� Objective quality assessment , which estimates QoE by subjective testing, is the solution!


Overview of objective quality assessment models

� From the viewpoint of input information into a mode l, objective quality assessment models can be categorized as fol lows:

� Media-layer models

� Parametric packet -layer models


� Parametric packet -layer models

� Bitstream-layer models

� Parametric planning models

� Hybrid models


Media-layer models

Source signal

System under test

Source video

Model

Degradedsignal

Full-reference (FR) models

Source video Degraded

Example:• Codec optimization• Off-service quality check• Content-encoding quality monitoring at head end

• On-site quality check

Example:


System under test Model

No-reference (NR) models

System under test Model

Reduced-reference (RR) modelsSide info.

Source video Degradedsignal

Source video Degradedsignal

Example:• In-service quality monitoring at user end

• Content-encoding quality monitoring at head end

Example:• In-service quality monitoring at user end


Current standards of media-layer models (video)

� ITU-T standardized Rec. J.144, which recommends four full-reference models for Standard Definition TV , in 2004.� Coding distortion only (no transmission errors such as packet loss)

� VQEG finalized its evaluation on full-reference and reduced-reference models applicable to QCIF ~ VGA. The draft Recommendation has been consented at ITU -T SG9 as Rec. J.247,


Recommendation has been consented at ITU -T SG9 as Rec. J.247, which includes four models, and Rec. J.246, respectively in May 2008.� Packet-loss degradation as well as coding distortio n is in the scope

of Recommendation.

� ITU-T SG9 considered that current no-reference tech nologies seemed to be premature in terms of quality-estimati on accuracy.


Parametric packet-layer models

� Only uses packet-header information.� Light-weight implementation.� Can be applied even if payload is encrypted.

Example:• In-service quality monitoring Decoded media Media-layer

model


• In-service quality monitoring at user end

• Network mid-point monitoring

Cannot consider…•coding distortion•quality dependence of QoE on content

IPUDPRTP

TS, MP4

Decoded media

Parametric packet-layer

model

model

ES (bitstream)


Bitstream-layer models

� Use up to coded bitstream (payload), but not decode r output.� Can take into account the content dependence of the impact of

packet loss.

Example:• In-service quality monitoring

Decoded mediaMedia-layer

model


• In-service quality monitoring at user end

• Computational load• Cannot be applied if

payload is encryptedIP

UDPRTP

TS, MP4

Decoded media

Parametric packet-layer

model

Bitstream-layer model

model

ES (bitstream)


Standardization activities for packet-layer and bitstream-layer models

� Parametric packet-layer models� ITU-T SG12 Q.14 & VQEG

• P.NAMS (non-intrusive parametric model for the asse ssment of performance of multimedia streaming)


� Bitstream-layer models� ITU-T SG12 Q.14 & VQEG

• P.NBAMS (non-intrusive parametric bitstream model f or the assessment of performance of multimedia streaming)


Parametric planning models� Use quality planning parameters as input.� ITU-T Recommendation G.1070 for videophone applicat ions.� ITU-T SG12 Q.13 is working on G.OMVAS (opinion mode l for video

and audio streaming applications).

1) Speech quality Speech-related quality parameters.

Quality estimation model defined in Quality estimation model defined in Quality estimation model defined in Quality estimation model defined in ITUITUITUITU----T Recommendation G.1070T Recommendation G.1070T Recommendation G.1070T Recommendation G.1070

Coding scheme, bitrate


1) Speech quality estimation model

(E-model)

2) Video qualityestimation model

(proposed by NTT)

Video-related quality parameters.

EstimatedMOS

3) Multimedia qualityintegration model(proposed by NTT)

� Coding scheme, bitrate� Packet-loss rate� Delay� Loudness, echo

� Coding scheme, bitrate� Packet-loss rate� Delay� Image resolution, display size Example:

• Quality planning


Concluding remarksMarket trend:� QoE evaluation technologies are needed in IPTV mark et.� Testing equipment market will shift from R&D usage to

maintenance and monitoring applications.

Application scenarios:Application scenarios:� There are various applications of objective quality models in

different quality design/management phases.� Standardization activities are quite active in ITU- T SG9, SG12, and

ITU Video Quality Experts Group (VQEG).

Future issues:� Real-time quality estimation for monitoring� Light-weight, but accurate models for end user moni toring


applications of objective quality assessment … class standards agenda introduction scenarios of...

Documents