Quality of Service and Quality of Experience@ T-Labs Berlin

Sebastian Möller, Alexander Raake and Marcel Wältermann

Quality and Usability LabDeutsche Telekom LaboratoriesTU Berlin

{sebastian.moeller, alexander raake, marcel.waeltermann}@telekom.de

1

Outline

Quality of Service (QoS) and Quality of Experience (QoE)

– Definitions

– Quality taxonomy

Subjective evaluation methods

Quality prediction

2

QoS and QoEDefinitions.

System developer‘s point-of-view:

Performance: “The ability of a unit to provide the function it has been designed for.”(Möller, 2005)

Quality of Service (QoS): “The collective effect of service performance which determines the degree of satisfaction of the user of the service.” (ITU-T Rec. E.800, 1994)

Includes service support, service operability, serveability, and service security

User‘s point-of-view:

Quality: “Result of appraisal of the perceived composition of the service with respect to its desired composition.”(ITU-T Rec. P.851, 2003, following Jekosch, 2000, 2005)

Quality of Experience (QoE): “The overall acceptability of an application or service,as perceived subjectively by the end user.” (ITU-T Rec. P.10, 2007)

Includes the complete end-to-end system effects

May be influenced by user expectations and context

3

QoS and QoEDefinitions.

Definitions discussed by participants of the Dagstuhl Seminar 09192"From Quality of Service to Quality of Experience“ (May 2009)

Quality of Experience (QoE): “Degree of delight of the user of a service.In the context of communication services, it is influenced by content,network, device, application, user expectations and goals, and context of use.”

Service: “An event in which an entity takes the responsibility that something desirable happens on the behalf of another entity.”

Acceptability: “Characteristic of a service describing how readily a person willuse the service. Acceptability is the outcome of a decision which is partially based on the Quality of Experience.”

4(Möller et al., 2009)

QoS and QoETaxonomy.

Hedonic Pragmatic

User Context

Cognitiveworkload

Perceptualeffort

Physical responseeffort

Formappropriatness

Output modalityappropriatness Contextual

appropriateness DialogmanagementperformanceInput

performance Input modalityappropriatness

Interpretationperformance

Environmentalfactors

Servicefactors

Inputquality

Output quality

Cooperativity

Efficiency

Interaction quality

Effectiveness

System Personality Utility

Usefulness

Acceptability

Joy of use

Learnability

IntuitivityEase of use

Usability

Quality of Experience (Q

oE)

Staticfactors

Dynamicfactors

User System

System

Agentfactors

Functionalfactors

Appeal

Aesthetics

Quality of Service (Q

oS)

Interactionperform

anceInfluencing

factors

5

Outline

Quality of Service (QoS) and Quality of Experience (QoE)

Subjective evaluation methods

– Dimension-based approach

– Evaluation of conversational quality

Quality prediction

6

Dimension-based Assessment of Speech QualityAuditory analysis.

(Wältermann et al., 2006; 2008)

Auditory tests

– Multidimensional Scaling

– Semantic Differential

Quality Dimensions:

– “Discontinuity”

– “Noisiness”

– “Coloration”

7

Dimension-based Assessment of Speech QualityModelling principle.

(Heute et al., 2005; Wältermann et al. 2009)

ModelEstimatedQuality Index

TransmissionSystem

Estimated Quality Dimensions

Knowledge

SubjectiveQuality and FeatureJudgements

dis noi col

tot

8

Subjective Evaluation MethodsEvaluation of conversational quality – from 2 to 3 subjects.

21 3

welcome

persons

summary

discussion of open question

goodbye

interactivetask

open question

request/proposal

objection/proposal

necessaryinformation

9

3CTs conversation scenarios. Conversation tests.

Two conversation test series conducted, yielding a total of 2*8*10 = 160 (recorded) 3-person conversations.

Conversation analysis ongoing.

– Basis for further research.

Speech quality judgments

– 3D Audio versus diotic.

– In conversation, reduced spatial quality advantage when compared with listening only (e.g. Baldis, 2001; Raake et al., 2007)

Listening test in preparation including memory & performance.

(Raake and Schlegel, 2008)

10

Outline

Quality of Service (QoS) and Quality of Experience (QoE)

Subjective evaluation methods

Quality prediction

– Signal-based: P.OLQA

– Parametric: E-model

– IPTV

11

Quality Prediction ModelsSignal-based speech quality prediction: P.OLQA.

(Wältermann et al., 2008)

TransmissionSystem

Model

Listener

Measurement Tool

MOS

Estimated MOS

Linear Dist.

Signal Processing

Packet loss

RoomAcoustics Noise

Codec...

12

P.OLQA Candidate Model.Signal-based Core & Single Dimension Estimators.

TransmissionSystem

InternalRepresent.

InternalRepresent.

Pre-Processing

Pre-Processing

Comparison Integration Transform.

dMOS

NoisinessIndicator

ColorationIndicator

LoudnessIndicator

DiscontinuityIndicator

(Côté 2008; Wältermann et al., 2008b,c)

bIdisbInoibIcol ≈ IbwbIlou

13

Quality Prediction ModelsParametric speech quality prediction: E-model.

Backgr.noise,acousticcoupling

Linear distortion, delay

Codec Packetloss

Jitterbuffer,VAD

Talker echo,listener echo

Circuitnoise

Backgr.noise,acousticcoupling

IP WANIP WAN

4

4

14

IP WANIP WAN

4

4

Overall quality R = Ro - Is - Id - Ie,eff

Estimated user judgment MOS = f (R )

Impairments SNR simultaneous delayed nonlin./timevar.

Ps, Ds,STMR

SLR, RLR, Ta

Ie, qdu PplBpl TELR, T,WEPL, Tr

Nc, Nfor Pr, Dr,LSTR

Quality Prediction ModelsParametric speech quality prediction: E-model.

15

Quality Prediction ModelsParametric speech quality prediction: E-model.

Extension to wideband and beyond

(Raake, 2006; Möller et al, 2006; Appendix II, ITU-T Rec. G.107, 2006)

0 20 40 60 80 1000

20

40

60

80

100

120

140Rmax=129→

RNB/WBR

NB

Ro,max = 129AMR-WB (23.05)

AMR-WB (6.6)

16

Quality of IPTVOverview – perceptual bitstream model

(Raake & Garcia, 2008; Garcia & Raake, 2009)

17

Quality of IPTVVideo quality model – first algorithm

VVVVV FdisFtraFcodFresQoQv ++++=

bitrateccbbPpl

PplFcodbFtra

abitrateaaFcodQvFdisFresQo

VV

V

subjectiveVVV

/

)(

)exp(

)max(

211

12

321

+=+

⋅−=

+⋅⋅=

=++

(Raake et al., 2008)

BplPplPplIeIeeffIe+

⋅−+= )95(,

effIeIdIsRoR ,−−−=

Video VoIP (E-model)

Comparison with test results: r > 0.95,RMSE < 7

18

G.729, 2 frames/packet, native PLC

Qua

lity

(test

)

Quality of IPTVVideo quality under packet loss

2-state Markov lossParemeter: Average number of packets lost in a row μ10 = 1/q

H.264, SD, 4 MBit/s, 1 fpsPLCs: 1 slice/2 Macroblock lines vs. skipping

Video VoIP

Thank you for your attention.

Visit www.qu.tlabs.tu-berlin.de for more information.