h. w. gierlich - itu. w. gierlich head acoustics gmbh ... 11 participating companies, test lab head...
TRANSCRIPT
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006
International Telecommunication Union
ITU-T
Speech Quality Testing for Speech Quality Testing for
VoIPVoIP Terminals and Terminals and
Gateways: Gateways:
Input from ETSI “Input from ETSI “PlugtestPlugtest” ”
Speech Quality Test Events Speech Quality Test Events
H. W. GierlichHEAD acoustics GmbH
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 2
ITU-T
Overview
o Introduction - 1st, 2nd and 3rd
SQTE
oDiscussion of Results
•Gateways
•IP phones
o Summary & Outlook
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 3
ITU-T
Overview
o Introduction - 1st, 2nd and 3rd
SQTE
oDiscussion of Results
•Gateways
•IP phones
o Summary & Outlook
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 4
ITU-T
Introduction
The ETSI SSpeech QQuality TTest EEvents
• 1st SQTE October 2000 organized and hosted by the ETSI “Bake-off Service”
6 participating companies
test labs HEAD acoustics, Deutsche Telekom
• 2nd SQTE April 2002organized and hosted by the ETSI “Plugtests Service”
5 participating companies
test labs HEAD acoustics, Deutsche Telekom
• 3rd SQTE June and September 2004, “twin event” EU, US organized by the ETSI “Plugtests Service” and HEAD acoustics, hosted by HEAD acoustics
11 participating companies, test lab 11 participating companies, test lab HEAD acousticsHEAD acoustics
•• 44thth SQTESQTE June 2006June 2006, EU – just ongoing
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 5
ITU-T
The Goal
Evaluate all conversational aspects• Listening speech quality (“status quo”)
• Detailed parameter tests
• Echo measurements, EC implementation
• Double talk performance tests
• Quality of background noise transmission
In order to•• Compare different implementationsCompare different implementations
•• Document the state of the arte of technologyDocument the state of the arte of technology
•• Give input to standardizationGive input to standardization
•• Help companies to optimize their implementationsHelp companies to optimize their implementations
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 6
ITU-T
Test Setup Gateways
Condition Packet Loss (Equal) Additional Delay1 Delay Variation
1a 0 0 No
2a 1% 0 No
3a 2% 0 No
4a 3% 0 No
5a 5% 0 No
6a 1% 50 ms 20 ms (2)
Gateway
IP
E1
NISTNet
Packet MONITOR
E1
Packet Loss Delay
Output
HEAD acoustics Test System AACCQQUUAA
Input
Gateway
IP IP
ISDN DSS1 ISDN DSS1
PBX
Condition Packet Loss (Equal) Additional Delay1 Delay Variation
1b 0 0 No
2b 5% 0 No
3b 0 50 ms 20 ms (2)
4b 5% 50 ms 20 ms(2)
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 7
ITU-T
Test Setup IP-Phones
IP
NISTNet
Packet MONITOR
E1
Packet Loss Delay
Output
HEAD acoustics Test System AACCQQUUAA
Input
Gateway
IP
ISDN DSS1
PBX
IP
IP Terminal
(Phone or PC)
Condition Packet Loss (Equal) Additional Delay1) Delay Variation
1c 0 100 ms No
2c 0 100 ms 20 ms(2)
3c 1% 100 ms No
4c 1% 100 ms 20 ms(2)
5c 3% 100 ms No
Condition Packet Loss (Equal) Additional Delay1) Delay Variation
1d 0 100 ms No
2d 3% 100 ms No
3d 0 100 ms 20 ms(2)
4d 3% 100 ms 20 ms(2)
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 8
ITU-T
Overview
o Introduction - 1st, 2nd and 3rd
SQTE
oDiscussion of Results
•Gateways
•IP phones
o Summary & Outlook
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 9
ITU-T
Listening Speech Quality PESQ P.862 (MOSPESQ P.862 (MOS--LQO)LQO)
1
2
3
4
5
0%, 0ms,
VAD on
0%, 0ms 1%, 0ms 2%, 0ms 3%, 0ms 5%, 0ms 1%, 20ms,
r=0.9 (fix
delay 50ms)
1%, 20ms,
r=0.5 (fix
delay 50ms)
network conditions
MO
S-L
QO
0
100
200
300
400
500
600
de
lay
/ m
s
PESQ delaymax. MOSmax. MOS--LQOLQO
minmin
averageaverage
max. max. delaydelay
minminaverageaverage
G.711
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 10
ITU-T
Listening Speech Quality TOSQA2001 (TMOS)TOSQA2001 (TMOS)
1
2
3
4
5
0%, 0ms, VAD
on
0%, 0ms 1%, 0ms 2%, 0ms 3%, 0ms 5%, 0ms 1%, 20ms,
r=0.9 (fixdelay 50ms)
1%, 20ms,
r=0.5 (fixdelay 50ms)
network conditions
TM
OS
0
100
200
300
400
500
600
de
lay
/ m
s
TOSQA delay
G.711
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 11
ITU-T
Listening Speech Quality (MOS(MOS--LQO vs. TMOS)LQO vs. TMOS)
1
2
3
4
5
0%, 0ms,
VAD on
0%, 0ms 1%, 0ms 2%, 0ms 3%, 0ms 5%, 0ms 1%, 20ms,
r=0.9 (fix
delay 50ms)
1%, 20ms,
r=0.5 (fix
delay 50ms)
network conditions
TM
OS
, M
OS
-LQ
O
PESQ TOSQA
For further SQTE results (G.729, G.723) see:For further SQTE results (G.729, G.723) see:
AnonymizedAnonymized Report Report –– “Gateways” “Gateways”
SQTE Results IP Phones: SQTE Results IP Phones:
AnonymizedAnonymized Report Report –– “IP Phones” “IP Phones”
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 12
ITU-T
Packet Loss Concealment
Highly influencing MOS score - optimization
criteriaTest Test signalsignal (5s)(5s) •• ITUITU--T T RecommendationRecommendation P.501 P.501
•• periodicalperiodical repetitionrepetition of a of a voicedvoiced soundsound
•• pitchpitch frequencyfrequency approxapprox. 330 Hz . 330 Hz
Relative ApproachRelative Approach •• hearinghearing modelmodel basedbased forwardforward estimationestimation
•• comparisoncomparison to to actualactual signalsignal
•• displaysdisplays ““estimationestimation errorerror“ “
Cross Cross correlationcorrelation
vs. Timevs. Time•• analysisanalysis of of signalsignal phasephase
•• periodicalperiodical patternpattern ((approxapprox. 3 ms). 3 ms)
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 13
ITU-T
Packet Loss Concealment
U-V
-0.3
-0.2
0.2
0.3
t/s0.5 1 1.5 2 3 3.5 4 4.5 5
f-
Hz
300
2k
4k
t/s0.5 1 1.5 2 3 3.5 4 4.5 5
cp/cPa0 2.5 5 10 12.5 15
t-s
-3m
3m
5m
t/s0.5 1 1.5 2 3 3.5 4 4.5 5
%-100 -75 -50 -25 25 50 75 100
U-V
-0.5
-0.3
0.3
0.5
t/s0.5 1 1.5 2 3 3.5 4 4.5 5
f-
Hz
300
2k
4k
t/s0.5 1 1.5 2 3 3.5 4 4.5 5
cp/cPa0 2.5 5 10 12.5 15
t-s
-3m
3m
5m
t/s0.5 1 1.5 2 3 3.5 4 4.5 5
%-100 -75 -50 -25 25 50 75 100
U-V
-0.3
-0.2
0.2
0.3
t/s0.5 1 1.5 2 3 3.5 4 4.5 5
f-
Hz
300
2k
4k
t/s0.5 1 1.5 2 3 3.5 4 4.5 5
cp/cPa0 2.5 5 10 12.5 15
t-s
-3m
3m
5m
t/s0.5 1 1.5 2 3 3.5 4 4.5 5
%-100 -75 -50 -25 25 50 75 100
U-V
-0.4
-0.2
0.2
0.4
t/s0.5 1 1.5 2 3 3.5 4 4.5 5
f-
Hz
300
2k
4k
t/s0.5 1 1.5 2 3 3.5 4 4.5 5
cp/cPa0 2.5 5 10 12.5 15
t-s
-3m
3m
5m
t/s0.5 1 1.5 2 3 3.5 4 4.5 5
%-100 -75 -50 -25 25 50 75 100
MOSMOS--LQO LQO
5% packet 5% packet lossloss
3.63.63.73.7
3.13.1 3.33.3
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 14
ITU-T
VAD and Comfort Noise
IP
Packet Loss
Delay
GatewayGateway
IP
PBX PBX PBX PBX ISDN
Simulator
ISDN
Simulator
ISDN
Simulator
ISDN
Simulator
NISTNet
Voice Activity Detection
silence suppression
silence packets
comfort noise generation
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 15
ITU-T
VAD und Comfort Noise
Level vs. time Manual(5.0 ms) L/dB[V]
-70
-65
-60
-55
-50
-45
-40
-35
-30
t/s4.5 4.6 4.7 4.8 4.9 5.1 5.2 5.3 5.4 5.5
Level vs. t ime Manual(5,0 ms) L/dB[V]
-70
-65
-60
-55
-50
-45
-40
-35
-30
t/s4.5 4.6 4.7 4.8 4.9 5.1 5.2 5.3 5.4 5.5
Level vs. t ime Manual(5,0 ms) L/dB[V]
-70
-65
-60
-55
-50
-45
-40
-35
-30
t/s4.5 4.6 4.7 4.8 4.9 5.1 5.2 5.3 5.4 5.5
Level vs. time Manual(5,0 ms) L/dB[V]
-70
-65
-60
-55
-50
-45
-40
-35
-30
t/s4.5 4.6 4.7 4.8 4.9 5.1 5.2 5.3 5.4 5.5
Level vs. time Manual(5,0 ms) L/dB[V]
-70
-65
-60
-55
-50
-45
-40
-35
-30
t/s4.5 4.6 4.7 4.8 4.9 5.1 5.2 5.3 5.4 5.5
voiced sound,
decreasing and increasing
level vs. time
test test signalsignal levellevel
measuredmeasured signalsignal levellevel
activationactivation thresholdsthresholds, “, “hysteresishysteresis““
adaptive adaptive comfortcomfort noisenoise
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 16
ITU-T
VAD und Comfort Noise II
Level vs. time Manual(5,0 ms) L/dB[V]
-80
-70
-60
-50
-40
-30
-20
-10
0
t/s5 6 7 8 9 10
Level vs. time Manual(5,0 ms) L/dB[V]
-80
-70
-60
-50
-40
-30
-20
-10
0
t/s5 6 7 8 9 10
Level vs. time Manual(5.0 ms) L/dB[V]
-80
-70
-60
-50
-40
-30
-20
-10
0
t/s5 6 7 8 9 10
Level vs. time Manual(5,0 ms) L/dB[V]
-80
-70
-60
-50
-40
-30
-20
-10
0
t/s5 6 7 8 9 10
Level vs. time Manual(5,0 ms) L/dB[V]
-80
-70
-60
-50
-40
-30
-20
-10
0
t/s5 6 7 8 9 10
Realitsic noise scenario
(students cafe)
noisenoise contrastcontrast
quicklyquickly adaptingadapting comfortcomfort noisenoise
comfortcomfort noisenoise levellevel slightlyslightly tootoo high high
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 17
ITU-T
Setup for Echo Canceller Testing
IP
Packet Loss
Delay
GatewayGateway
IP
PBX PBX PBX PBX ISDN
Simulator
ISDN
Simulator
ISDN
Simulator
ISDN
Simulator
InRCV
OutRCV
InSND
OutSND
NISTNet
USB
MFE VI
Receive direction
Send direction
(Echo (Echo pathpath
realizedrealized in in
MFE VI) MFE VI)
EC under test
far endnear end
Test system
ACQUA„„infinite“infinite“
ERL 40 dB and ERL 40 dB and
ERL 6 dB ERL 6 dB
EC off
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 18
ITU-T
Double Talk Performance
IP
Packet Loss
Delay
GatewayGateway
IP
PBX PBX PBX PBX ISDN
Simulator
ISDN
Simulator
ISDN
Simulator
ISDN
Simulator
InRCV
OutRCV
InSND
OutSND
NISTNet
USB
MFE VI
Testsystem
ACQUA„„infinite“infinite“
ERL 40 dB andERL 40 dB and
ERL 6 dBERL 6 dB
EC under test EC off
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 19
ITU-T
Double Talk Performance 6 dB ERL
U/V
-0.15
-0.1
-50m
0
50m
0.1
0.15
t/s14 14.5 15 15.5 16.5 17 17.5
U/V
-0.15
-0.1
-50m
0
50m
0.1
0.15
t/s14 14.5 15 15.5 16.5 17 17.5
U/V
-0.15
-0.1
-50m
0
50m
0.1
0.15
t/s14 14.5 15 15.5 16.5 17 17.5
U/V
-0.15
-0.1
-50m
0
50m
0.1
0.15
t/s14 14.5 15 15.5 16.5 17 17.5
DT_VAR_S.DAT p/Pa
-1.25
-1
-0.75
-0.5
-0.25
0
0.25
0.5
0.75
1
1.25
t/s0 5 10 20 25 30 echo echo componentscomponents
clippingclipping
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 20
ITU-T
Summary Results: „Gateway Pie“
conversationalconversational aspectsaspects
likelike echo echo behaviourbehaviour, ,
double talk double talk performanceperformance
and and backgroundbackground noisenoise
transmissiontransmission
listening speech quality
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 21
ITU-T
Summary Results: „Gateway Pie“ (ITU-T P.505)
conversational aspects
like echo behaviour,
double talk performance
and background noise
transmission
listening speech qualityEachEach ““slideslide“ “ representsrepresents
oneone parameterparameter
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 22
ITU-T
Summary Results: „Gateway Pie“(ITU-T P.505)
conversational aspects
like echo behaviour,
double talk performance
and background noise
transmission
listening speech qualityAssumptionAssumption: : IndependencyIndependency ofof
parametersparameters, ,
no no interactioninteraction
aspectsaspects
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 23
ITU-T
Summary Results: „Gateway Pie“(ITU-T P.505)
conversationalconversational aspectsaspects
likelike echo echo behaviourbehaviour, ,
double talk double talk performanceperformance
and and backgroundbackground noisenoise
transmissiontransmission
listeninglistening speechspeech qualityqualityMinimum Minimum requirementrequirement
(red (red circlecircle) ) representsrepresents
““averageaverage performanceperformance“ “
of 3of 3rdrd SQTESQTE
……oror
recommendedrecommended
numbersnumbers
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 24
ITU-T
Example Results (4 of 10)
““balancedbalanced“ “
implementationimplementation
jitterjitter bufferbuffer
performanceperformance G.711,G.711,
significantsignificant
disturbancesdisturbances
G.711 PLCG.711 PLC
EC EC insertsinserts high BGN high BGN
modulationmodulation (NLP/CN) (NLP/CN)
BGN BGN modulationmodulation
causedcaused byby VAD/CN VAD/CN
violationviolation of of
requirementrequirement onlyonly
causedcaused byby CN CN levellevel(echo (echo attenuationattenuation high high enoughenough))
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 25
ITU-T
Overview
o Introduction - 1st, 2nd and 3rd
SQTE
oDiscussion of Results
•Gateways
•IP phones
o Summary & Outlook
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 26
ITU-T
Testing IP Phones @ 3rd SQTE
Tests in handset and hands-free mode
For handset
• Frequency response,
…
• TMOS using
TOSQA2001
• Echo measurements
• Double talk
performance
• Quality of
background noise
transmission
2 N
13 N
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 27
ITU-T
Testing IP Phones @ 3rd SQTE
Receiving
frequency
responses
and TMOS
L/dB[Pa/V]
-30
-20
-10
0
10
20
30
f/Hz240 300 400 500600 1200 1600 2400 4000
L/dB[Pa/V]
-40
-30
-20
-10
0
10
20
f/Hz240 300 400 600 1200 1600 2400 4000
L/dB[Pa/V]
-30
-20
-10
0
10
20
30
f/Hz240 300 400 600 1200 1600 2400 4000
L/dB[Pa/V]
-30
-20
-10
0
10
20
30
f/Hz240 300 400 500600 1200 1600 2400 4000
3.1 TMOS 2N
3.6 TMOS 8N
3.7 TMOS 13N
3.2 TMOS 2N
3.7 TMOS 8N
3.9 TMOS 13N
2.9 TMOS 2N
3.5 TMOS 8N
3.8 TMOS 13N
3.1 TMOS 2N
3.8 TMOS 8N
4.0 TMOS 13N
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 28
ITU-T
Listening Speech Quality TOSQA2001 (TMOS)TOSQA2001 (TMOS)
1
2
3
4
5
sending 2N 8N 13N 0%, 0ms 0%, 20ms,
r=0.9
1%, 0ms 1%, 20ms,
r=0.9
3%, 0ms 1%, 20ms,
r=0.5
network conditions
TM
OS
0
100
200
300
400
500
600
700
800
de
lay
/ m
s
TOSQA delay
G.711, sending & receiving direction
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 29
ITU-T
Double Talk Test Hands-Free Terminals
Level vs . t ime Manual(5,0 ms) Ch.1 - Ch.2 L/dB
-60
-50
-40
-30
-20
-10
0
10
t/s0.25 0.5 0.75 1 1.25 1.75 2 2.25 2.5
Level vs. time Manual(5,0 ms) Ch.1 - Ch.2 L/dB
-60
-50
-40
-30
-20
-10
0
10
t/s0.5 0.75 1 1.25 1.75 2 2.25 2.5 2.75
Level vs. time Manual(5,0 ms) Ch.1 - Ch.2 L/dB
-60
-50
-40
-30
-20
-10
0
10
t/s0.25 0.5 0.75 1 1.25 1.75 2 2.25 2.5
Level vs. time Manual(5,0 ms) Ch.1 - Ch.2 L/dB
-60
-50
-40
-30
-20
-10
0
10
t/s0.25 0.5 0.75 1 1.25 1.75 2 2.25 2.5
Reference
Double talk test:
Attenuation
range
in sending
direction
(P.340)
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 30
ITU-T
Summary Results: “IP-Phone Pies“(ITU-T P.505)
conversationalconversational aspectsaspects
handsethandset
-- echo, double talk, echo, double talk,
BGN BGN transmissiontransmission --
conversationalconversational aspectsaspects
handshands--freefree
-- echo, double talk, echo, double talk,
BGN BGN transmissiontransmission --
handset
listening speech quality
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 31
ITU-T
Summary Results: “IP-Phone Pies“
““balancedbalanced“ “
handsethandset
implementationimplementation……
““balancedbalanced“ “ handsethandset
ImplementationImplementation……
BGN BGN modulationmodulation
ITUITU--T P.340 T P.340
““typetype 3“ 3“ implemimplem..
BGN BGN modulationmodulation
ITUITU--T P.340 T P.340
““typetype 3“ 3“ implemimplem..
G.729 and G.729 and
handshands--freefree
notnot testedtested
BGN BGN modulationmodulation
ITUITU--T P.340 T P.340
““typetype 3“ 3“ implemimplem.,.,
ViolationViolation causedcaused
byby noisenoise levellevel
(echo (echo attenuationattenuation
high high enoughenough))
ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 32
ITU-T
Some quotes from the 3rd SQTE event:
o Michael Metzger, Executive Director of Marketing, Mindspeed Technologies:• "... bringing together the world leaders in VoIP technology in a co-operative
and neutral environment, this prestigious event continues to improve thequality of VoIP...„
o Peter Faxel, member of the executive board, AVM;• "... The ETSI event has helped us towards achieving our goals, and given us
valuable information about the market trend and technologicaldevelopments."
o Daniel Hartnett, Infineon Technologies:• Plugtests(tm) service of ETSI is an invaluable exercise for a company that
sees voice quality as a key element for the success of our telephoneterminal product families. Infineon will continue to "The VoIP SpeechQuality Test Event (SQTE) organised by the work closely with HEAD acoustics and ETSI to benchmark their products for voice qualityexcellence"
Overall Overall qualityquality of of thethe eventevent: Score 4.6 out of 5: Score 4.6 out of 5