etsi stq, taiwan workshop, february 13th, 2006 1 recent improvements in transmission quality...
Post on 27-Dec-2015
214 Views
Preview:
TRANSCRIPT
ETSI STQ, Taiwan Workshop, February 13th, 2006
1
Recent improvements in transmission quality assessment :
Background noise transmission
Results of STF 273, STF 294
Jean-Yves Monfort, France Telecom
ETSI TC STQ Chairman
ETSI STQ, Taiwan Workshop, February 13th, 2006
2
STFs
STF : Special Task Force Founded by ETSI (STF 273) and e-Europe (STF 294) Several Experts involved in a project new inputs in
standards
STF 273 Speech quality performance in the presence of background noise
A noise data base, freely available
EG 202 396-1 STF 294 Improving the quality of eEurope wideband
speech applications by developing a standardised performance testing and evaluation methodology for background noise transmission
EG 202 396-2
ETSI STQ, Taiwan Workshop, February 13th, 2006
3
STF 273 Speech quality performance in the presence of background noise
Objectives : a close-to-original recording and reproduction system
The background noise recording technique should be– Easy to use – Easy to calibrate – Capable of wideband recording– Available at reasonable costs– Mostly compatible with existing standards and procedures used in telecommunications testing
ETSI STQ, Taiwan Workshop, February 13th, 2006
4
STF 273 Speech quality performance in the presence of background noise
Objectives : a close-to-original recording and reproduction system
The background noise simulation (reproduction) arrangement should be:– easy to set up – usable in typical office type room environments without special acoustical treatment– provide a mostly realistic background noise simulation for all typical background noises occurring in telecommunication applications– easy to calibrate – mostly insensitive to the positioning of (test) objects in the simulated sound field– applicable to all typical terminals used in telecommunication– available at reasonable costs
ETSI STQ, Taiwan Workshop, February 13th, 2006
5
STF 273 : The Recording Setup
Recordings for the setup in office rooms with Artificial heads (ITU-T Rec.
P.58) or Binaural probe microphone
(MIRE technique)
artificial headequalizer right
AH or BMEqualizer right
Artificial Head (AH)
or
Binaural Microphone
worn by a subject (BM)
Upload toData Base
AH or BMEqualizer left
ETSI STQ, Taiwan Workshop, February 13th, 2006
6
STF 273 :The Background Noise Simulation Setup: 4.1 - Technique
Subwoofer
Analyser (1/2-channel)
2m 2m
2m 2m
Playbackfrom
database
Terminal under test
ETSI STQ, Taiwan Workshop, February 13th, 2006
7
STF 273 : Comparison of Terminal Performance: Simulated vs. Real
Background Noise
L/dB[V]
-100
-90
-80
-70
-60
-50
-40
-30
f /Hz100 200 500 1000 2000 5000
MobPh1_BGN SP_original.3rdOctave(4096,50.0%)MobPh1_BGN SP_simulated.3rdOctave(4096,50.0%)
L/dB[V]
-100
-90
-80
-70
-60
-50
-40
-30
f /Hz20 50 100 200 1000 2000 5000 10k 20k
Headset_original.3rdOctave(4096,50.0%)Headset_simulated.3rdOctave(4096,50.0%)
L/dB[V]
-100
-90
-80
-70
-60
-50
-40
-30
f /Hz100 200 500 1000 2000 5000
DesktopHFT_original.3rdOctave(4096,50.0%)DesktopHFT_simulated.3rdOctave(4096,50.0%)
L/dB[V]
-100
-90
-80
-70
-60
-50
-40
-30
f /Hz50 100 200 500 1000 2000 5000 10k
WB_HFT_original.3rdOctave(4096,50.0%)WB_HFT_simulated.3rdOctave(4096,50.0%)
Headset
Mobile phone
Desktop HFTWidebandconference system
Green – originalRed - simulated
ETSI STQ, Taiwan Workshop, February 13th, 2006
8
STF 273 Background noise Recording in cars
Instead of an artificial head, two methods for sound pick-up and equalization :
• the microphone of the terminal under test,• a pair of cardioid microphones
Background noise Reproduction in cars
ETSI STQ, Taiwan Workshop, February 13th, 2006
9
STF 294
Improving the quality of eEurope wideband speech applications by developing a standardised performance testing and evaluation methodology for background noise transmission
Objectives : to develop a quality model for wideband speech for new multimedia services (e-health, e-learning, e-bussiness,…)
Program in two phases Phase 1: Network simulation, Subjective tests (in french
and csez langages) on realistic scenarios with background noise. Finalized in end 2006;
• EG 202 396-2 available
Phase 2 : Development and validation of an objective meyhodology for background noise transmission.
Under Progress August 2006
ETSI STQ, Taiwan Workshop, February 13th, 2006
10
Why Background noise is a problem in mostly all situations and conditions?Both networks and terminals involved
In mobile communication background noise is almost always present; Problem emphasized due to the acoustical construction of
mobile terminals. Situation even worse for mobile hands-free terminals and
mobile car hands-free terminals. In multimedia conference and classroom systems
hands-free type terminal with quite high distance between microphone und speaker.
Computer terminals (e.g. in VoIP scenarios) often used as hands-free terminals Background noise may be produced by the computer/handheld
device itself but may also be picked up from the environment such as an open plan office.
Effects of transmission network impairments in packet based communication networks, Wireless LAN,…
ETSI STQ, Taiwan Workshop, February 13th, 2006
11
Scope of the work STF 294
The aim of the work is to find methods for background noise transmission quality evaluation for real life network scenarios with speech present at the near end and at the far end as well as during periods where no speech is present but only background noise is transmitted.
Steps :• Setup and verification of a transmission network simulation
environment using realistic network scenarios for laboratory use;
• Setup of a database containing the relevant transmission network models and traffic patterns that would be used for subjective and objective evaluation;
• Subjective tests forming the basis for objective evaluation procedures, using the background noise database developed in the currently active ETSI STF 273;
• Evaluation and validation of background noise transmission evaluation procedures.
ETSI STQ, Taiwan Workshop, February 13th, 2006
12
Phase 1 Tasks – References for the development of the new objective Method
Phase 1 (Leader France Telecom) Task 1: Transmission network simulation: Telefonica
A transmission network simulation environment needs to be setup, to provide comparable network characteristics and traffic patterns as found in conditions of the real transmission networks.
Task 2: Database for transmission networks and conditions evaluation/simulation Telefonica
A database containing the typical and realistic transmission network scenarios and traffic patterns needs to be created. There are no risks expected in this task, the procedures are known by the experts in this field.
Task 3: Subjective tests, France Telecom (leader), Mesaqin
Subjective test results providing the speech quality parameters in the background noise situation by the individual parameters are to be conducted. This task must be conducted by experts in the field of subjective testing.
ETSI STQ, Taiwan Workshop, February 13th, 2006
13
STF 294
SFT-Project„BackgroundNoise Simulation“- Background noise simulation
setup- Background noise database
STF 273(existing STF-Project)
Task 1, 2eEurope„NetworkSimulation“
Task 3eEurope„SubjectiveTesting“
Task 4eEurope„ObjectiveMethod“
Task 5eEurope„Validation“
Task 6eEurope„FinalStandardization“
Phase 1 Phase 2
ETSI-guide,background Noise
database
ETSI-guide,database with subj.
results
ETSI-guide,network conditions
database
ETSI-guide,draft objective test
method
ETSI-guide, validated objective test
method
final ETSI-guide(s),databases & evaluation
method
ETSI STQ, Taiwan Workshop, February 13th, 2006
14
Task 1 : Transmission network simulation
Evaluation of simulation methodologies and tools Choice of the best simulation methodologies and tools Setup of simulation environment in the laboratories of the STF
experts Validation of the simulation environment The transmission network simulation tool chosen by the
experts of STF 294 is an IP network simulator (namely : NIST Net). The network simulation is supposed to provide comparable network characteristics and traffic patterns as found in conditions of the real transmission networks (in terms of packet loss, delay and delay variation distributions).
ETSI STQ, Taiwan Workshop, February 13th, 2006
15
Task 2 : Database for transmission networks and conditions evaluation/simulation
Definition of the different network scenarios and conditions to be recorded
Preparation of the simulation recordings
Editing and preparing the data base
Description of the data base (STQ document)
ETSI STQ, Taiwan Workshop, February 13th, 2006
16
Steps in Recording Process
Step1.wav
Speech sample database (4320 samples)
Database of noise type/wideband terminal/network impairment combinations
Speech samples with noise and terminal conditions (720 samples)
Speech samples with background noise conditions (80 samples)
Noise-free speech samples (8 samples)
Step 1 :Clean speech sequences recording
Step 2 :Noisy conditions adding
Step 3 : Wideband terminal simulation
Step 4 : Network simulation
Step2.wav
ETSI STQ, Taiwan Workshop, February 13th, 2006
17
Recorded speech samples
For the recordings performed by ETSI STF 294 the following conditions have been used: 48 kHz (16 bit) sampling rate; wave format; and the active speech level has been equalized to -26 dBoV (see ITU-T
Recommendation P.56 [24], no filtering applied). Also for STF 294, the number of samples to have a
representative database large enough for subjective and objective evaluation has been decided: 4 speakers (2 male, 2 female), 8 sentences each; two languages: Czech and French. The length of recordings varies between 24 and 73 s. They contain
neutral sentences of 2-3s length separated by pauses. Speech activity factor varies between 30 and 60 %. This arrangement allows for easy later editing to any required format.
Simulated terminals : Mobile Handset in the street Hands-Free with PC Hands-Free in car
ETSI STQ, Taiwan Workshop, February 13th, 2006
18
Noisy Conditions / Noisy signal Processing
Noisy Conditions 5 background noise conditions selected:
• cafeteria noise;• office room noise;• road noise;• crossroads; and• car noise (car hands‑free at 130 km/h).
Each combination of speech file plus background noise is recorded at two microphone‑loudspeaker positions:
• typical handset microphone position (with loudness ratings adjusted to 7 dB); and
• hands‑free microphone position (with loudness ratings adjusted to 11 dB).
Noisy signal Processing Three parameters
• With and without VAD (Voice Activity Detection)• 2 different noise reduction filters• 2 different noise reduction levels
ETSI STQ, Taiwan Workshop, February 13th, 2006
19
Task 3 : Subjective tests
Preparation of the test material
Conducting experts tests
Evaluation of the experts test results
Preparing test material for naïve listeners tests
Conducting naïve listeners tests (2 languages : French and Czech)
Evaluation of the results
Statistical analysis the test results
ETSI STQ, Taiwan Workshop, February 13th, 2006
20
Listening testsMethod used : according to ITU-T Recommendation P.835
three questions :- perception and judgment of noise level reduction,- perception and judgment of possible distortion on
speech signal,- overall perception of quality.
All these questions must be answered on a discrete 5 level scale.
France Telecom R&DTest Laboratory
Mesaquin Test Lab
ETSI STQ, Taiwan Workshop, February 13th, 2006
21
Exemples of results (1)
MOS distribution and associated 95% CI per noise condition for Czech and French data
C zech
French
Car
-han
dset
Car
-han
ds-f
ree
Cro
ssro
ad-h
ands
et
Cro
ssro
ad-h
ands
-fre
e
Roa
d-ha
ndse
t
Offi
ce-h
ands
et
Offi
ce-h
ands
-fre
e
Caf
eter
ia-h
ands
et
NO ISE
1,0
1,5
2,0
2,5
3,0
3,5
4,0
4,5
5,0
MO
S
ETSI STQ, Taiwan Workshop, February 13th, 2006
22
Exemples of results (2)
MOS distribution and associated 95% CI per network condition for Czech and French data (NETWORK I: delay jitter 0ms. packet loss 0%. NETWORK II: delay jitter 10ms. packet loss 1%. NETWORK III: delay jitter 20ms. packet loss 3%)
C zech
French1 2 3
NETW O R K
1,0
1,5
2,0
2,5
3,0
3,5
4,0
4,5
5,0
MO
S
ETSI STQ, Taiwan Workshop, February 13th, 2006
23
Exemples of results (3)
MOS distribution and associated 95% CI per NSA condition
No
NS
A
no v
ad_s
moo
th_9
no v
ad_s
moo
th_1
8
no v
ad_s
harp
_9
no v
ad_s
harp
_18
vad_
smoo
th_9
vad_
smoo
th_1
8
vad_
shar
p_9
vad_
shar
p_18
NSA
1,0
1,5
2,0
2,5
3,0
3,5
4,0
4,5
5,0
MO
S
ETSI STQ, Taiwan Workshop, February 13th, 2006
24
What do we also find in EG 202 396-2
Performance parameter descriptions (IP Transmission) Delay (from codec, packetization, network,…) Jitter Packet loss
Parameter interaction and dependences
Wideband speech codecs Characteristics Descriptions
ETSI STQ, Taiwan Workshop, February 13th, 2006
25
What do we also find in EG 202 396-2
Wideband speech codecs Characteristics
• Speech coding algorithm• DTX (discontinuous transmission)• Packet Loss Concealment (PLC)• …
Description of some wideband codecs• ITU-T G.722• AMR WB• …
ETSI STQ, Taiwan Workshop, February 13th, 2006
26
Next Phase – Development, Validation and sharing of he new objective Method
Task 4: Objective methodAn objective method for the quantification of
background noise transmission performance needs to be developed. New methods shall be developed and/or existing methods have to be adapted to the problem.
Task 5: Validation of the new methodThe validation has to be performed independently from
the development of the background noise evaluation method based on unknown data acquired in task 3.
Task 6: Final standardization of the new method
The final standard(s) are drafted and adapted to the results of the previous tasks. An STQ workshop is planned and realized.
Workshop in Praha, November 13-14, 2006
ETSI STQ, Taiwan Workshop, February 13th, 2006
27
Complementary informations
Visit the STF 273 and STF 294 Web sites
http://portal.etsi.org/STFs/STQ/STF273.asp http://portal.etsi.org/STFs/STQ/STF294.asp
You will find :
ToRs of the STF More informations on the work programme Coordinates of the experts Database, results,…
top related