quality and performance evaluation of voip end-points wenyu jiang henning schulzrinne columbia...
TRANSCRIPT
Quality and Performance Quality and Performance Evaluation of VoIP End-pointsEvaluation of VoIP End-points
Wenyu JiangHenning Schulzrinne
Columbia UniversityNYMAN 2002
September 3, 2002
MotivationsMotivations The quality of VoIP depends on both
the network and the end-points Extensive QoS literature on network
performance, e.g., IntServ, DiffServ Focus is on limiting network loss & delay
Little is known about the behavior of VoIP end-points
Performance Metrics for Performance Metrics for VoIP End-pointsVoIP End-points Mouth-to-ear (M2E) delay
C.f. network delay Clock skew
Whether it causes any voice glitches Amount of clock drift
Silence suppression behavior Whether the voice is clipped (depends much
on hangover time) Robustness to non-speech input, e.g., music
Robustness to packet loss Voice quality under packet loss
Measurement ApproachMeasurement Approach Capture both original and output audio Use “adelay” program to measure M2E delay Assume a LAN environment by default
Serve as a baseline of reference, or lower bound
ethernet
IP phonecoupler
InOut
notebookoriginalaudio
PCsignalstereo
LAN
speaker
ethernet
coupler IP phoneInOut
line in
(mouth)
(ear)
outputaudio
VoIP End-points TestedVoIP End-points Tested Hardware End-points
Cisco, 3Com and Pingtel IP phones Mediatrix 1-line SIP/PSTN Gateway
Software clients Microsoft Messenger, NetMeeting (Win2K,
WinXP) Net2Phone (NT, Win2K, Win98) Sipc (Solaris, Ultra-10)
Operating parameters: In most cases, codec is G.711 -law, packet
interval is 20ms
Example M2E Delay PlotExample M2E Delay Plot 3Com to Cisco, shown with gaps > 1sec Delay adjustments correlate with gaps,
despite 3Com phone has no silence suppression
35
40
45
50
55
60
0 50 100 150 200 250 300 350
M2E
del
ay (m
s)
time (sec)
experiment 1-1experiment 1-2
silence gaps
Visual Illustration of M2E Visual Illustration of M2E Delay Drop, Snapshot #1Delay Drop, Snapshot #1
3Com to Cisco 1-1 case
Left/upper channel is original audio
Highlighted section shows M2E delay (59ms)
Effect of RTP M-bits on Effect of RTP M-bits on Delay AdjustmentsDelay Adjustments Cisco phone sends M-bits, whereas Pingtel
phone does not Presence of M-bits results in more adjustments
20
30
40
50
60
70
80
90
100
0 50 100 150 200 250 300
M2E
del
ay (m
s)
time (sec)
Cisco to 3Com 1-1Pingtel to 3Com 2-1
new talkspurt (M-bit=1)
Sender CharacteristicsSender Characteristics Certain senders may introduce delay
spikes, despite operating on a LAN
50
100
150
200
250
300
0 50 100 150 200 250 300
M2E
del
ay (m
s)
time (sec)
Mediatrix to 3Com 3-1Mediatrix to Cisco 1-1
Mediatrix to Pingtel 1-1
Average M2E Delays for IP Average M2E Delays for IP phones and sipcphones and sipc Averaging the M2E delay allows more compact
presentation of end-point behaviors Receiver (especially sipc) plays an important role
in M2E delay
0
50
100
150
200
250
Receiver
M2E
del
ay (m
s)
3Com Cisco Mediatrix Pingtel sipc Cisco G.729
Average M2E Delays for Average M2E Delays for PC Software ClientsPC Software Clients Messenger 2000 wins the day
Its delay as receiver (68ms) is even lower than Messenger XP, on the same hardware
It also results in slightly lower delay as sender NetMeeting is a lot worse (> 400ms) Messenger’s delay performance is similar to or
better than a GSM mobile phone.
A B AB BA
MgrXP (pc) MgrXP (notebook) 109ms 120ms
Mgr2K (pc) 96.8ms
68.5ms
NM2K (pc) NM2K (notebook) 401ms 421ms
Mobile (GSM)
PSTN (local number)
115ms 109ms
Delay Behaviors for PC Delay Behaviors for PC Clients, contd.Clients, contd. Net2Phone’s delay is also high
~200-500ms V1.5 reduces PC->PSTN delay PC-to-PC calls have fairly high delays
A B AB BA
N2P v1.1 NT P-2 (pc2)
PSTN (local number)
292ms 372ms
N2P v1.5 NT P-2 (pc2)
201ms 373ms
N2P v1.5 W2K K7 (pc)
196ms 401ms
N2P v1.5 W2K K7 (pc)
N2P v1.5 W98 P-3(notebook2)
525ms 350ms
Effect of Clock Skew: Cisco Effect of Clock Skew: Cisco to 3Com, Experiment 1-1to 3Com, Experiment 1-1
Symptom of playout buffer underflow
Waveforms are dropped
Occurred at point of delay adjustment
Bugs in software?
Clock Drift RatesClock Drift Rates Mostly symmetric between two devices Sipc has unusually high drift rates, >
300 ppm (parts per million)
Drift Rates (in ppm)
3Com Cisco Mediatrix
Pingtel sipc
3Com 55.4 43.3 41.2 -333
Cisco -55.2 -0.4 -11.8 -12.1 -381
Mediatrix -43.1 11.7 -0.8
Pingtel -40.9 12.7 2.8 -380
sipc 343 403 376
Drift Rates for PC ClientsDrift Rates for PC Clients Drift Rates not always symmetric!
But appears to be consistent between Messenger 2K/XP and Net2Phone on the same PC
Existence of 2 clocking circuits in sound card?
A B AB BA
MgrXP (pc) MgrXP (notebook)
172 87.7
Mgr2K (pc) 165 85.6
NM2K (pc) NM2K (notebook)
? -33?
Net2Phone NT (pc2)
PSTN 290 -287
Net2Phone 2K (pc) 166 82
Mobile (GSM) 0 0
Packet Loss ConcealmentPacket Loss Concealment Common PLC methods
Silence substitution (worst) Packet repetition, with optional fading Extrapolation (one-sided) Interpolation (two-sided), best quality
Use deterministic bursty loss pattern 3/100 means 3 consecutive losses out of
every 100 packets Easier to locate packet losses Tested 1/100, 3/100, 1/20, 5/100, etc.
PLC BehaviorsPLC Behaviors Loss tolerance (at 20ms interval)
By measuring loss-induced gaps in output audio
3Com and Pingtel phones: 2 packet losses Cisco phone: 3 packet losses
Level of audio distortion by packet loss Inaudible at 1/100 for all 3 phones Inaudible at 3/100 and 1/20 for Cisco phone,
yet audible to very audible for the other two. Cisco phone is the most robust
Probably uses interpolation
Effect of PLC on DelayEffect of PLC on Delay No affirmative effect on M2E delay
E.g., sipc to Pingtel
50
55
60
65
70
75
80
0 10 20 30 40 50 60
mou
th-t
o-ea
r de
lay
(ms)
time (sec)
0/1003/100
1/20
Silence SuppressionSilence Suppression Why?
Saves bandwidth May reduce processing power (e.g., in
conferencing mixer) Facilitates per-talkspurt delay adjustment
Key parameters Silence detection threshold Hangover time, to delay silence
suppression and avoid end clipping of speech
Usually 200ms is long enough [Brady ’68]
Hangover TimeHangover Time Measured by feeding ON-OFF
waveforms and monitor RTP packets Cisco phone’s is the longest (2.3-2.36
sec), then Messenger (1.06-1.08 sec), then NetMeeting (0.56-0.58 sec)
A long hangover time is not necessarily bad, as it reduces voice clipping Indeed, no unnatural gaps are found Does waste a bit more bandwidth
Robustness of Silence Robustness of Silence Detectors to MusicDetectors to Music On-hold music is often used in customer
support centers Need to ensure music is played without any
interruption due to silence suppression Tested with a 2.5-min long soundtrack Messenger starts to generate many
unwanted gaps at input level of -24dB Cisco phone is more robust, but still
fails from input level of -41.4dB
M2E Delay under JitterM2E Delay under Jitter Delay properties under the LAN environment
serves as a baseline of reference When operating over the Internet:
Fixed portion of delay adds to M2E delay as a constant Variable portion (jitter) has a more complex effect
90100110120130140150160170180
0 20 40 60 80 100 120 140 160 180
mou
th-t
o-ea
r de
lay
(ms)
time (sec)
High jitter (uplink)Low jitter (downlink)
Preliminary results Used typical cable
modem delay traces Tested sipc to Cisco No audible distortion
due to late loss Added delay is normal
ConclusionsConclusions Average M2E delay:
Low (mostly < 80ms) for hardware IP phones Software clients: low (< 120ms) for Messenger, particularly
Messenger 2000 (68.5ms) The application (receiver) is the most vital in determining delay
Poorly implemented end-points can easily undo good network QoS Clock drift rates
Are high for some software clients (sipc, Net2Phone) In some cases non-symmetric!
Packet loss concealment quality Acceptable in all 3 IP phones tested, w. Cisco being most robust
Silence detectors in Cisco phone, Messenger, NetMeeting Long hangover time, no audible unwanted gaps for speech input May falsely predict music as silence