1-800-OVERLAYS:Using Overlay Networks to Improve VoIP Quality

Yair Amir, Claudiu Danilov, Stuart Goose,

David Hedqvist, Andreas Terzis

NOSSDAV 2005 2

Voice over IP

• To call or not to call The Internet started as an overlay on top of the

telephone network

• People expect high quality and reliable telephony While not impressed by constant good quality, users are

easily disappointed by a few bad or dropped calls

• High quality calls demand predictable performance The best-effort service offered by the Internet was not

designed to offer any quality guarantees Communication subject to dynamic loss, delay, jitter,

path failures VoIP is interactive. Humans perceive delays at 100ms

NOSSDAV 2005 3

Quality Degradation with Loss

2.5

3

3.5

4

4.5

0 1 2 3 4 5 6 7 8 9 10

Loss rate (%)

PE

SQ

- A

vera

ge Normal

25% burst

50% burst

75% burst

2.5

3

3.5

4

4.5

0 1 2 3 4 5 6 7 8 9 10

Loss rate (%)

PE

SQ

- 5

per

cen

tile

Normal

25% burst

50% burst

75% burst

• G.711 – High quality voice codec• PESQ – Standardized measure of voice quality• Internet characteristics in 2003 [Andersen et al, 03]:

Average loss rate: 0.42% can be as high as 13% Conditional loss probability (burstiness): 66%

PSTN

50ms network delay

NOSSDAV 2005 4

Solution Highlights

• Use overlay networks to break end-to-end connections into multiple, shorter hops

• Localized real-time recovery on overlay links Retransmission is attempted only once Only if the packet is likely to arrive in time at destination

• Flexible routing avoids chronically congested paths Cost metric based on measured latency and loss rate of

the links Link cost equivalent to the expected packet latency when

retransmissions are considered

NOSSDAV 2005 5

Spines

• A generic overlay network research platform Spines builds an overlay router (daemon) on top of UDP,

running as a regular user application Researchers can build protocols within its framework,

experimenting with routing, link protocols, etc.

• Transparent API API similar to the socket interface, giving TCP, UDP and

IP Multicast functionality

• A deployable platform Improving application performance over the Internet Enabling new services Open source (www.spines.org)

NOSSDAV 2005 6

The Spines Architecture

• Daemons create an overlay network on the fly• Clients are identified by the IP address of their daemon and a port ID• Clients feel they are working with UDP and TCP using their IP and port

identifiers• Up to several hundreds of daemons (locations). Each daemon can

handle up to about 1000 clients

NOSSDAV 2005 7

Real-time Recovery Protocol

loss≈2⋅p2

• Each node keeps a history of the packets forwarded in the last 100ms When the other end of a link detects a loss, requests a

retransmission, and moves on If the upstream node still has the packet in the history, resends it

• Not a reliable protocol No blocking. No ACKs. No duplicates.

• Able to recover packets only for links shorter than 30 ms Overlay links are short !

retrdelay=3⋅TΔ

NOSSDAV 2005 8

VoIP Quality Improvements

• Spines overlay – 5 links of 10ms each• 10 VoIP streams sending in parallel• Loss on middle link C-D

2.5

3

3.5

4

4.5

0 1 2 3 4 5 6 7 8 9 10

Loss rate (%)

PE

SQ

- A

vera

ge

Spines Normal

Spines 25%

Spines 50%

Spines 75%

UDP Normal

UDP 75%

2.5

3

3.5

4

4.5

0 1 2 3 4 5 6 7 8 9 10

Loss rate (%)

PE

SQ

- 5

per

cen

tile Spines Normal

Spines 25%

Spines 50%

Spines 75%

UDP Normal

UDP 75%

A B C D E F1 0 m s1 0 M b p s

1 0 m s1 0 M b p s

1 0 m s1 0 M b p s

1 0 m s1 0 M b p s

1 0 m s1 0 M b p s

… …

50ms network delay

NOSSDAV 2005 9

Real-time Routing Metric

• Routing algorithm that takes

into account retransmissions

• Which path maximizes the number of packets arriving at node E in under 100 ms ?

• Finding the best path by computing loss and delay distribution on all the possible routes is very expensive

• Weight metric for links that approximates the best path

A

B

D E

C

?

1 0 m s , 2 % l o s s

2 0 m s , 1 % l o s s

Explatency=1− p ⋅T p−2⋅p2 ⋅3⋅TΔ2⋅p2⋅T max

NOSSDAV 2005 10

Routing Simulation

• Evaluated different routing metrics on random topologies generated by BRITE [Medina et al, 01] On each topology, the nodes defining the diameter of the network

(further appart) were chosen as sender and receiver Random loss rate from 0% to 5% on half of the links

• Optimizing Exp_latency metric compared with: hops: Number of hops in the path latency: Delay of the path loss: Cumulative loss on the path greedy: Dijkstra algorithm that computes delay distributions at each

iteration and selects the partial path with maximum delivery ratio best path: Computed out of all the possible paths

NOSSDAV 2005 11

Routing Simulation (2)

• Each point in the graphs is an average over 1000 different topologies generated with BRITE

• Our simulator could not compute best path for topologies with more than 16 nodes in a timely manner

15 nodes; 30 links 50 nodes; 100 links

96

96.5

97

97.5

98

98.5

99

99.5

100

0 10 20 30 40 50 60 70

Network diameter (ms)

Avg. delivery ratio (%)

best path

exp latency

latency

greedy

hops

loss

96

96.5

97

97.5

98

98.5

99

99.5

100

010203040506070

Network diameter (ms)

Avg. delivery ratio (%)

exp latency

latency

greedy

hops

loss

NOSSDAV 2005 12

Summary

• The voice quality is aversely affected by network conditions

• An overlay network approach breaks end-to-end connections into multiple, shorter hops

• Localized recovery on overlay links Takes care of sudden increases in loss

• Routing metric equivalent to the expected packet latency when retransmissions are considered Avoids long term congested paths

NOSSDAV 2005 13

Thanks!

NOSSDAV 2005 14

Additional Slides

• For the challenging questions from the audience…

NOSSDAV 2005 15

Application-level Routing

• Scheduling – expensive when competing with other processes Time-sensitive messaging

systems should be run with high priority

• Spines overhead < 0.15ms when the overlay nodes run with real-time priority, even under a load of 10

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0 50 100 150 200

VoIP streams

Ave

rag

e p

acke

t d

elay

(m

s)

Spines

UDP

1 0 0 M b p s L A NA B C

1 0 0 M b p s L A N

… …

NOSSDAV 2005 16

The Spines Architecture (2)

Datalink (UDP/IP unicast)

Control LinkBest EffortData Link ReliableData Link

ReliableDatagram

HelloProtocolStateFlood

Routing

DataForwarder

SessionAPILibrary

Daemon-Client Interface

Ove

rlay

Nod

e

Ove

rlay

Lin

k

Link StateGroup State

Real-timeData Link

ReliableSession

Application Client

NOSSDAV 2005 17

Packet Delay Distribution

0

1 0

2 0

3 0

4 0

5 0

6 0

7 0

8 0

9 0

1 0 0

8 0 8 2 8 4 8 6 8 8 9 0 9 2 9 4 9 6 9 8 1 0 0

P a c k e t s ( % )

De

lay

(m

s) U n i f o r m

2 5 % b u r s t

5 0 % b u r s t

7 5 % b u r s t

0

1 0

2 0

3 0

4 0

5 0

6 0

7 0

8 0

9 0

1 0 0

8 0 8 2 8 4 8 6 8 8 9 0 9 2 9 4 9 6 9 8 1 0 0

P a c k e t s ( % )

Del

ay (

ms)

U n i f o r m

• Most of the packets arrive within network delay

• Retransmissions incur additional delay

• Small fraction of the packets are lost

• Burstiness affects retransmissions delay

1 link, 10ms, 5% loss 2 concatenated links, 5% loss each

NOSSDAV 2005 18

Related Work

• VoIP: Skype, Vonage, AT&T, etc.

• MPLS (Multi Protocol Label Switching) Pre-allocating resources (LSPs) in the Internet Packets follow established paths

• FEC (Forward Error Correction) Sending redundant information together with the data

based on loss rate estimates

• Overlay Networks RON – resilient routing using alternate paths XBone – flexible routing using IP in IP tunneling OverQoS – Link protocol that combines ARQ and FEC