Helping TCP Work at Gbps

Cheng Jin

the FAST project at Caltech

http://netlab.caltech.edu/FAST

Talk Outline

• TCP Reno does not perform well at Gbps

• TCP protocol has stability problems

• How FAST improves stability of TCP

• Project status

High Energy Physics

• Clear and present need for high bandwidth– Global cooperation

• 2000 physicists from 150 institutions in the world

• 300 - 400 physicists in US from > 30 universities and labs

– Large file transfers ~ 1 TB• At 622 Mbps ~ 4 hrs

• At 2.5 Gbps ~ 1 hr

• At 10 Gbps ~ 15 minutes

Highspeed TCP Performance

ns-2: 100 sources, 100 ms round trip propagation delay

155Mbps

622Mbps

2.5 Gbps

5 Gbps

10Gbps

J. Wang (Caltech)

TCP Window Evolution

ns-2: capacity = 10 Gbps

FAST TCP/RED

J. Wang (Caltech)

Current TCP Protocol

• Stability problems:– Slow-timescale oscillation as delay or capacity

increases– Independent of packet-level AIMD dynamics– Independent of network noise

0 2000 4000 6000 8000 100000

100

200

300

400

500

600

700

800

Instantaneous Queue

Stable: Small (20 ms) Delay

0 2000 4000 6000 8000 100000

10

20

30

40

50

60

70

individual window

average window

Window

50 identical FTP sources, single link 9 pkts/ms, RED marking

0 2000 4000 6000 8000 100000

10

20

30

40

50

60

70

individual window

average window

Unstable: Large (200ms) Delay

0 2000 4000 6000 8000 100000

100

200

300

400

500

600

700

800

Instantaneous Queue

50 identical FTP sources, single link 9 pkts/ms, RED marking

Congestion Window

Other Effects on Queue Length

same RTT 20ms

same RTT 200ms

30% noise

30% noise

mean RTT 16ms

mean RTT 208ms

Stability Regions

8 9 10 11 12 13 14 1550

55

60

65

70

75

80

85

90

95

100

capacity (pkts/ms)

del

ay (

ms)

N = 40

N = 30

N = 20

N = 50

N = 60 Unstable for Large delay Large

capacity Small load

Loss vs. Delay

• TCP Reno uses loss as congestion measure

• Loss becomes noisy as capacity increases

• TCP’s increase and decrease of cwnd not adaptive to system response

• FAST can use either queueing delay or loss

• Queueing delay has the right scaling with respect to capacity

• FAST adapts to capacity or end-to-end delay

FAST: Fast AQM Scalable TCP

• If loss is used– Both sender TCP and router AQM need to be

changed

• If queueing delay is used– Only sender TCP needs to be changed – Injecting x ms of queueing delay into the

network and change the send rate based on the observed queueing delay and its rate of change

Project Status

• Designed improved TCP/AQM protocols with the right scaling

• Compare FAST to existing approaches for highspeed TCP

• Linux kernel implementation of FAST

• Router implementation of AQM

• Experiments on “real” networks

Linux Kernel Implementation

• RedHat 7.3 with 2.4.18 kernel

• Modifications to the TCP layer

• Monitoring tool as loadable kernel module

• Incorporate features such as tunable socket buffer size and MTU

Floyd’s Highspeed TCP

• Slow increase of congestion window requires extremely small loss probability

• Tuning TCP’s AIMD window adjustment– More rapid increase of cwnd– Less aggressive reduction of cwnd

• More simulations/experiments are needed

Equation-Based Approach

• Remove packet-level AIMD effect

• Estimate congestion signal (usually packet loss) and compute transmission rate

• Needs the right scaling with respect to delay and capacity

Effect of Protocol Instability

• Large jitters, bad for real-time traffic

• Creating bursty queues, causing packet losses

• Lower network utilization at high speed