Routers with Small Buffers

Yashar GanjaliHigh Performance Networking GroupStanford University

yganjali@stanford.eduhttp://yuba.stanford.edu/~yganjali/

Joint work with:

Guido Appenzeller, Mihaela Enachescu, Ashish Goel, Tim Roughgarden, Nick McKeown

Special thanks to: Level 3 Communications

NANOG, October 25, 2005

October 2005 Routers with Small Buffers 2

The Story

)(logO2

2 )2()1( Wn

CTCT

)(logO

22 )2()1( W

n

CTCT

(1) Assume: Large number of desynchronized flows; 100% utilization(2) Assume: Large number of flows; <100% utilization

1,000,000 10,000 20# packetsat 10Gb/s

SawtoothPeak-to-trough

Smoothing of many sawtooths

Non-bursty arrivals

Intuition& Proofs

SimulatedSingle TCP Flow

Simulations, Experiments

Simulated ManyTCP Flows

Evidence

October 2005 Routers with Small Buffers 3

Universally applied rule-of-thumb: A router needs a buffer size:

2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link

Context Mandated in backbone and edge routers. Appears in RFPs and IETF architectural guidelines. Usually referenced to Villamizar and Song: “High Performance

TCP in ANSNET”, CCR, 1994. Already known by inventors of TCP [Van Jacobson, 1988] Has major consequences for router design

CTB 2

CRouterSource Destination

2T

Backbone Router Buffers

October 2005 Routers with Small Buffers 4

Rule for adjusting W If an ACK is received: W ← W+1/W If a packet is lost: W ← W/2

Single TCP Flow

Only W packets may be outstanding

October 2005 Routers with Small Buffers 5

Rule for adjusting W If an ACK is received: W ← W+1/W If a packet is lost: W ← W/2

Single TCP Flow

Only W packets may be outstanding

Source Dest

maxW

2maxW

t

Window size

CT 2

CT 2

October 2005 Routers with Small Buffers 6

Time evolution of a single TCP flow through a router. Buffer is < 2T*C

Time Evolution of a Single TCP FlowTime evolution of a single TCP flow through a router. Buffer is 2T*C

October 2005 Routers with Small Buffers 7

Synchronized Flows

Aggregate window has same dynamics Therefore buffer occupancy has same dynamics Rule-of-thumb still holds.

2maxW

t

max

2

W

maxW

maxW

October 2005 Routers with Small Buffers 8

ProbabilityDistribution

B

0

Buffer Size

W

Many TCP Flows

October 2005 Routers with Small Buffers 9

2T C

n

Simulation

Required Buffer Size

October 2005 Routers with Small Buffers 10

Real Network Experiments

Stanford University dorm traffic Network Lab (Cisco routers) at University of

Wisconsin Internet2 Operational Internet backbone

October 2005 Routers with Small Buffers 11

Internet Backbone Experiment Buffer sizes 190ms,

10ms, 5ms, 2.5 and 1ms Load balancing High link utilization Long duration (about two

weeks)

Drops, utilization data collected every 30 seconds

Test flows

October 2005 Routers with Small Buffers 12

Packet Drops vs. Link Load

Buffer size = 190ms, 10ms, 5ms

MAX

October 2005 Routers with Small Buffers 13

Packet Drops vs. Link LoadBuffer size = 1ms

October 2005 Routers with Small Buffers 14

Relative Link UtilizationUtilization of the link with 1ms buffer /Utilization of the link with 190ms buffer

October 2005 Routers with Small Buffers 15

Relative Utilization (Cont’d)

October 2005 Routers with Small Buffers 16

Theory vs. Practice

M/D/1

Theory (benign conditions) Practice

Typical OC192 router linecard buffers over 1,000,000 packets

Can we make traffic look “Poisson-enough” when it arrives to the routers…?

Can we make traffic look “Poisson-enough” when it arrives to the routers…?

Poisson

BD

B loss

%1loss pkts20 80%, .. Bei

Loss independent of link rate, RTT, number of flows, etc.

5 orders of magnitudedifference!

5 orders of magnitudedifference!

October 2005 Routers with Small Buffers 17

Assume: Buffer size > Distance between consecutive packets of a

single flow S > Limited injection rate

Flows are not synchronized; and Start times picked randomly and independently

We can prove that the packet drop probability is very low.

Paced Injections

Similar results from Cambridge/UCL, UMass and StanfordSee papers in: ACM Computer Communications Review, July 2005

Similar results from Cambridge/UCL, UMass and StanfordSee papers in: ACM Computer Communications Review, July 2005

October 2005 Routers with Small Buffers 18

O(log W) BuffersAssumptions:

Internet core is over-provisioned Example: Load < 80%

There is spacing between packets of the same flow: Natural: Slow access links Artificial: Paced TCP

Result:Traffic is very smooth, and loss rate is very low,

independent of RTT, and number of flows.

With a buffer size of just 10-20 packets we can gain high throughput.

October 2005 Routers with Small Buffers 19

Leaky Bucket – Paced vs. RenoBucket drains with a constant rate. Load is 90% for both cases.

October 2005 Routers with Small Buffers 20

TCP RenoTCP Reno sends packets in a burst High drop rate

October 2005 Routers with Small Buffers 21

Paced TCPSpacing packets Much lower drop rate

October 2005 Routers with Small Buffers 22

O(log W) BuffersRegular TCPRegular TCP

TCP WithPacing

TCP WithPacing

October 2005 Routers with Small Buffers 23

O(log W) Buffers

October 2005 Routers with Small Buffers 24

Ideal Experiment

Highly loaded link, with real/realistic traffic Precisely controlled router buffers Packet traces with precise timestamps Work in progress: Sprint, Verizion, Telcordia, Lucent,

…

Packet Trace Monitor

Packet Trace Monitor

October 2005 Routers with Small Buffers 25

Conclusion and Future Work Theory:

Reducing buffer sizes by a factor of sqrt(N) does not affect the network performance.

Reducing the buffer sizes to O(logW) does not affect the network performance if: The network is over provisioned; and We use Paced TCP; or Have slow access links

Experimental Validation: Thousands of ns2 simulations Stanford dorm, University of Wisconsin Testbed, Internet2, Level

3 Communications, … Ongoing work and need your help

October 2005 Routers with Small Buffers 26

Thanks!Thanks!

More Info?More Info?

yganjali@stanford.eduyganjali@stanford.eduhttp://www.stanford.edu/~yganjalihttp://www.stanford.edu/~yganjali

http://yuba.stanford.edu/~yganjali/research/http://yuba.stanford.edu/~yganjali/research/bsizing/bsizing/

October 2005 Routers with Small Buffers 27

O(log W) BuffersWith a large ratio between core and access link bandwidth

Bottleneck Bandwidth = 1Gb/s