High Performance All-Optical Networks with

Small Buffers

Yashar GanjaliHigh Performance Networking GroupStanford University

yganjali@stanford.eduhttp://www.stanford.edu/~yganjali

Joint work with:Prof. Ashish GoelProf. Nick McKeownProf. Tim Roughgarden

October 4, 2004 - UCSB

October 4, 2004 - UCSB 2

Outline Issues specific to All-optical networks Why do we need buffers?

Contention resolution Congestion control

How large are buffers today? How small can buffers be?

Congestion Contention

• Scheduling• Load balancing in time and space

Summary and conclusions

October 4, 2004 - UCSB 3

All-Optical Network Design Issues

Wavelength conversion Without full wavelength conversion

• Routing problem is very hard With full wavelength conversion

• Routing is the same as electronic networks

Buffering Existing systems

• Huge buffers are assumed to be available Need to reduce the buffer size

October 4, 2004 - UCSB 4

Leverages

Full wavelength conversion We don’t need to worry about it

Huge amount of capacity We can tradeoff capacity to address

small buffers problem Deflection routing

Not necessary at this stage

October 4, 2004 - UCSB 5

Issues specific to All-optical networks Why do we need buffers?

Contention resolution Congestion control

How large are buffers today? How small can buffers be?

Congestion Contention

• Scheduling• Load balancing in time and space

Summary and conclusions

Outline

October 4, 2004 - UCSB 6

Why do we need buffers?

Internet traffic is bursty in nature Variations in

• Starting time• Flow length• Rate

Short-term fluctuations Contention

Long-term fluctuations Congestion

October 4, 2004 - UCSB 7

Contention Contention is caused by

Synchronization of flows Stochastic collisions

October 4, 2004 - UCSB 8

Congestion Control

Rule for adjusting W If an ACK is received:W ← W+1/W If a packet is lost: W ← W/2

Source Dest

maxW

2maxW

t

Window size

Only W packets may be outstanding

October 4, 2004 - UCSB 9

Rule for adjusting W If an ACK is received:W ← W+1/W If a packet is lost: W ← W/2

Congestion Control (Cont’d)

Only W packets may be outstanding

October 4, 2004 - UCSB 10

Issues specific to All-optical networks Why do we need buffers?

Contention resolution Congestion control

How large are buffers today? How small can buffers be?

Congestion Contention

• Scheduling• Load balancing in time and space

Summary and conclusions

Outline

October 4, 2004 - UCSB 11

Universally applied rule-of-thumb: A router needs a buffer size:

• 2T is the two-way propagation delay• C is capacity of bottleneck link

Context Mandated in backbone and edge routers. Appears in RFPs and IETF architectural guidelines. Usually referenced to Villamizar and Song: “High

Performance TCP in ANSNET”, CCR, 1994. Already known by inventors of TCP [Van Jacobson, 1988]

Backbone Router Buffers

CTB 2

CRouterSource Destination

2T

October 4, 2004 - UCSB 12

Single TCP FlowRouter with large enough buffers for full link

utilization

October 4, 2004 - UCSB 13

Single TCP FlowRouter with large enough buffers for full link

utilization

Observation If buffer doesn’t go empty when window size

halves, then we have 100% throughput.

maxW

2maxW

t

Window size RTT

It follows that CTB 2

October 4, 2004 - UCSB 14

Buffer = Rule of Thumb

October 4, 2004 - UCSB 15

Under-buffered Link

October 4, 2004 - UCSB 16

Issues specific to All-optical networks Why do we need buffers?

Contention resolution Congestion control

How large are buffers today? How small can buffers be?

Congestion Contention

• Scheduling• Load balancing in time and space

Summary and conclusions

Outline

October 4, 2004 - UCSB 17

Rule-of-thumb

Rule-of-thumb makes sense for one flow Typical backbone link has > 20,000 flows Does the rule-of-thumb still hold?

Answer: If flows are perfectly synchronized, then Yes. If flows are desynchronized then No.

October 4, 2004 - UCSB 18

If flows are synchronized

Aggregate window has same dynamics Therefore buffer occupancy has same dynamics Rule-of-thumb still holds.

2maxW

t

max

2

W

maxW

maxW

October 4, 2004 - UCSB 19

If flows are not synchronized

ProbabilityDistribution

B

0

Buffer Size

W

October 4, 2004 - UCSB 20

It turns out that The rule of thumb is wrong for core

routers today

Required buffer is instead of

[Appenzeller, Keslassy, McKeown

2004]

Backbone Router Buffers

CT 2n

CT 2

October 4, 2004 - UCSB 21

2T C

n

Simulation

Required Buffer Size

October 4, 2004 - UCSB 22

TCP with ALMOST no buffers

Utilization of bottleneck link = 75%

October 4, 2004 - UCSB 23

TCP with almost no buffers

With almost no buffering and just a single flow we loose about 25% of the capacity. Capacity is not a bottleneck anymore More flows = less capacity loss Huge number of flows in the core

October 4, 2004 - UCSB 24

Issues specific to All-optical networks Why do we need buffers?

Contention resolution Congestion control

How large are buffers today? How small can buffers be?

Congestion Contention

• Scheduling• Load balancing in time and space

Summary and conclusions

Outline

October 4, 2004 - UCSB 25

Contention resolution

Two extreme approaches No scheduling: Deal with contention by

having large buffers. Tight Scheduling: Precisely schedule exact

packet injection times, so that contention is minimized.

No scheduling

Tight scheduling

October 4, 2004 - UCSB 26

Constraints No buffer: If S2 sends a packet at time

t, S1 cannot send a packet at time t+1

Buffer size B: S1 and S2 cannot send more than T+B packets in any interval of length T

Scheduling

S1

S2

D

October 4, 2004 - UCSB 27

Scheduling Problem

The optimal solution is hard to find in general In no buffers case:

•Exact solution: NP-Complete (Job-shop scheduling)

•50% Throughput: Easy to solve With buffers: Open problem

Extreme synchronization needed Distributed algorithm needed

October 4, 2004 - UCSB 28

Randomization

Randomization: Randomly delay the injection time of the packets Alleviates short-term contention Simple to implement Guaranteed performance

No scheduling

Tight scheduling

???Randomization

October 4, 2004 - UCSB 29

Randomization (cont’d)

Time

Input #1

Input #2

Before randomization

Time

Input #1

Input #2

After randomization

October 4, 2004 - UCSB 30

Randomization (cont’d)

Shape the traffic at injection time (make Poisson)

Reshape at each router

When the load is low we can bound the loss rate

Buffer size

Loss rate

kkXP

EX

1

11

M/M/1

X

October 4, 2004 - UCSB 31

Preliminary Results

Theorem. We can achieve constant factor throughput (roughly 70-80%) with very

small amount of loss using buffers of size O(log L), where L is the length of the

longest path in the network.

Assumption No reactive flow control

Currently maximum window size is 12 and 64 for Linux and Windows XP

October 4, 2004 - UCSB 32

Issues specific to All-optical networks Why do we need buffers?

Contention resolution Congestion control

How large are buffers today? How small can buffers be?

Congestion Contention

• Scheduling• Load balancing in time and space

Summary and conclusions

Outline

October 4, 2004 - UCSB 33

Preliminary Results

Conjecture. Routers in the core of the current Internet only need buffers of

size O(log L) instead of the 2TxC.

Need to study The interaction between randomization

and congestion control Impact of co-existing flows Emerging applications (non-TCP or

modified TCP) which will allow much larger windows per flow

October 4, 2004 - UCSB 34

Summary and conclusions

We need buffers to address Contention Congestion

Aggregation takes care of congestion Use randomization to reduce

contention At the price of loosing some capacity,

a network with small buffers can have high performance.Many thanks to Guido Appenzeller at Stanford for

providing the flash animations.