reducing the buffer size in backbone routers yashar ganjali high performance networking group...
Post on 21-Dec-2015
216 views
TRANSCRIPT
![Page 1: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/1.jpg)
Reducing the Buffer Size in Backbone Routers
Yashar Ganjali
High Performance Networking Group
Stanford University
February 23, 2005
http://www.stanford.edu/~yganjali
![Page 2: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/2.jpg)
23 February 2005 High Performance Networking Group 2
Motivation
Problem– Internet traffic is doubled every year– Disparity between traffic and router growth
(space, power, cost)
Possible solution– All-optical networking
Consequences– Large capacity large traffic– Very small buffers
![Page 3: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/3.jpg)
23 February 2005 High Performance Networking Group 3
Outline of the Talk
Buffer sizes in today’s Internet From huge to small (Guido’s results)
– 2-3 orders of magnitude reduction
From small to tiny– Constant buffer sizes?
![Page 4: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/4.jpg)
23 February 2005 High Performance Networking Group 4
Backbone Router Buffers
Universally applied rule-of-thumb– A router needs a buffer size: B=2TxC
2T is the two-way propagation delay C is the capacity of the bottleneck link
Known to the inventors of TCP Mandated in backbone routers Appears in RFPs and IETF architectural guidelines
CRouterSource Destination
2T
![Page 5: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/5.jpg)
23 February 2005 High Performance Networking Group 5
Review: TCP Congestion Control
Only W packets may be outstanding
Rule for adjusting W– If an ACK is received: W ← W+1/W– If a packet is lost: W ← W/2
Source Dest
maxW
2maxW
t
Window size
![Page 6: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/6.jpg)
23 February 2005 High Performance Networking Group 6
Multiplexing Effect in the Core
ProbabilityDistribution
B
0
Buffer Size
W
![Page 7: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/7.jpg)
23 February 2005 High Performance Networking Group 7
Backbone router buffers
It turns out that– The rule of thumb is wrong for a core routers
today
– Required buffer is instead of CT 2n
CT 2
![Page 8: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/8.jpg)
23 February 2005 High Performance Networking Group 8
Required Buffer Size
2T C
n
Simulation
![Page 9: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/9.jpg)
23 February 2005 High Performance Networking Group 9
Impact on Router Design
10Gb/s linecard with 200,000 x 56kb/s flows– Rule-of-thumb: Buffer = 2.5Gbits
Requires external, slow DRAM
– Becomes: Buffer = 6Mbits Can use on-chip, fast SRAM Completion time halved for short-flows
40Gb/s linecard with 40,000 x 1Mb/s flows– Rule-of-thumb: Buffer = 10Gbits– Becomes: Buffer = 50Mbits
![Page 10: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/10.jpg)
23 February 2005 High Performance Networking Group 10
How small can buffers be?
Imagine you want to build an all-optical router for a backbone network…
…and you can build a few dozen packets in delay lines.
Conventional wisdom: It’s a routing problem (hence deflection routing, burst-switching, etc.)
Our belief: First, think about congestion control.
![Page 11: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/11.jpg)
23 February 2005 High Performance Networking Group 11
TCP with ALMOST No Buffers
Utilization of bottleneck link = 75%
![Page 12: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/12.jpg)
23 February 2005 High Performance Networking Group 12
Problem Solved?
75% utilization with only one unit of buffering More flows Less buffer Therefore, one unit of buffering is enough
![Page 13: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/13.jpg)
23 February 2005 High Performance Networking Group 13
TCP Throughput withSmall Buffers
TCP Throughput vs. Number of Flows
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 200 400 600 800 1000Number of Flows
Th
rou
gh
pu
t
![Page 14: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/14.jpg)
23 February 2005 High Performance Networking Group 14
TCP Reno Performance
Buffer Size = 10; Load = 80%
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 1000 2000 3000 4000 5000 6000
Bottleneck Capacity Mbps
Th
rou
gh
pu
t
![Page 15: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/15.jpg)
23 February 2005 High Performance Networking Group 15
Two Concurrent TCP Flows
![Page 16: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/16.jpg)
23 February 2005 High Performance Networking Group 16
Simplified Model
Flow 1 sends W packets during each RTT Bottleneck capacity = C packets per RTT Example: C = 15, W = 5
Flow 2 sends two consecutive packets during each RTT
Drop probability is increased with W
1 RTT
![Page 17: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/17.jpg)
23 February 2005 High Performance Networking Group 17
Simplified Model (Cont’d)
W(t+1) = p(t)x[W(t)/2] + [1-p(t)]x[W(t)+1] But, p grows linearly with W E[W] = O(C½) Link utilization = W/C As C increases, link utilization goes to zero.
Snow model!!!
![Page 18: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/18.jpg)
23 February 2005 High Performance Networking Group 18
Q&A
Q. What happens if flow 2 never sends any consecutive packets?
A. No packet drops unless utilization = 100%.
Q. How much space we need between the two packets?
A. At least the size of a packet.
Q. What if we have more than two flows?
![Page 19: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/19.jpg)
23 February 2005 High Performance Networking Group 19
Per-flow Queueing
Let us assume we have a queue for each flow; and
Server those queues in a round robin manner.
Does this solve the problem?
![Page 20: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/20.jpg)
23 February 2005 High Performance Networking Group 20
Per-flow Buffering
![Page 21: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/21.jpg)
23 February 2005 High Performance Networking Group 21
Per-Flow Buffering
Flow 3, does not have a packet at time t; Flows 1 and 2 do.
At time t+RTT we will see a drop.
Temporarily Idle
Time t Time t + RTT
![Page 22: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/22.jpg)
23 February 2005 High Performance Networking Group 22
Ideal Solution
If packets are spaced out perfectly; and The starting times of flows are chosen randomly; We only need a small buffer for contention
resolution.
![Page 23: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/23.jpg)
23 February 2005 High Performance Networking Group 23
Randomization
Mimic an M/M/1 queue
Under low load, queue size is small with high Probability
Loss can be bounded
kkXP
EX
1
11
M/M/1
X
b
P(Q > b)Buffer B
Packet Loss
![Page 24: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/24.jpg)
23 February 2005 High Performance Networking Group 24
TCP Pacing
Current TCP: – Send packets when ACK received.
Paced TCP: – Send one packet every W/RTT time units.– Update W, and RTT similar to TCP
![Page 25: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/25.jpg)
23 February 2005 High Performance Networking Group 25
CWND: Reno vs. Paced TCP
![Page 26: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/26.jpg)
23 February 2005 High Performance Networking Group 26
TCP Reno: Throughput vs. Buffer Size
![Page 27: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/27.jpg)
23 February 2005 High Performance Networking Group 27
Paced TCP: Throughput vs. Buffer Size
![Page 28: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/28.jpg)
23 February 2005 High Performance Networking Group 28
Early Results
Congested core router with 10 packet buffers.Average offered load = 80%RTT = 100ms; each flow limited to 2.5Mb/s
router
source
server
source
10Gb/s
>10Gb/s
>10Gb/s
![Page 29: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/29.jpg)
23 February 2005 High Performance Networking Group 29
What We Know
ArbitraryInjectionProcess
If Poisson Processwith load < 1
CompleteCentralized
Control
Any rate > 0need unbounded
buffers
Theory Experiment
Need buffersize of approx:
O(logD + logW)i.e. 20-30 pkts
D=#of hopsW=window size
[Goel 2004]
TCP Pacing:Results as goodor better than forPoisson
Constant fractionthroughput withconstant buffers
[Leighton]
![Page 30: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/30.jpg)
23 February 2005 High Performance Networking Group 30
Limited Congestion WindowR
EN
OP
AC
ING
Limited Window Unlimited Window
![Page 31: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/31.jpg)
23 February 2005 High Performance Networking Group 31
Slow Access Links
router
source
server
source
10Gb/s
5Mb/s 5Mb/s
Congested core router with 10 packet buffers.RTT = 100ms; each flow limited to 2.5Mb/s
![Page 32: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/32.jpg)
23 February 2005 High Performance Networking Group 32
Conclusion
We can reduce 1,000,000 packet buffers to 10,000 today.
We can “probably” reduce to 10-20 packet buffers:– With many small flows, no change needed.– With some large flows, need pacing in the access
routers or at the edge devices.
Need more work!
![Page 33: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/33.jpg)
23 February 2005 High Performance Networking Group 33
Extra Slides
![Page 34: Reducing the Buffer Size in Backbone Routers Yashar Ganjali High Performance Networking Group Stanford University February 23, 2005 yganjali@stanford.edu](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d5a5503460f94a3aeb0/html5/thumbnails/34.jpg)
23 February 2005 High Performance Networking Group 34
Pathological Example
Flow 1: S1 D; Load = 50% Flow 2: S2 D; Load = 50%
If S1 sends a packet at time t, S2 cannot send any packets at time t, and t+1.
To achieve 100% throughput we need at least one unit of buffering.
1 2
S1
S2
D3 4