seeding cloud-based services: distributed rate limiting (drl)
DESCRIPTION
Seeding Cloud-based services: Distributed Rate Limiting (DRL). Kevin Webb , Barath Raghavan , Kashi Vishwanath , Sriram Ramabhadran , Kenneth Yocum , and Alex C. Snoeren. Seeding the Cloud. T echnologies to deliver on the promise cloud computing - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/1.jpg)
SEEDING CLOUD-BASED SERVICES:DISTRIBUTED RATE LIMITING (DRL)
Kevin Webb, Barath Raghavan, Kashi Vishwanath, Sriram Ramabhadran, Kenneth Yocum, and Alex C. Snoeren
![Page 2: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/2.jpg)
Seeding the CloudTechnologies to deliver on the promise cloud
computing Previously: Process data in the cloud (Mortar)
Produced/stored across providers Find Ken Yocum or Dennis Logothetis for more info
Today: Control resource usage: “cloud control” with DRL Use resources at multiple sites (e.g., CDN) Complicates resource accounting and control Provide cost control
![Page 3: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/3.jpg)
DRL Overview Example: Cost control in a Content Distribution Network Abstraction: Enforce global rate limit across multiple sites
Simple example: 10 flows, each limited as if there was a single, central limiter
Src DstLimiter
Src DstLimiter
Src DstLimiter
DRL
10 flows
2 flows
8 flows
20 KB/s
100 KB/s
80 KB/s
![Page 4: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/4.jpg)
Goals & Challenges Up to now
Develop architecture and protocols for distributed rate limiting (SIGCOMM 07)
Particular approach (FPS) is practical in the wide area
Current goals: Move DRL out of the lab and impact real services Validate SIGCOMM results in real-world conditions Provide Internet testbed with ability to manage bandwidth in a distributed
fashion Improve usability of PlanetLab
Challenges Run-time overheads: CPU, memory, communication Environment: link/node failures, software quirks
![Page 5: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/5.jpg)
PlanetLab World-wide test
bed Networking and
systems research Resources donated
by Universities, Labs, etc.
Experiments divided into VMs called “slices” (Vservers)
PostgreSQLPLC APIWeb server
Linux 2.6
Internet
Controller
Vservers
Slice1
Linux 2.6
Slice2
SliceN
Vservers
Slice1
Linux 2.6
Slice2
SliceN
Nodes
![Page 6: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/6.jpg)
PlanetLab Use Cases PlanetLab needs DRL!
Donated bandwidth Ease of administration
Machine room Limit local-area nodes to a single rate
Per slice Limit experiments in the wide area
Per organization Limit all slices belonging to an organization
![Page 7: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/7.jpg)
PlanetLab Use Cases Machine room
Limit local-area nodes with a single rate
1 MBps1 MBps
1 MBps
1 MBps
1 MBps
DRL
DRL
DRL
DRL
DRL
5 MBps
![Page 8: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/8.jpg)
DRL Design Each limiter - main event
loop Estimate: Observe and record
outgoing demand Allocate: Determine rate
share of each node Enforce: Drops packets
Two allocation approaches GRD: Global random drop
(packet granularity) FPS: Flow proportional share
Flow count as proxy for demand
Input Traffic
Output traffic
Estimate
Allocate
EnforceRegularInterval
OtherLimiters
FPS
![Page 9: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/9.jpg)
Implementation Architecture
Abstractions Limiter
Communication Manages identities
Identity Parameters (limit, interval, etc.) Machines and Subsets
Built upon standard Linux tools… Userspace packet logging (Ulogd) Hierarchical Token Bucket Mesh & gossip update protocols
Integrated with PlanetLab software
Input Data
Output Data
Estimate
FPS
EnforceRegularInterval
Ulogd
HTB
![Page 10: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/10.jpg)
Estimation using ulogd Userspace logging daemon
Already used by PlanetLab for efficient abuse tracking
Packets tagged with slice ID by IPTables Receives outgoing packet headers via netlink
socket
DRL implemented as ulogd plug-in Gives us efficient flow accounting for estimation Executes the Estimate, Allocate, Enforce loop Communicates with other limiters
![Page 11: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/11.jpg)
Enforcement with Hierarchical Token Bucket
Linux Advanced Routing & Traffic Control
Hierarchy of rate limits
Enforces DRL’s rate limit
Packets attributed to leaves (slices)
Packets move up, borrowing from parents
B C D Y Z
A X
Root
Packet (1500b)
1000b
100b
600b
Packet (1500)
0b
0b
200b
![Page 12: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/12.jpg)
Enforcement with Hierarchical Token Bucket
Uses same tree structure as PlanetLab
Efficient control ofsub-trees Updated every loop Root limits whole
node
Replenish each level B C D Y Z
A X
Root
![Page 13: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/13.jpg)
Citadel Site The Citadel (2 nodes)
Wanted 1 Mbps traffic limit Added (horrible) traffic shaper Poor responsiveness (2 – 15 seconds)
Running right now! Cycles on and off every four minutes
Observe DRL’s impact without ground truth
Shaper
DRL
![Page 14: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/14.jpg)
Citadel Results – Outgoing Traffic
Data logged from running nodes
Takeaways: Without DRL, way over limit One node sending more than other
Time
Outg
oing
Traffi
c
1Mbit/s
On On On OnOffOffOff Off
![Page 15: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/15.jpg)
Citadel Results – Flow Counts
Time
# of
Flo
ws
FPS uses flow count as proxy for demand
![Page 16: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/16.jpg)
Citadel Results – Limits and Weights
Time
Rate
Lim
itFP
S W
eigh
t
![Page 17: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/17.jpg)
Lessons Learned Flow counting is not always the best proxy for demand
FPS state transitions were irregular Added checks and dampening/hysteresis in problem cases
Can estimate after enforce Ulogd only shows packets after HTB FPS is forgiving to software limitations
HTB is difficult HYSTERESIS variable TCP Segmentation offloading
![Page 18: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/18.jpg)
Ongoing work Other use cases Larger-scale tests Complete PlanetLab administrative
interface
Standalone version
Continue DRL rollout on PlanetLab UCSD’s PlanetLab nodes soon
![Page 19: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/19.jpg)
Questions? Code is available from PlanetLab svn
http://svn.planet-lab.org/svn/DistributedRateLimiting/
![Page 20: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/20.jpg)
![Page 21: Seeding Cloud-based services: Distributed Rate Limiting (DRL)](https://reader035.vdocuments.us/reader035/viewer/2022062520/56816211550346895dd23d79/html5/thumbnails/21.jpg)
Citadel Results