anycast for any service
DESCRIPTION
Anycast for Any Service. Michael J. Freedman Karthik Lakshminarayanan David Mazières http://oasis.coralcdn.org/. What’s the replica-selection problem?. mycdn. Client needs to choose a “good” replica server Performance and cost dependent on replica selection. ?. What do we currently do?. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/1.jpg)
Anycast for Any Service
Michael J. Freedman
Karthik LakshminarayananDavid Mazières
http://oasis.coralcdn.org/
![Page 2: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/2.jpg)
mycdn
?
What’s the replica-selection problem?
Client needs to choose a “good” replica server Performance and cost dependent on replica selection
![Page 3: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/3.jpg)
What do we currently do?
![Page 4: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/4.jpg)
How bad can it get?
![Page 5: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/5.jpg)
Anycast is the solution
Anycast = automated “good” replica selection OASIS is a flexible anycast system for multiple services
?
mycdn
![Page 6: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/6.jpg)
The need for anycast Internet systems rely on replicated content and services
Distributed mirrors: Web servers, FTP servers, …
Content Distribution Networks: Akamai, CoralCDN, …
Internet Naming Systems: DNS, SFR, DOA, …
Distributed File Systems: CFS, Shark, ….
Routing Overlays: RON, Detour, i3, …
Distributed Hash Storage Systems: OpenDHT, …
All could benefit from anycast service
![Page 7: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/7.jpg)
How should one implement anycast?
![Page 8: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/8.jpg)
Strawman: probe & find nearestmycdn
ID
B
C
A
E ICMP
![Page 9: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/9.jpg)
Strawman: probe & find nearestmycdn
ID
B
C
A
E ICMP
, ,E I D
Result highly accurate Lots of probing Slow to compute
![Page 10: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/10.jpg)
mycdn
ID
B
C
A
E ICMP
, ,E I D
Result highly accurate Lots of probing Slow to compute
Strawman: probe & find nearest
![Page 11: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/11.jpg)
mycdn
ID
B
C
A
E ICMP
, ,E I D
Result highly accurate Lots of probing Slow to compute
Avoid probing on-demand
![Page 12: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/12.jpg)
mycdn
ID
B
C
A
E ICMP
, ,E I D
Avoid probing on-demand
Result highly accurate Lots of probing Slow to compute
18.0.0.0/8
[IMC05] shows IP prefixes often preserve locality( 99% of /24s by stub AS at the same location )
This is the problem Akamai must solve
![Page 13: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/13.jpg)
What about yourcdn?
mycdn E I D yourcdn M N O
18.0.0.0/8
Result highly accurate Lots of probing Slow to compute
![Page 14: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/14.jpg)
Idea: Use geographic coordinates
mycdn yourcdn
18.0.0.0/8
(42N,71W)
Result highly accurate Lots of probing Slow to compute
Stable across services, time, and failures
Assume all replicas know geo-coords
(42N,71W)
Amortize costs
![Page 15: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/15.jpg)
OASIS provides…
Result highly accurate Stable across time, services, and failures Amortize costs
Fast response time Supports flexible anycast policies
Balances tension between: Performance: finding nearest replica Cost: minimizing 95% bandwidth usage
![Page 16: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/16.jpg)
Outline
Architecture and design decisions
Detailed design
Evaluation
Deployment and integration lessonsOASIS deployed since November 2005Currently in use by 10 services
![Page 17: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/17.jpg)
Two-tier architecturemycdn
OASIS core
Large set of replicas that assist in measurementReliable core of hosts that implement anycast
= DNS OASIS
nodeRPC
mycdnreplicaproxy =
![Page 18: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/18.jpg)
Using OASIS via DNS
1. Client issues DNS request for mycdn.nyuld.net
2. OASIS redirects client to nearby application replica
mycdn
OASIS core
Client Resolver
12
![Page 19: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/19.jpg)
1 23OASIS core
mycdn
1. Client issues HTTP request
2. Web cgi-bin issues RPC to OASIS core
3. Client redirected to nearby application replica
Client
Using OASIS via HTTP
![Page 20: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/20.jpg)
servicebucketing
proximity replicas
IP prefix policy
coords
response
requestIP addr name
Using OASIS via HTTPHow does core answer anycast?
![Page 21: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/21.jpg)
How does core answer anycast?
servicebucketing
proximity replicas
IP prefix
coords
requestIP addr name
policy
response
mycdn
18.26.4.9171.66.3.181
216.165.109.81
mycdn.nyuld.net18.71.0.3
18.0.0.0/8
18.26.4.9
![Page 22: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/22.jpg)
How to map IP prefix to coords?
proximityIP prefix ( Lat, Lng, RTT distance )
location accuracy
![Page 23: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/23.jpg)
Two-pronged approach Find closest replica proxy
How to map IP prefix to coords?
:18.0.0.0/8
(42N,71W)(42N,71W)
proximityIP prefix ( Lat, Lng, RTT distance )
location accuracy
![Page 24: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/24.jpg)
Two-pronged approach Find closest replica proxy Use closest replica’s geo-coords + error RTT as location
How to map IP prefix to coords?
:18.0.0.0/8
(42N,71W)
, 6.0 ms
(42N,71W)
proximityIP prefix ( Lat, Lng, RTT distance )
location accuracy
![Page 25: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/25.jpg)
Two-pronged approach Find closest replica proxy with less probing Use closest replica’s geo-coords + error RTT as location
18.168.0.23
Find replica nearest prefix efficiently
18.0.0.0/8
“Probe 18.0.0.0/8”
![Page 26: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/26.jpg)
Two-pronged approach Find closest replica proxy with less probing Use closest replica’s geo-coords + error RTT as location
18.168.0.23
Find replica nearest prefix efficiently
[ Meridian 05 ]
![Page 27: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/27.jpg)
18.168.0.23
Find replica nearest prefix efficiently
[ Meridian 05 ] Two-pronged approach
Find closest replica proxy with less probing Use closest replica’s geo-coords + error RTT as location
:18.0.0.0/8 , 6.0 ms(42N,71W)
![Page 28: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/28.jpg)
Geographic distance vs. RTT
Strong correlation b/w geographical distance and RTT
![Page 29: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/29.jpg)
Geographic distance vs. RTT
Strong correlation b/w geographical distance and RTT RTT accuracy has real-world meaning
Check if new coordinates improve accuracy vs. old coords
![Page 30: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/30.jpg)
Geographic distance vs. RTT
Strong correlation b/w geographical distance and RTT RTT accuracy has real-world meaning
Check if new coordinates improve accuracy vs. old coords
[ Meridian 05 ] :18.0.0.0/8 , 6.0 ms(42N,71W) :18.0.0.0/8 , 3.0 ms(42N,72W) :18.0.0.0/8 , 3.0 ms(42N,72W)
![Page 31: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/31.jpg)
Geographic distance vs. RTT
Strong correlation b/w geographical distance and RTT RTT accuracy has real-world meaning
Check if new coordinates improve accuracy vs. old coords
[ Meridian 05 ] :18.0.0.0/8 , 6.0 ms(42N,71W) :18.0.0.0/8 , 3.0 ms(42N,72W)
:18.0.0.0/8 , 3.0 ms(42N,72W)
![Page 32: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/32.jpg)
Geographic distance vs. RTT
Strong correlation b/w geographical distance and RTT RTT accuracy has real-world meaning
Check if new coordinates improve accuracy vs. old coords
Useful for sanity check for network peculiarities Do multiple results satisfy constraints (e.g., speed of light) ?
![Page 33: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/33.jpg)
Outline
Architecture and design decisions
Detailed design
Evaluation
Deployment and integration lessonsOASIS deployed since November 2005Currently in use by 10 services
![Page 34: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/34.jpg)
mycdn opendht
OASIS core Global membership view Epidemic gossiping
• Scalable failure detection• Spread policies, prefix→coords
Consistent hashing• Divide up responsibility for prefixes
Service replicas Heartbeats to OASIS node Form global Meridian
overlay for probing
OASIS core
![Page 35: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/35.jpg)
How to find “nearby” nodes?
replicas
IP prefix
coords
requestIP addr name
servicebucketing
proximity
policy
response
mycdn
mycdn.nyuld.net
18.26.4.9171.66.3.181
216.165.109.81
18.26.4.9
18.26.4.9
18.0.0.0/8
Local info from gossiping(stale data okay)
![Page 36: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/36.jpg)
How to find “nearby” nodes?
replicas
IP prefix
coords
requestIP addr name
servicebucketing
proximity
policy
response
mycdn
18.26.4.9Local info from gossiping(stale data okay)
mycdn.nyuld.net18.26.4.9
18.0.0.0/8
18.26.4.9171.66.3.181
216.165.109.81
Clients react poorly to stale data
![Page 37: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/37.jpg)
Aggregate replica information
OASIS
mycdn
Define service’s rendezvous node via consistent hashing
Service replicas send keepalives to nearby OASIS nodes
Update rendezvous when replicas join, leave, large load change
OASIS
H(srv)
![Page 38: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/38.jpg)
Aggregate replica information
OASIS
mycdn
Define service’s rendezvous node via consistent hashing
Service replicas send keepalives to nearby OASIS nodes
Update rendezvous when replicas join, leave, large load change
OASIS
Bottleneck?H(srv)
![Page 39: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/39.jpg)
Aggregate replica information
OASIS
mycdn
Aggregate over k nodes for scalability
Rendezvous gossip liveness state for loose consistency
k can be dynamic for better scalability
OASIS
H(srv)
![Page 40: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/40.jpg)
A client’s view: Finding a nameserver
OASISOASIS
Core lookup: Contacts 1 of 13 nameservers for .nyuld.net
OASIS “uses itself” to discover replica for service dns
H(dns)
Client
![Page 41: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/41.jpg)
A client’s view: Finding a nameserver
OASISOASIS
Core lookup: Contacts 1 of 13 nameservers for .nyuld.net
OASIS “uses itself” to discover replica for service dns
Returns nearby nameservers for subsequent requests
H(dns)
Client
![Page 42: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/42.jpg)
OASISOASIS
H(mycdn)
Replica lookup: Client contacts nearby nameserver
OASIS discover replica for service mycdn
Returns nearby replicas for application
A client’s view: Finding a replica
R
H(dns)
Client
![Page 43: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/43.jpg)
Evaluation
Deployed on PlanetLab since November 2005
How much end-to-end benefit from OASIS?
How accurate is OASIS?
Effective for load balancing?
What are OASIS’s bandwidth costs?
![Page 44: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/44.jpg)
E2E download of web page
290% faster than Meridian
500% faster than RR
Cached virtual coordshighly inaccurate
![Page 45: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/45.jpg)
Client RTT to chosen replica
Outperforms Meridian 60% of time
![Page 46: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/46.jpg)
OASIS minimizes bandwidth spikes
CA TX NY Germany
Latency Only 23.3 0.0 0.0 0.0
Load + Latency
95% bandwidth usage per replica (MB)
metricloc
8 clients in CA repeatedly request 1 MB file
Replicas report load as log (95% bandwidth per 1-min slot)
![Page 47: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/47.jpg)
CA TX NY Germany
Latency Only 23.3 0.0 0.0 0.0
Load + Latency
9.0 11.3 9.6 9.2
95% bandwidth usage per replica (MB)
metricloc
8 clients in CA repeatedly request 1 MB file
Replicas report load as log (95% bandwidth per 1-min slot)
OASIS minimizes bandwidth spikes
![Page 48: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/48.jpg)
Bandwidth costs: OASIS v. on-demand
1-2 orders of magnitude
# DNS reqsto CoralCDN
![Page 49: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/49.jpg)
Outline
Architecture and design decisions
Detailed design
Evaluation
Deployment and integration lessonsOASIS deployed since November 2005Currently in use by 10 services
![Page 50: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/50.jpg)
Sanity check for network peculiarities Employ measurement redundancy Easy visualization significantly helped debugging
![Page 51: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/51.jpg)
Probing generates abuse complaints Your service can get blacklisted!
Keyword frequency on PlanetLab support lists9 months, 1820 threads, 4682 msgs
Netops have low tolerance for probing
![Page 52: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/52.jpg)
Netops have low tolerance for probing
Be careful what you probe Probe slowly and rarely No random ports or obvious attack vectors (TCP port 22/23)
Be careful whom you probe Check blacklist for netblock and target IP (after traceroute)
![Page 53: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/53.jpg)
Make it easy to integrate
replicaproxy
dns
nakikaOASIScorenode
listen(7060)
ServiceNamenakikaLocalPort 7060SecretCode 555555
ServiceName nakika ServiceAlias nakika.net SortTypelatencycap MaxAddrs 2 AddrTTLs 120
![Page 54: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/54.jpg)
replicaproxy
dns
nakikaOASIScorenodecode
loadcap
ServiceNamenakikaLocalPort 7060SecretCode 555555
ServiceName nakika ServiceAlias nakika.net SortTypelatencycap MaxAddrs 2 AddrTTLs 120
listen(7060)?
Clients immediate use nakika.nyuld.net
Make it easy to integrate
![Page 55: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/55.jpg)
Current services using OASIS… Chunkcast block anycast (Berkeley)
CoralCDN (NYU)
Na Kika content distribution (NYU)
OASIS RPC, DNS, HTTP interfaces
OCALA overlay convergence (Berkeley) Separate services for client and server IPs gateways
OpenDHT public DHT service (Berkeley)
OverCite distributed library (MIT): Deployed on RON
![Page 56: Anycast for Any Service](https://reader036.vdocuments.us/reader036/viewer/2022070502/56813061550346895d963133/html5/thumbnails/56.jpg)
Summary OASIS is a general, open anycast service
Supports multiple services: more are betterPerforms accurate server selectionRemoves all on-demand probingProvides easy integration
Use OASIS for your distributed system!
http://oasis.coralcdn.org/