![Page 1: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/1.jpg)
Bob-the-Builder vs. Fix-it-Felix
Maintaining Overlays in Dynamic Graphs
Seth Gilbert
![Page 2: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/2.jpg)
What is an overlay?Overlay Networks
Given a collection of (changing) servers:
• Choose a good subset of the edges.
![Page 3: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/3.jpg)
What is an overlay?Overlay Networks
Given a collection of (changing) servers:
• Choose a good subset of the edges.
• Subgraph has low degree.
• Subgraph has low diameter.
![Page 4: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/4.jpg)
What is an overlay?Overlay Networks
Given a collection of (changing) servers:
• Choose a good subset of the edges.
• Subgraph has low degree.
• Subgraph has low diameter.
• Maintain edges as servers come and go.
Can we build it?
Yes we can!
We can fix it!
![Page 5: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/5.jpg)
Bitcoin is broken(So why are you wasting your time here when you could be busy hacking bitcoins?)
![Page 6: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/6.jpg)
What goes wrong?
Bitcoin is a peer-to-peer overlay network• Overlay is used for all communication.• Overlay is assumed to be reliable.
![Page 7: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/7.jpg)
What goes wrong?
Bitcoin is a peer-to-peer overlay network• Overlay is used for all communication.• Overlay is assumed to be reliable.• Nodes arrive and leave all the time.• Malicious/greedy users can partition the
network.
![Page 8: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/8.jpg)
What goes wrong?
Basic overlay maintenance idea:• Connect to a few arbitrary neighbors.• Maintain a fixed number of neighbors.• Accept incoming connection requests.
Details: Eclipse Attacks on Bitcoin’s Peer-to-Peer Network by Heilman, Kendler, Zohar, Goldberg
Bitcoin is a peer-to-peer overlay network• Overlay is used for all communication.• Overlay is assumed to be reliable.• Nodes arrive and leave all the time.• Malicious/greedy users can partition the
network.
![Page 9: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/9.jpg)
Rapid and significant churn• Users constantly arriving and departing.• Network changes continuously.• No stable state.
Malicious / greedy users• Some users may not follow the protocol.• Or all users may be greedy!
Long-lived system
BitcoinWhy is this a hard problem?
![Page 10: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/10.jpg)
I am not going to solve the Bitcoin problem today.
![Page 11: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/11.jpg)
A few existing overlay networks…
ChordKademlia
Pastry Tapestry
Skip+
HSkip+
Patricia Tries
DexXHeal
Forgiving Tree
PGrid
Skipnet
RN Protocol
dHamiltonianCycles
Avatar
Ca-Re-Chord
HyperCubes
HyperRing
Chameleon
Re-Chord
Tiara
Corona
![Page 12: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/12.jpg)
What do we know?Discussion of existing approaches.Is the problem solved?Are the existing solutions practical?
Ongoing workSome ideas that we are thinking about.
Challenge for us to work onTo work on after the workshop is over.
Today’s GoalsA Workshop Talk
Collaborators: Gopal Pandurangan, Peter Robinson, and Amitabh Trahan
![Page 13: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/13.jpg)
Overlay Networks
![Page 14: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/14.jpg)
Overlay NetworksGround Rules
Underlying network:• Collection of nodes.
• Nodes arrive (join). Joining node is connected to someone.
• Nodes leave (fail).
• Precise model TBA.
![Page 15: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/15.jpg)
Overlay NetworksGround Rules
Communication:• Every node has an address.
• Every node has an address book.
• A node can send a message to every node in the address book.
• A node can send an address to another node.
Joe
Sue
JoeSue
< Sue >
![Page 16: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/16.jpg)
Overlay NetworksGoals
Overlay network:• Low degree constant or logarithmic.
• Low diameter logarithmic or polylogarithmic
Note: every existing solution guarantees these properties.
![Page 17: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/17.jpg)
Overlay NetworksGoals
Routable:• There exist short paths…
• … and we can find short paths.
Note: random graphs may not be good!
![Page 18: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/18.jpg)
Overlay NetworksGoals
Other properties:• Good expansion
• Good conductance
Random walks converge quickly
Diameter is small
![Page 19: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/19.jpg)
Overlay NetworksIssues
Churn:• Nodes arrive.
• Nodes leave.
Note: Not considering Byzantine/malicious failures today!
![Page 20: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/20.jpg)
Overlay NetworksIssues
Synchrony vs. Asynchrony:• Synchronous: computation proceeds in rounds.
• Asynchronous: arbitrary message/computation delays
Today: assume computation proceeds in rounds.
![Page 21: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/21.jpg)
Overlay NetworksIssues
Oblivious vs. Adaptive scheduler/adversary:• Oblivious: schedule/arrivals/failures fixed in advance.
• Adaptive: schedule/arrivals/failures depends on execution.
Questions: • Do actions of the algorithm correlate with crashes?
• Is there an attacker using knowledge of the system?
Attacker can crash critical nodes?
Sending many messages can overload a link?
![Page 22: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/22.jpg)
Overlay NetworksMetrics
Rate of churn:• How fast can nodes join and leave?
• What is the maximum rate that an algorithm can tolerate?
![Page 23: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/23.jpg)
Overlay NetworksMetrics
Rate of recovery:• What happens when something goes wrong?
• How fast does the algorithm reconstruct a good overlay?
![Page 24: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/24.jpg)
Overlay NetworksMetrics
Costs:• Message complexity how many messages per round?
• Communication complexity how many bits of communication?
• Quiescent complexity what happens if changes stop?
• Adaptive complexity how do costs relate to the changes?
![Page 25: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/25.jpg)
Overlay NetworksOverlay Networks
Given a collection of (changing) servers:
• Choose a good subset of the edges.
• Subgraph has low degree.
• Subgraph has low diameter.
• Maintain edges as servers come and go.
![Page 26: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/26.jpg)
OverlaysA Play in Three Acts
Act I : Fix-it-Felix and the Half-Life-Hobgoblin
Wherein we meet our hero Felix, as he races to keep up with the Half-Life-Hobgoblin, fixing the overlay as fast as it is being destoryed.
Act II : Bob-the-Builder and the Destabilizing-Demon
Wherein we meet our hero Bob, as he counters the Destabilizing-Demon, rebuilding the overlay no matter how badly damaged.
Act III : Here Be Dragons
Wherein we explore the dangerous and unknown path forward and away from the hobgoblins and demons of our despair.
![Page 27: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/27.jpg)
Basic Statistics• Home: Niceland
• Role: Good Guy
• Antagonist: Ralph
• Tool: magic hammer
• Powers: fixing anything
Introductions
Fix-it-FelixI can fix it!
![Page 28: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/28.jpg)
Fix it:• On insert: fix it!
• On departure: fix it!
• On any change: fix it!
Basic philosophy:• Fix changes as soon as
they occur.
• Fix it right then and there.
• Fixes are local and immediate.
Fix-it-Felix Approach
How to cope with changes?I can fix it!
![Page 29: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/29.jpg)
Approach:• Start with good
overlay.
• Bound rate of churn.
• Fix the overlay faster than the churn destroys it.
• Maintain good properties throughout.
Fix-it-Felix Approach
How to cope with changes?I can fix it!
![Page 30: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/30.jpg)
Measuring ChurnMetric: per-period-churn
In every round:• At most k nodes join every r
rounds.
• At most k nodes leave every r rounds.Ex: k=1, r=log n At most 1 node joins / leaves every log(n) rounds.
2520151050
arrive arrivearrive
departdepart depart
departdepart
![Page 31: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/31.jpg)
Measuring ChurnHalf-life of a System
Minimum time in which either:
• Number of nodes doubles.
• Number of nodes halves.If half life is H and there are n nodes at time t, then between [t, t+H], there are at least (n/2) and at most 2n nodes in the system.
2520151050
arrive arrive arrive arrive departarrive
departdepartarrive
H = 5n = 6
![Page 32: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/32.jpg)
Measuring ChurnHalf-life of a System
Ideal Theorem:
If the half-life of the system is at least θ(log n), then the overlay is always good.
If half life is H and there are n nodes at time t, then between [t, t+H], there are at least (n/2) and at most 2n nodes in the system.
Minimum time in which either:• Number of nodes doubles.
• Number of nodes halves.
![Page 33: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/33.jpg)
Tolerating ChurnChord
Advantages:• Tolerates high rate of churn.
• Overlay has many good properties.
• Simple to implement.
Disadvantages: • Oblivious adversary
• Fragile
• Limitations on where nodes can join?
• Only supports one topology.
Once the ring is disrupted, all is lost!
Easy to attack, at risk of correlated failures.
[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
![Page 34: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/34.jpg)
Basic overlay:• Ring topology
• Nodes distributed randomly on ring.
• Edges:- successor- predecssor- “fingers”
• Approximates a hypercube.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
![Page 35: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/35.jpg)
Maintenance:
• Ignore “fingers.”
- Only matters for routing/lookup.
- Easy to rebuild.
• Maintain successor/predecessor.
• Ensures that ring remains a ring.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
![Page 36: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/36.jpg)
Assume adversary is oblivious:
• Adversary does not know where nodes live on the ring.
• New nodes are inserted “randomly.”
• Nodes are deleted “randomly.”
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
![Page 37: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/37.jpg)
Think about crashes:
• Assume at most n/2 nodes crash.
• Ring is disconnected?
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
![Page 38: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/38.jpg)
Think about crashes:
• Assume at most n/2 nodes crash.
• Maintain connection to log(n) successors.
• With high probability, at least 1 successor survives.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
![Page 39: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/39.jpg)
Think about joining:
• Assume at most n/2 nodes join.
• Joining node is attached “randomly.”
• Creates appendages hanging off ring.
• At most O(log n) joining nodes in one place.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
![Page 40: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/40.jpg)
Repair mechanism:
• Query successors for their successors.
• Query successors for their predecessor.
• Update local state.
• Notify successors of changes.
Note: I’m being a bit imprecise.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
![Page 41: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/41.jpg)
Repair mechanism:
• Query successors for their successors.
• Query successors for their predecessor.
• Update local state.
• Notify successors of changes.
Note: I’m being a bit imprecise.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
Beware: what if this node is deleted?
![Page 42: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/42.jpg)
Time analysis:
• Every O(1) steps (in expectation), one insert/delete is resolved.
• Within O(log n) steps, with high probability, all inserts/deletes from the last half-life are resolved.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
![Page 43: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/43.jpg)
Analysis ideas:
• Define an almost good state.
• Show that in the almost good state, the overlay has good properties.
• Show that if the half-life length Ω(log n), then the overlay is always almost good.
Tolerating Churn
Chord[Liben-Nowell, Balakrishnan, Karger, PODC‘02]
![Page 44: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/44.jpg)
Tolerating ChurnChord Summary
Advantages:• Tolerates high rate of churn.
• Overlay has many good properties.
• Simple to implement.
Disadvantages: • Oblivious adversary
• Fragile
• Limitations on where nodes can join.
• Only supports one topology.
Once the ring is disrupted, all is lost!
Easy to attack, at risk of correlated failures.
*Re-Chord (KKS11) yields O(n log n) stabilization.
![Page 45: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/45.jpg)
Think about crashes:
• Within O(nc) steps, log(n) consecutive nodes crash.
• If you wait long enough, bad events happen.
• Then the ring is disconnected.
• It will take a long time to fix!
Tolerating Churn
Chord is fragile?
![Page 46: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/46.jpg)
Recent work by Pamela Zave has found bugs/imprecision in original Chord descriptions:
Using lightweight modeling to understand Chord
Why the Chord Ring-Maintenance Protocol is not
Correct How to make Chord correct
Conclusion: Chord works, but only if you really get it right.
Tolerating ChurnChord details are tricky
![Page 47: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/47.jpg)
Approach:• Start with good
overlay.
• Bound rate of churn.
• Fix the overlay faster than the churn destroys it.
• Maintain good properties throughout.
Fix-it-Felix Approach
Chord tolerates churn.I can fix it!
![Page 48: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/48.jpg)
Tolerating ChurnAlternate Fix-it-Felix approach
Load-balanced hypercube:• At most O(log n) arrivals/departures per round.
• Basic idea: map O(log n) nodes to each vertex of a hypercube.
• Load-balance among hypercube vertices to cope with churn.
• Grow/shrink hypercube as needed.
[Kuhn, Schmid, Wattenhofer, IPTPS’05]
![Page 49: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/49.jpg)
Tolerating ChurnAlternate Fix-it-Felix approach
Load-balanced hypercube:• At most O(log n) arrivals/departures per round.
• Basic idea: map O(log n) nodes to each vertex of a hypercube.
• Load-balance among hypercube vertices to cope with churn.
• Grow/shrink hypercube as needed.
[Kuhn, Schmid, Wattenhofer, IPTPS’05]
Interesting ideas: • Map real nodes to a virtual topology.
• Use load balancing to keep real nodes well distributed.
• Tolerates adaptive (and omniscient) adversary.
Note: (unavoidable) weaker per-round adversarial limit.
![Page 50: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/50.jpg)
Tolerating ChurnAlternate Fix-it-Felix approach
Random (dynamic) graph:• Maintain a random constant-degree graph.
• Up to θ(n/log n) churn per round.
• Based on random walks.
[Augustine, Pandurangan, Robinson, Roche, Upfal, FOCS’15]
Disadvantages: • Not routable.
• Oblivious adversary
• Network size remains fixed.
• Limitation on where new nodes can be attached.
![Page 51: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/51.jpg)
Tolerating ChurnAlternate Fix-it-Felix approach
XHeal:• Fixes any graph that has deletions (and insertions).
• Carefully replaces missing nodes with small expanders.
• Preserves good properties of original graph:- Bounded stretch: log(n)- Bounded degree-growth
[Pandurangan, Trehan, PODC’11]
![Page 52: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/52.jpg)
Tolerating ChurnAlternate Fix-it-Felix approach
XHeal:• Fixes any graph that has deletions (and insertions).
• Carefully replaces missing nodes with small expanders.
• Preserves good properties of original graph:- Bounded stretch: log(n)- Bounded degree-growth
[Pandurangan, Trehan, PODC’11]
Interesting aspects: • One change at a time.
• Local repair.
• Works for any topology/graph.
![Page 53: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/53.jpg)
Fix-it-Felix ApproachMoral of the story:
General ideas:• Local repair.• Simple updates.• Maintain approximate structure.
Several existing solutions:• Oblivious adversary.• Varying rates of churn.• Subtle dependence on the joining model.
Challenges:• Adaptive adversary • Tricky analysis, since failures are ongoing.• Hard to keep costs proportional to changes.
I can fix it!
![Page 54: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/54.jpg)
OverlaysA Play in Three Acts
Act I : Fix-it-Felix and the Half-Life-Hobgoblin
Wherein we meet our hero Felix, as he races to keep up with the Half-Life-Hobgoblin, fixing the overlay as fast as it is being destoryed.
Act II : Bob-the-Builder and the Destabilizing-Demon
Wherein we meet our hero Bob, as he counters the Destabilizing-Demon, rebuilding the overlay no matter how badly damaged.
Act III : Here Be Dragons
Wherein we explore the dangerous and unknown path forward and away from the hobgoblins and demons of our despair.
![Page 55: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/55.jpg)
Basic Statistics• Home: Sunflower
Valley
• Role: Building Contractor
• Antagonist: None.
• Tool: Distributed team of building machinery.
• Powers: Teamwork.
Introductions
Bob-the-Builder Can we build it? Yes we can!
![Page 56: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/56.jpg)
Rebuild good overlay:• On insert: rebuild!
• On departure: rebuild!
• On any change: rebuild!
Basic philosophy:• Assume network is in
some arbitrary topology.
• Build a good overlay.
• Build it.. fast!
Bob–the-Builder Approach
How to cope with changes? Can we build it? Yes we can!
![Page 57: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/57.jpg)
From any initial state:• Build a good overlay.
• Fast construction.
• No ongoing churn.
Self-stabilization:• Arbitrary initial
state corruption.
• Converges to a good state.
Bob–the-Builder Approach
How to build an overlay? Can we build it? Yes we can!
![Page 58: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/58.jpg)
Dynamic Graph ModelAssume an arbitrary initial state:
Initially:• Graph is given in an arbitrary connected topology.
• State of the nodes may be corrupted.
![Page 59: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/59.jpg)
Dynamic Graph ModelConstruct a good overlay:
In every round:• Exchange messages with neighbors.
• Adjust edges.
• Improve the overlay.
No further joins/leaves allowed (until overlay is constructed).
![Page 60: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/60.jpg)
Dynamic Graph ModelStabilization:
Eventually:• Overlay is constructed.
• Good properties are guaranteed.
• In good state, joins and leaves may be supported.
![Page 61: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/61.jpg)
From any initial state:• Build a good overlay.
• Fast construction.
• No ongoing churn.
Self-stabilization:• Arbitrary initial
state corruption.
• Converges to a good state.
Bob–the-Builder Approach
How to build an overlay? Can we build it? Yes we can!
![Page 62: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/62.jpg)
Self-Stabilizing OverlaysSkip+ Overlay Graphs:
Advantages:• Stabilizes quickly.
• Supports efficient joins and leaves.
• Simple rules.
Disadvantages: • Large message / communication complexity (in the worst
case).
• Large degree during construction (in the worst case).
• Oblivious adversary (for departures).
• Churn??
• Only supports one topology.
[Jacob, Richa, Scheideler, Schmid, Taubig, PODC’09]
![Page 63: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/63.jpg)
Skip+ OverlaysClassical Skip List
011010 001000 010000 010001 001111 011
![Page 64: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/64.jpg)
Skip+ OverlaysClassical Skip List
011010 001000 010000 010001 001111 011
Advantages:•Fast search / insert in log(n) time.•Fault-tolerant
![Page 65: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/65.jpg)
Skip+ OverlaysClassical Skip List
011010 001000 010000 010001 001111 011
Disadvantage: congestion•Only one root.•Load is not balanced over nodes.
![Page 66: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/66.jpg)
Skip+ OverlaysSkip Graph
011010 001000 010000 010001 001111 011
![Page 67: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/67.jpg)
Skip+ OverlaysSkip+ Graph
011010 001000 010000 010001 001111 011
![Page 68: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/68.jpg)
Skip+ OverlaysHow to build it?
Overlay guarantees:• Fast searches / routing.
• Fast joins / leaves.
• Small diameter.
• Small degree.
Overlay construction issues: • Initially, diameter of the graph may be large.
• How do nodes find their neighbors efficiently?
• How do nodes sort themselves properly into linked lists?
• Leverage parallelism? Not one insertion at a time!
![Page 69: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/69.jpg)
Skip+ OverlaysSimple trick: pointer doubling
In every round:• Send your entire address book to all your neighbors.
• In every round, your “knowledge diameter” doubles.
• Note: each rounds squares the adjacency matrix.
Within log(n) rounds, graph is a clique.
42 8 12
![Page 70: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/70.jpg)
Skip+ OverlaysSimple trick: pointer doubling
Once you have a clique:• Delete edges not in the overlay.
• Final graph matches good topology.
Within log(n)+1 rounds, graph is Skip+.
[Example: Skip List]
![Page 71: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/71.jpg)
Skip+ OverlaysSimple trick: pointer doubling
Bad news:• Degree grows very large: n-1.
• Messages grow very large: θ(n).
• Number of messages per round is large: θ(n2).
Conclusion: fast, but inefficient, overlay construction.
42 8 12
![Page 72: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/72.jpg)
Skip+ OverlayMore efficient construction:
Basic idea:• Minimize expensive doubling steps.• Route nodes directly using existing overlay, if
possible.
Local view:• Each node uses its local view to determine who its
neighbors should be in the final overlay.
Stable edges:• An edge (u,v) is stable if it appears to be in the final
Skip+ overlay according to both the views of u and v.• Otherwise, an edge is temporary.
![Page 73: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/73.jpg)
Skip+ OverlaysSimple rules:
Rule 1: Introduce Friends • Notify stable neighbors of neighbors they should
know about (including yourself).
Rule 2: Forward Temporary Edges (Routing)• Forward a temporary neighbor to a stable neighbor
with the largest shared prefix.
Rule 3a: Introduce All (Pointer Doubling)• If your set of stable neighbors changes (e.g., due to a
change in views), then introduce all your neighbors to each other.
Rule 3b: Linearize• Help organize your neighbors at the same “level” into
a linked list.
• d
![Page 74: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/74.jpg)
Skip+ OverlayHow does it work?
In general:• Guarantees fast O(log n) overlay construction.• Best effort to restrain message size / degree during
construction.
Locally checkable:• Nodes can locally determine whether the overlay is
correct.• If every node thinks it is correct, then it is!
Bad news:• Still can lead to large degree, large messages, and
many messages per round, in the worst case.
![Page 75: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/75.jpg)
Self-Stabilizing OverlaysSkip+ Overlay Graphs:
Advantages:• Stabilizes quickly.
• Supports efficient joins and leaves.
• Simple rules.
Disadvantages: • Large message / communication complexity (in the worst
case).
• Large degree during construction (in the worst case).
• Oblivious adversary (for departures).
• Churn??
• Only supports one topology.
![Page 76: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/76.jpg)
Self-Stabilizing OverlaysSkip+ Overlay Graphs:
Many improvements, similar ideas:• HSkip+ : Heterogeneous bandwidth
• Corona : Deterministic
• TCF-Skip+: Generalization, local detection analysis, etc.
011010 001000 010000010001 001111 011
![Page 77: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/77.jpg)
Self-Stabilizing OverlaysAlternate Bob-the-Builder approach
Patricia-Tree-like Overlay:• Forms a tree from a weakly connected graph.• Asynchronous.• Communication-efficient:
• Small messages, O(1) per round per nod.• Low contention
[Angluin, Aspnes, Chen, Wu, Yin, SPAA’05]
Disadvantages: • Initially: every node has low degree.
• Builds a tree.
• Later version (SSS’07) builds better overlay.
![Page 78: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/78.jpg)
Bob-the-Builder ApproachMoral of the story:
General ideas:• Deterministic, exact final structure.• Converge to final structure.• Graph square or component merging
to converge fast.
Several existing solutions:• Oblivious adversary.• No joins/leaves during stabilization.
Challenges:• How to keep messages small?• How to avoid graph squaring?• How to tolerate ongoing changes?
Can we build it? Yes we can!
![Page 79: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/79.jpg)
OverlaysA Play in Three Acts
Act I : Fix-it-Felix and the Half-Life-Hobgoblin
Wherein we meet our hero Felix, as he races to keep up with the Half-Life-Hobgoblin, fixing the overlay as fast as it is being destoryed.
Act II : Bob-the-Builder and the Destabilizing-Demon
Wherein we meet our hero Bob, as he counters the Destabilizing-Demon, rebuilding the overlay no matter how badly damaged.
Act III : Here Be Dragons
Wherein we explore the dangerous and unknown path forward and away from the hobgoblins and demons of our despair.
![Page 80: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/80.jpg)
Good overlay:• Small diameter / small degree: O(log n).• Routable / good expansion.
Fast and robust construction:• Rapidly formation from any initial state.• Self-stabilizing.
Churn tolerant:• Tolerates large fractions of nodes joining and leaving
every round.• Oblivious adversary (?).
Efficient construction/maintenance:• Small messages (e.g., size O(log n)).• Message-efficient (e.g., 1 message per node per round).
Big PictureGoals:
![Page 81: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/81.jpg)
Collaborators: Gopal Pandurangan, Peter Robinson, and Amitabh Trahan
![Page 82: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/82.jpg)
OverviewPlan:
Virtual overlays:• Define good topologies.• Map virtual nodes to real servers.
Merging two good overlays:• Start with two good overlays, connected by one link.• Build new good overlay.
Construction algorithm:• Divide-and-conquer.• Merge and collapse components.
Robustness:• Self-stabilization.• Churn.
![Page 83: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/83.jpg)
Virtual OverlaysWhat is a good topology?
Sequence of graphs G1, G2, …, Gn
• Graph Gj has [j, 2j] nodes.
• Graph Gj has O(log j) degree and diameter.
Expandable:• There exists a mapping from each node in Gj to 1 or 2
nodes in G2j.
Random sampling:• Supports a mechanism for randomly sampling nodes.• E.g., fast random walks.
Permutation routing:• Supports efficiently routing permutations.
Note: many (deterministic) expanders satisfy these requirements.
![Page 84: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/84.jpg)
Virtual OverlaysMap virtual topology to real nodes
For each real node:• Assign O(1) virtual nodes to it.
• Connect two real nodes if any of their virtual nodes are connected.
Inspiration: The Forgiving Tree [HST’12], DeX [PRT’14] and [KSW’05]
Virtual Real
![Page 85: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/85.jpg)
OverviewPlan:
Virtual overlays:• Define good topologies.• Map virtual nodes to real servers.
Merging two good overlays:• Start with two good overlays, connected by one link.• Build new good overlay.
Construction algorithm:• Divide-and-conquer.• Merge and collapse components.
Robustness:• Self-stabilization.• Churn.
![Page 86: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/86.jpg)
MergingCombining two overlay networks
Assume two overlay networks:• Each overlay is a properly mapped virtual topology.
• The two overlays are connected by one edge.
Key challenge: merge the two overlay networks.
![Page 87: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/87.jpg)
MergingCombining two overlay networks
Step 1: Grow the virtual topology• Double one overlay (using topology mapping Gn G2n).
• Creates excess virtual nodes.
![Page 88: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/88.jpg)
MergingCombining two overlay networks
Step 1: Grow the virtual topology• Double one overlay (using topology mapping Gn G2n).
• Creates excess virtual nodes.
Step 2: Create new edges• Use permutation routing.
![Page 89: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/89.jpg)
MergingCombining two overlay networks
Step 3: Distribute excess virtual nodes • Send new virtual nodes on a random walk of old
topology.• (Use random sampling of target topology.)
![Page 90: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/90.jpg)
MergingCombining two overlay networks
Step 3: Distribute excess virtual nodes • Send new virtual nodes on a random walk of old
topology.• (Use random sampling of target topology.)
Problem: only one bridge.
![Page 91: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/91.jpg)
MergingCombining two overlay networks
Step 3: Distribute excess virtual nodes • Send new virtual nodes on a random walk of old
topology.• (Use random sampling of target topology.)• Every success creates a new bridge.• Number of bridges doubles (i.e., exponential growth).
![Page 92: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/92.jpg)
MergingCombining two overlay networks
Step 3: Distribute excess virtual nodes • Send new virtual nodes on a random walk of old
topology.• (Use random sampling of target topology.)
Problem: eventually, hard to find an empty node.
![Page 93: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/93.jpg)
MergingCombining two overlay networks
Step 3: Distribute excess virtual nodes • Full nodes send new virtual nodes on random walk of old
topology.
If many nodes are empty, they are easy to find.• Empty nodes send requests on random walk of the old
topology.
If most nodes are full, then there are lots of bridges.
![Page 94: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/94.jpg)
Step 4: Rebalancing and clean up• Balance/reduce the virtual nodes (maybe).• Drop the old topology. (Or keep it as a backup.)
End result: a new instantiation of the virtual topology.
MergingCombining two overlay networks
![Page 95: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/95.jpg)
OverviewPlan:
Virtual overlays:• Define good topologies.• Map virtual nodes to real servers.
Merging two good overlays:• Start with two good overlays, connected by one link.• Build new good overlay.
Construction algorithm:• Divide-and-conquer.• Merge and collapse components.
Robustness:• Self-stabilization.• Churn.
![Page 96: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/96.jpg)
Overlay ConstructionDivide-and-Conquer Algorithm
1. Collections: maintain a collection of overlays.• Initially, each node is its own collection.• Collections are connected by edges between (real) nodes.
2. Matching: find a matching in the graph of collections.
• Pair up collections.
3. Merge: combine pairs of collections.• Merge components.• Combine two components into a single new collection.
4. Repeat: while there is > 1 collection, go to Step 2.
Inspired by [Angluin, Aspnes, Chen, Wu, Yin, SPAA’05]
![Page 97: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/97.jpg)
Overlay ConstructionDivide-and-Conquer Algorithm
1. Collections: maintain a collection of overlays.• Initially, each node is its own collection.• Collections are connected by edges between (real) nodes.
2. Matching: find a matching in the graph of collections.
• Pair up collections.
3. Merge: combine pairs of collections.• Merge components.• Combine two components into a single new collection.
4. Repeat: while there is > 1 collection, go to Step 2. Problem: matching may be
small!
![Page 98: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/98.jpg)
Overlay ConstructionDivide-and-Conquer Algorithm
1. Collections: maintain a collection of overlays.• Initially, each node is its own collection.• Collections are connected by edges between (real) nodes.
2. Sparsify: reduce degree of graph of collections.
• Create constant degree graph.• Each node organizes its “children” into a line.
3. Matching: find a matching in the graph of collections.
• Pair up collections.
4. Merge: combine pairs of collections.• Merge components.• Combine two components into a single new collection.
5. Repeat: while there is > 1 collection, go to Step 2.
![Page 99: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/99.jpg)
Overlay ConstructionDivide-and-Conquer Algorithm
Initially: connected graph
![Page 100: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/100.jpg)
Overlay ConstructionDivide-and-Conquer Algorithm
Sparsify: reduce the degree
![Page 101: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/101.jpg)
Overlay ConstructionDivide-and-Conquer Algorithm
Repeat: match and merge
![Page 102: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/102.jpg)
Overlay ConstructionDivide-and-Conquer Algorithm
Repeat: match and merge
![Page 103: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/103.jpg)
Overlay ConstructionDivide-and-Conquer Algorithm
Repeat: match and merge
![Page 104: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/104.jpg)
Overlay ConstructionDivide-and-Conquer Algorithm
Repeat: match and merge
![Page 105: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/105.jpg)
Overlay ConstructionDivide-and-Conquer Algorithm
Repeat: match and merge
![Page 106: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/106.jpg)
Divide-and-Conquer:• Collection has small diameter O(log n) cost to coordinate.• Collection merging: O(polylog n) cost per merge step.• Number of iterations: O(log n) matchings• Overall: O(polylog n) time to form overlay.
Overlay ConstructionEfficiency Analysis
Merging:• Random walks: O(log n) cost• Bridge doubling: O(log n) iterations• Overall: O(polylog n) time to merge topologies.
![Page 107: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/107.jpg)
OverviewPlan:
Virtual overlays:• Define good topologies.• Map virtual nodes to real servers.
Merging two good overlays:• Start with two good overlays, connected by one link.• Build new good overlay.
Construction algorithm:• Divide-and-conquer.• Merge and collapse components.
Robustness:• Self-stabilization.• Churn.
![Page 108: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/108.jpg)
Self-StabilizationDetecting Trouble
![Page 109: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/109.jpg)
Self-StabilizationDetecting Trouble
Local detection:• If the topology is bad, some node can detect it.• Virtual topology is deterministic.• Can verify assignment of virtual nodes to real nodes, since
each real node is assigned 1 or 2 virtual nodes.
![Page 110: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/110.jpg)
Detection diameter:• Assume there is a topology problem.• Detection diameter = max distance of any node from a
node that can detect the problem.• In any expander, detection diameter is O(log n), even if
real graph diameter is much larger!
Self-StabilizationDetecting Trouble
Local detection:• If the topology is bad, some node can detect it.• Virtual topology is deterministic.• Can verify assignment of virtual nodes to real nodes, since
each real node is assigned 1 or 2 virtual nodes.
[Defined by [Berns, Ghosh, Pemmaraju: SSS’11]
![Page 111: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/111.jpg)
Detection diameter:• Assume there is a topology problem.• Detection diameter = max distance of any node from a
node that can detect the problem.• In any expander, detection diameter is O(log n), even if
real graph diameter is much larger!
Self-StabilizationDetecting Trouble
Local detection:• If the topology is bad, some node can detect it.• Virtual topology is deterministic.• Can verify assignment of virtual nodes to real nodes, since
each real node is assigned 1 or 2 virtual nodes.Conclusion:
Within O(log n) time, every node learns of a topology error.
![Page 112: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/112.jpg)
Self-stabilizatinoBasic idea:
If a node detects an error:• Flood notification to everyone.• Time: O(log n).
On notification of error:• Restart overlay construction from scratch.• Within O(polylog n) time, new overlay.
![Page 113: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/113.jpg)
Self-stabilizatinoBasic idea:
If a node detects an error:• Flood notification to everyone.• Time: O(log n).
On notification of error:• Restart overlay construction from scratch.• Within O(polylog n) time, new overlay.
Remaining issue: synchronization• Different nodes begin rebuilds at different times.• Components synchronize as they merge.
![Page 114: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/114.jpg)
OverviewPlan:
Virtual overlays:• Define good topologies.• Map virtual nodes to real servers.
Merging two good overlays:• Start with two good overlays, connected by one link.• Build new good overlay.
Construction algorithm:• Divide-and-conquer.• Merge and collapse components.
Robustness:• Self-stabilization.• Churn.
![Page 115: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/115.jpg)
ChurnDepartures and Failures
![Page 116: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/116.jpg)
ChurnDepartures and Failures
What types of failures?• Graceful: execute an exit protocol.• Oblivious: adversary decides failures in advance. • Random: adversary fails nodes at random.• Adaptive: adversary chooses failure on-line.• Connected: adversary never disconnects graph.
![Page 117: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/117.jpg)
Overlay tolerates failures• Expanders are highly fault-tolerant.• For “good” target topologies, overlay maintains good
properties, even if nodes fail. • As long as not too many nodes fail, all is good.
ChurnDepartures and Failures
What types of failures?• Graceful: execute an exit protocol.• Oblivious: adversary decides failures in advance. • Random: adversary fails nodes at random.• Adaptive: adversary chooses failure on-line.• Connected: adversary never disconnects graph.
![Page 118: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/118.jpg)
ChurnRepairing failures
Option 1. Local repair • If a node leaves gracefully, hand-off virtual nodes.• If a node crashes, neighbors regenerate virtual nodes.• Rebalance (via random walks).
Conjecture: tolerates small changes.
[as in Kuhn, Schmid, Wattenhofer, IPTPS’05]
![Page 119: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/119.jpg)
Option 2. Periodic Rebuild• Every so often, initiate a complete rebuild.• Assume failures oblivious or random (in some sense).• As long as:
“half-life” > “rebuild time”
then everything continues to work.
ChurnRepairing failures
Option 1. Local repair • If a node leaves gracefully, hand-off virtual nodes.• If a node crashes, neighbors regenerate virtual nodes.• Rebalance (via random walks).
Conjecture: tolerates small changes.
[as in Kuhn, Schmid, Wattenhofer, IPTPS’05]
![Page 120: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/120.jpg)
ChurnKey Question:
Does the construction procedure work when there are ongoing failures?
1. Merging fails if links connecting collections fail:
• Random/oblivious failures not too many links fail / iteration.
2. Collection coordination is more difficult• Can no longer rely on a leader or an aggregation tree in
the collection.• Matching is still doable.• Sparsification is harder.
![Page 121: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/121.jpg)
Joining is easy• Find an extra virtual node.• Route virtual node to new entrant.• Or wait until the next rebuild.
ChurnJoining nodes
How do nodes join?• Random: introduced to a random node.• Oblivious: introduced to a node chosen by the
adversary in advance.• Limited adversarial: introduced to a node, but
adversary cannot introduce more than one at each location.
![Page 122: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/122.jpg)
OverviewPlan:
Virtual overlays:• Define good topologies.• Map virtual nodes to real servers.
Merging two good overlays:• Start with two good overlays, connected by one link.• Build new good overlay.
Construction algorithm:• Divide-and-conquer.• Merge and collapse components.
Robustness:• Self-stabilization.• Churn.
![Page 123: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/123.jpg)
OverlaysA Play in Three Acts
Act I : Fix-it-Felix and the Half-Life-Hobgoblin
Wherein we meet our hero Felix, as he races to keep up with the Half-Life-Hobgoblin, fixing the overlay as fast as it is being destoryed.
Act II : Bob-the-Builder and the Destabilizing-Demon
Wherein we meet our hero Bob, as he counters the Destabilizing-Demon, rebuilding the overlay no matter how badly damaged.
Act III : Here Be Dragons
Wherein we explore the dangerous and unknown path forward and away from the hobgoblins and demons of our despair.
![Page 124: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/124.jpg)
Wrap-upOverlay Networks
Many approaches:• One zillion
different base topologies.
• Many different repair techniques.
• Real-world implementations.
• Algorithms and theory.
![Page 125: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/125.jpg)
A few existing overlay networks…
ChordKademlia
Pastry Tapestry
Skip+
HSkip+
Patricia Tries
DexXHeal
Forgiving Tree
PGrid
Skipnet
RN Protocol
dHamiltonianCycles
Avatar
Ca-Re-Chord
HyperCubes
HyperRing
Chameleon
Re-Chord
Tiara
Corona
![Page 126: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/126.jpg)
Wrap-upA few things I have learned
Churn analysis is hard:
• Relatively few papers really allow failures to happen at any time.
• Analysis is handwavy, and sometimes wrong (see, e.g., Chord).
• Can we do a better job of understanding / proving results regarding churn?
We can fix it!
![Page 127: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/127.jpg)
Wrap-upA few things I have learned
Self-stabilization is also hard:
• System can start in any state! Easy to ignore corner cases.
• Still, easier because there are no ongoing failures.
• Tends to be expensive, due to lots of local checking and rebuilding.
Can we build it?
Yes we can!
![Page 128: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/128.jpg)
Wrap-upA few things I have learned
Integrating both churn and self-stabilization:
• Hard to stabilize if failures are ongoing!
• Competing demands: Self-stabilization
treats every failures as a disaster!
Churn-tolerance tries to ignore/repair small failures.
![Page 129: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/129.jpg)
Wrap-upA few things I have learned
Communication costs:
• Many existing solutions rely on “graph squaring” to handle worst-case.
• Churn-tolerance is relatively cheap (when it works).
• Self-stabilization tends to be much more (communication) expensive.
![Page 130: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/130.jpg)
Wrap-upA few things I have learned
No one has any idea how to deal with Byzantine/malicious or greedy participants in a dynamic overlay network!
Exception: see “Brahms: Byzantine resilient random membership sampling” by Bortnikov, Gurevitch, Keidar, Kliot, Shraer
Exception: see “Self-stabilizing and Byzantine-Tolerant Overlay Network” by Dolev, Hoch, and van Renesse
![Page 131: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/131.jpg)
Wrap-upA few things I have learned
Many interesting ideas.
Many interesting techniques.
No one “right” solution yet.
![Page 132: Bob-the-Builder vs. Fix-it-Felix Maintaining Overlays in Dynamic Graphs Seth Gilbert](https://reader038.vdocuments.us/reader038/viewer/2022103007/56649f4d5503460f94c6d731/html5/thumbnails/132.jpg)
Can we build it?
Yes we can!
We can fix it!