chord fay chang, jeffrey dean, sanjay ghemawat, wilson c. hsieh, deborah a. wallach, mike burrows,...

Chord

Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes,

Robert E. GruberGoogle, Inc.OSDI 2006

Introduction

Dynamo stores objects associated with a key through a simple interface: get(),put()

It should be possible to scale Dynamo incrementally

This requires the ability to partition data over the set of nodes (storage hosts)

Dynamo relies on a concept called consistent hashing The approach they used is similar to that

found in Chord.

Distributed Hash Tables (DHT)

Operationally like standard hash tables Stores (key, value) pairs

The key is like a filename The value can be file contents or pointer to

location Goal: Efficiently insert/lookup/delete

(key,value) pairs Each peer stores a subset of (key,

value) pairs in the system

DHT

Core operation: Find node responsible for a key Map key to node Efficiently route insert/lookup/delete request

to this node Allow for frequent node arrivals and

departures

DHT Introduce a hash function to map the object being

searched for to a unique global identifier: e.g., h(“NGC’02 Tutorial Notes”) → 8045

Distribute the range of the hash function among all nodes in the network

Each node must “know about” at least one copy of each object that hashes within its range (when one exists)

0-9999500-9999

1000-19991500-4999

9000-9500

4500-6999

8000-8999 7000-8500

8045

DHT:Desirable Properties

Key ID space (search space) is uniformly populated Mapping of keys to IDs using (consistent) hashing

A node is responsible for indexing all the keys in a certain subspace of the ID space

Nodes have only partial knowledge of other node’s responsibilities

Messages should be routed to a node efficiently (small number of hops)

Node arrival/departure should only affect a few nodes.

Consistent Hashing

The main idea: map both keys and nodes (node IPs) to the same (metric) ID space

Consistent Hashing

The main idea: map both keys and nodes (node IPs) to the same (metric) ID space

The ring is just a possibility.Any metric space will do

Consistent Hashing

With high probability, the hash function balances load (all nodes receive roughly the same number of keys).

With high probability, when a node joins (or leaves) the network, only an fraction of the keys are moved to a different location. This is clearly the minimum necessary to

maintain a balanced load.

Consistent Hashing

The consistent hash function assigns each node and key an m-bit identifier using SHA-1 as a base hash function.

A node’s identifier is chosen by hashing the node’s IP address.

A key identifier is produced by hashing the key. For more info see:

D. R. Karger, E. Lehman, F. Leighton, M. Levine, D. Lewin, and R.Panigrahy, “Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on theWorldWideWeb,” in Proc. 29th ACM Symp. Theory of Computing, El Paso, TX, May 1997, pp. 654–663.

P2P Middleware: Differences

Different P2P middlewares differ in: The choice of the ID space The structure of their network of nodes (i.e.

how each node chooses its neighbors) For each object, node(s) whose range(s)

cover that object must be reachable via a “short” path

This is a major research topic

Chord

m bit identifier space for both keys and nodes

Key identifier = SHA-1(key) Key = “LetItBe” ID=50 Key = “129.100.16.93” ID=70

How do we assign keys to nodes?

SHA-1

SHA-1

Chord

Nodes organized in an identifier circle based on node identifiers

Keys assigned to their successor node in the identifier circle e.g., node with next higher ID.

Chord Hash function

ensures even distribution of nodes and keys on the circle

Range covered by node is from previous ID up to its own ID

Assume an N node network

Chord: Search Possibilities

Routing table size vs search cost Every peer knows every other peer:

O(N) routing table size Every peer knows its successor: O(N)

search time. The “compromise” is to have each peer

know the next m successors.

Finger Table

Let m be the number of bits in the key/node identifiers

Each node, n, maintains a routing table with at most m entries called the finger table.

The ith entry in the table at node n contains the identity of the first node, s, that succeeds n by at least 2i-1. s = successor(n+2i-1) s is called the ith finger of node n

Chord:Finger Table

Finger table:finger[i] = successor (n + 2i-1)

where 1 ≤ i ≤ m

O(log N) table size

Chord: Finger Table

Finger table:finger[i] = successor (n + 2i-1)

Chord: Finger Table

Finger table:finger[i] = successor (n + 2i-1)

Chord: Finger Table

Finger table:finger[i] = successor (n + 2i-1)

Chord: Finger Table

Finger table:finger[i] = successor (n + 2i-1)

Chord: Finger Table

Finger table:finger[i] = successor (n + 2i-1)

Chord: Finger Table

Finger table:finger[i] = successor (n + 2i-1)

Chord: Finger Table

Finger table:finger[i] = successor (n + 2i-1)

Chord: Finger Table

Finger table:finger[i] = successor (n + 2i-1)

The Chord algorithm –Scalable node localization

Chord: Search

Assume node n is searching for key k. Node n does the following:

Find ith table entry of node n such that k[finger[i].start, finger[i+1].start])

If no such entry exists then return the node in the last entry of the finger table

The above two steps are repeated until the condition in the first step is satisfied.

Chord: Join

Nodes can join (and leave) at any time. Challenge: Preserving the ability to

locate every key in the network Chord must preserve the following:

Each node’s successor correctly maintained For every key k, node successor(k) is

responsible for k. For lookups to be fast, it is desirable for

the finger tables to be correct.

Chord: Join Implementation

Each node in Chord maintains a predecessor pointer. This consists of the Chord ID and IP address

of the immediate predecessor of that node. It can be used to walk counterclockwise

around the identifier circle. The new node to be added learns the

identify of an existing Chord node by some external mechanism

Chord: Join Initialization Steps

Assume n is the node to join. Find any existing node, n’. Find successor of n from n’. Label this

successor(n). Ask successor(n) for its predecessor.

This is labelled as predecessor(successor(n)).

Chord: Join Example

•Assume N26 wants tojoin; If finds N8

•N8’s finger table suggests that N26 will be “between” N21 and N32.

Chord: Join (Initialize finger table)

Node n needs to have its finger table initialized

Node n can ask one its predecessor to be for its finger table as a starting point

Chord: Join (Changing Existing Finger Tables)

Node n needs to entered into the finger tables of some existing nodes.

Node n becomes the ith finger of node p, iff p precedes n by at least 2i-1 ; and The ith finger of node p succeeds n.

The first node, p, that satisfies these conditions is the immediate predecessor of n-2i-1

For a given n, the algorithm starts with the ith

finger of node n and then continues to walk in the counter-clock-wise direction on the identifier circle until it encounters a node whose ith finger precedes n.

Chord: Join Example (add N26)

N21+1 N32

N21+2 N32

N21+4 N32

N21+8 N32

N21+16

N38

N21+32

N56

N21 (old finger table)

N21+1 N26

N21+2 N26

N21+4 N26

N21+8 N32

N21+16 N38

N21+32 N56

N21 (new finger table)

i=1: Does N21 precede N26 by at least 1 (2i-1); yes: N21+1 becomes N26;

i=2: Does N21 precede N26 by at least 2; yes: N21+2 becomes N26;

i=3: Does N21 precede N26 by at least 4; yes: N21+4 becomes N26;

i=4: Does N21 precede N26 by 8; no; evaluate N14;

Chord: Join Example (add N26)

N14+1 N21

N14+2 N21

N14+4 N21

N14+8 N32

N14+16

N32

N14+32

N48

N14 (new finger table)

N14+1 N21

N14+2 N21

N14+4 N21

N14+8 N26

N14+16 N32

N14+32 N48

N14 (new finger table)

i=4: Does N14 precede N26 by at least 8; yes; N14+8 becomes N26

i=5; Does N15 precede N26 by at least 16; no; evaluate N8

Etc

Chord: Join (Transferring Keys)

Move responsibility for all the keys for which node n is the successor.

Typically this involves moving data associated with each key to the new node.

Node n can become the successor for keys that were previously the responsibility of the node immediately following n.

Node n only needs to contact one node to transfer responsibility for all relevant keys.

Chord: Join

The previous discussion on join focuses on a single node join.

What if there are multiple node joins? Join requires that each node’s

successor is correctly maintained

Chord: Stabilization Protocol The successor/predecessor links are

rebuilt by periodic stabilize notification messages Sent by each node to its successor to inform

it of the (possibly new) identity of the predecessor

The successor pointers are used to verify and correct finger table entries.

Chord: Join/Stabilize Example

Chord: Join/Stabilize Example

• N26 joins the system

• N26 acquires N32 as its successor

• N26 notifies N32

• N32 acquires N26 as its predecessor

Chord: Join/Stabilize Example

• N26 copies keys

• N21 runs stabilize() and asks its successor N32 for its predecessor which is N26.

Chord: Join/Stabilize Example

• N21 aquires N26 as its successor

Chord Stabilization

Pointers and finger tables may be in a state of flux

Is it possible that data will not be found? Yes

Recovery: try again

Chord: Node Failure

N120

N113

N102

N80

N85

N80 doesn’t know correct successor, so incorrect lookup

N10

Lookup(90)

Chord: Node Failure

Solution: Use successor lists Each node knows r immediate successors After failure, will know first live successor Stabilize messages correct finger tables Replicas of the data associated with a key at

the r successor nodes might be used Application dependent

Chord Properties

In a system with N nodes and K keys, with high probability… each node receives at most K/N keys each node maintains info. about O(log N) other nodes lookups resolved with O(log N) hops Insertions O(log2N)

The developers of Chord validated this through simulation studies.

No consistency among replicas Hops have poor network locality

Chord: Network Locality

Nodes close on ring can be far in the network.

CA-T1CCIArosUtah

CMU

To vu.nlLulea.se

MITMA-CableCisco

Cornell

NYU

OR-DSLN20

N41N80N40

* Figure from http://project-iris.net/talks/dht-toronto-03.ppt