Performance improvement of Chord Distributed Hash Table under high churn rate

Giang Ngo Hoang, Hung Nguyen Chan, Khang Nguyen Van, Thu Le thi Xuan,

Thang Nguyen Manh Hanoi University of Technology

Hanoi, Vietnam {nhgiang-linc,chanhung}@mail.hut.edu.vn

Vinh Vu Thanh Faculty of Information Technology

Thai Nguyen University Thai Nguyen, Vietnam

Abstract—Structured peer-to-peer (P2P) networks is becoming popular for their advantages of high scalability and good performance. These networks are based on the Distributed Hash Tale or DHT mechanism to establish and maintain a certain topology. However, while adopting structured P2P network for wireless environment, one have to deal with the phenomenon of frequent join and leave of mobile nodes, namely churn. In this study, we focus on modification of Chord, a well-known DHT protocol, in order to improve its performance under high churn rate. We adopt the atomic ring maintenance mechanism to mitigate the effect of churn over lookup consistency and successfully implement this mechanism in OverSim simulator. Our simulation showed significant performance improvement of the modified Chord protocol under high churn rate in wireless environments.

Keywords: P2P, distributed hash table, DHT, churn, structured P2P network.

I. INTRODUCTION Since the introduction of the Napster file-sharing system,

P2P network has evolved through three generations. The first P2P model employs a central index server to store all the data information to facilitate searching and other management tasks. The second generation P2P model uses the flooding request mechanism for locating resources by broadcasting requests to all neighbor peers. Both these models have the problem of scalability, which motivated the innovation of Distributed Hash Table mechanism to better support scalability, load balancing and fault tolerance of P2P networks. In DHT-based P2P systems, data items are associated to keys created from their unique attributes. Each peer is assigned an ID and is responsible for storing a range of keys which is closest to its ID. When a node lookups for a key, the network will return the ID of the node where file associated with that key is stored. DHT allows nodes to exchange files based on their keys. Since this model has proved to be highly effective in large scale networks, most modern P2P systems are DHT-based. Some well-known DHTs are Chord [5], Kelips [11], Tapestry [9], Kademlia [10], etc.

The advent of advanced broadband wireless communication such as 3G, WiMax and WiFi is driving P2P applications from desktop-based file sharing to multimedia applications such as

P2PVoD [12] and P2P streaming [13], which expected to operate not only on laptops but also on handheld devices. However, most of studies on structured P2P networks focus on the network of long-live nodes, which do not join and leave the P2P network very frequently. In the wireless environment, where communication links is heavily affected by high packet loss rate and bandwidth fluctuations, the situation is much more complicated. In addition, various energy-saving mechanisms try to save battery life of mobile equipments by turning off unnecessary data transmission which shortens node online time. As a result, churn rate of mobile nodes are much higher than that of fixed node in wired networks, which require significant modifications in P2P mechanisms.

A. Background of the Chord DHT Chord [5] uses consistent hashing to assign peers and data

key to m-bit identifiers. A peer identifier is obtained by hashing peer’s IP address while a key identifier is made by hashing the data key. Chord structures its identifiers space in a circle of numbers from 0 to 2m – 1. Key k is assigned to the first node clockwise from k whose identifier is equal to or follows the identifier of k in the identifier space. This node is called the successor node of key k, denoted by successor (k). A node in b-base Chord keeps (b-1)log b(n) fingers whose its IDs lies at exponentially increasing fractions of the ID space away from itself. Each node keeps a successor list of n successor nodes. Lookup for a key return the successor of node whose ID most closely precedes the key. Figure 1 illustrates Chord algorithm with m =3.

The Chord protocol uses a stabilization protocol running periodically in the background to update the successor list (every tfinger) and the entries in the finger table (every tsucc). Successor list and finger table are stabilized separately.

B. The effects of churn on Chord DHT While in wire-line P2P, the join and leave of a node (named

as churn) is mostly under user control and occur in rather low frequency (every 1-2 hours), in the wireless and mobile P2P, the causes of churn is most likely out-of-control and its frequency is much higher (every 120 to 600 sec). Some possible reasons are: a) intermittent radio link, b) the battery

The 2009 International Conference on Advanced Technologies for Communications

978-1-4244-5139-5/09/$26.00 ©2009 IEEE 191

saver mechanism of mobile terminal, c) user turn off or put his mobile terminal in standby mode.

Figure 1: Illustration of Chord mechanism in the case m =3

Since DHT algorithms try to establish and maintain certain overlay topology (such as the ring topology in Chord), high frequency of node join and leave will greatly affect topology stability, causing increase in both communications cost to reestablish partnership between neighboring nodes and time of successful lookup. A number of studies tried to mitigate the effect of churn on DHTs [4],[1][8] and [3].

The remainder of this paper is organized as follows: After reviewing Chord problems under churn, in the next section, we present the customized Chord protocol, which assure better lookup consistency under high churn rate. The simulation study is given next with two subsections dedicated for describing simulation setup and analyzing the results. The fourth section review related work and highlight our contributions. Finally, the last section draws some concluding remarks and future work.

II. MODIFICATION OF CHORD DHT As indicated in previous section, one of the main reasons

for the degradation of Chord performance under churn is the inconsistency of node pointers, which are used intensively in searching activities.

When a node joins in network, basic Chord does not immediately correct successor and predecessor pointers of all relevant nodes. Instead, basic Chord only corrects some pointers and relies on periodical stabilization process to correct rest of pointers. Similarly, basic Chord uses periodical stabilization process to deal with pointer inconsistency caused by node failures. As a result, some pointers of relevant nodes are incorrect during the intervals between stabilization processes, which again decreases success lookup ratio.

The joining process in basic Chord is depicted in Figure 2. We assume that in a basic Chord ring, node q is successor of

node p and node r wants to join the network (0a). Node r send join request to a bootstrap node, which forward it again to node q which is a successor of node r. Node q sets predecessor pointer to r and sends the signal join response to node r. Upon receive this signal, node r set its successor pointer to q (b). Before the next stabilization process is invoked at node p, successor pointer of node p is incorrect (c). Consequently, node p will return incorrect result upon receive a lookup for node r.

Figure 2.The problem of joining process

The modified Chord tries to fix the problem of inconsistent pointers by forcing intermediate stabilization in joining/ leaving activities and reducing the inconsistency of pointers under failure. We consider node leave as a failure as mobile devices normally leave the network due to energy saving schemes or communication disruption.

Node p Node q

a. Before joining process

Join request

Join response

Node r (joining) Node q

Pred = r

Succ = q

b. Joining process

Node p Node r

(joining)

Node q

c. After joining process

192

Figure 3.Sequence diagram of the modified joining process.

We adopt a mechanism called “atomic ring maintenance” proposed by [1]. This mechanism uses lock to eliminate concurrent joining. Figure 3 is a sequence diagram describes the modified joining mechanism of a node.

As can be seen from the Figure 3, when node r wants to join in network, it takes its lock and send join request signal to a boot strap node. Join request is then forwarded to node q which is a successor of node r. If lock of node q is free, it takes the lock and sets predecessor pointer to r and then sends join response to node r. Upon receives join response, node r sets its successor pointer to q and predecessor pointer to p. Node r then sends successor hint message to node p to notify node p of new successor. After node p receive this message, it sets its successor to node r and also send new successor acknowledge message to node q. Node q receive new successor acknowledge message and sets its lock to free, then returns the join done message to node r. Finally, node r receives join done message and sets its lock to the “free” state.

To avoid deadlock situation, each node has a lock timeout. When a node sets its lock to the “taken” state, it also starts a timer. If the locking time exceeds the default timeout interval, lock is set back to the “free” state.

We deal with failure using a combination of periodical stabilization mechanism of basic Chord and lock timeout. Different from algorithm of Ali Godshi [1], stabilization not only runs periodically but also is triggered by timeout event. Upon lock timeout, the stabilization process starts immediately at the successor of the new joining node.

III. SIMULATION STUDY

A. Simulation setup In order to verify the efficiency of modified Chord

protocol, we implement the locking mechanism and stabilization algorithm in C++ code of OverSim [7], a new simulator for P2P network developed on top of OMNET++ framework [6], a popular graphical simulation. The interactive GUI and real-time messages of OverSim are extremely useful for debug and verification the new algorithms.

Performance of basic and modified Chord is evaluated by two main parameters: successful lookup ratio and average hop count taken by lookups. They are simulated under the same conditions. In simulations, various network sizes from 100 nodes to 2000 nodes are tested under various churn rates from 120s to 720s. Table 1 and Table 2 list the simulation scenarios and the parameters of basic and modified Chord.

TABLE I. SIMULATION SCENARIOS

Node numbers Churn rates 100, 500, 1000, 1500, 2000 (node) 120, 180, 360, 540, 720 (s)

TABLE II. CHORD PARAMETERS

Type Parameters Values (seconds) Common Chord parameters Stabilize delay 10

Fix fingers delay 20 Number of backup nodes 3 Successor list size 8

Modified Chord parameters Lock timeout 19

B. Results and comments Figure 4 and Figure 6 compare successful lookup ratio of

basic and modified Chord networks of various size under churn rates varying from 120 to 720 sec.

Successful lookup ratio vs. Churn rate

00.10.20.30.40.50.60.70.80.9

1

0 100 200 300 400 500 600 700 800

Churn rate

Succ

essf

ul lo

okup

ratio

basic 100modified 100basic 1000modified 1000basic 2000modified 2000

Figure 4. Successful lookup ratio of basic Chord and modified Chord operated

in 100, 1000, 2000 node networks under various churn rates.

Succ = r

Node p Node q Node r (joining)

Join request

Join

<New succ hint>

New succ ack

Join done

Pred = r

Succ = q Pred = p

Lock=taken

Lock=taken

Lock=free

Lock=free

193

From Figure 4 we can see that modified Chord achieve successful lookup ratio much higher than that of basic Chord of same network size and under the same churn rate. For example, at churn rate of 300 sec, modified Chord of 100 nodes out-performs basic Chord by a factor of 2. This performance gain is due to the improvement in consistency of successor and predecessor pointers of nodes in modified Chord.

When observing the area of very high churn rate, we found that there is a barrier of churn rate around 180 sec that both protocol failed with successful lookup rate down to under 0.4. The acceptable area of churn is generally higher than 500 seconds. But in the case of small network of 100 nodes, this limit is pushed down to 360 sec. This fact hints at the solution of clustering Chord ring for better churn tolerance.

Figure 5: The interactive GUI of Over Sim.

By observing OverSim GUI and real-time log, we also found several interesting effects of churn to Chord topology. One observed effect showed that Chord network tends to partitioned under high churn rate. A Chord ring is partitioned into two separate networks if all nodes in the successor lists of two nodes in Chord ring fail. Under higher churn rate, this phenomenon occurs more frequently and the number of partition also increases. Since nodes in a separated partition can not lookup for nodes in other partitions, overall successful lookup ratio dramatically decreases.

Successful lookup ratio vs. Node number

00.10.20.30.40.50.60.70.80.9

1

0 500 1000 1500 2000

Node Number

Succ

essf

ul lo

okup

ratio

basic 180modified 180basic 360modified 360basic 720modified 720

Figure 6. Successful lookup ratio of basic and modified Chord operated in

various network sizes under churn rate of 180s, 360s and 720s.

Figure 6 shows successful lookup ratio of basic and modified Chord under churn rate of 180s, 360s and 720s as a function of network size. It can be observed that modified Chord can achieve better performance in all cases, up to twofold in fairly large network of 2000 nodes under high churn rate of 360 sec. However, under extremely high churn rate of 180 sec, both basic and modified Chord show similar and low successful lookup rate regardless of network size.

Hop count vs. churn rate

0

1

2

3

4

5

6

0 100 200 300 400 500 600 700 800

Churn rate

Hop

coun

t

basic 100modified 100basic 1000modified 1000basic 2000modified 2000

Figure 7. Hop count of basic Chord and modified Chord operated in 100,

1000, 2000 node networks under various churn rates.

194

Hop count vs. Node number

0

1

2

3

4

5

6

0 500 1000 1500 2000

Node number

Hop

cou

nt

basic 180modified 180basic 360modified 360basic 720modified 720

Figure 8. Hop count of basic and modified Chord operated in various

networks under churn rate of 180s, 360s and 720s.

Figure 7 and Figure 8 compare the parameter hop count of successful lookup in basic and modified Chord networks of different sizes under different churn rates. As can be observed in these figures, modified Chord has very small effect on hop counts under every churn rate, (slight increases in some cases) in all network sizing from 100 to 2000 nodes. It is desired that the hop counts parameter is kept in small value as it directly related to lookup latency.

Successful lookup ratio vs. CCDF

00.10.20.30.40.50.60.70.80.9

1

0 0.2 0.4 0.6 0.8 1

Successful lookup ratio

CC

DF

basic_1000_360modified_1000_360basic_2000_360modified_2000_360basic_100_180modified_100_180

Figure 9.CCDF of successful lookup ratio in basic and modified Chord

networks of 100, 1000 and 2000 nodes under churn rate of 180s and 360s.

In order to obtain high fidelity results, we repeat simulations several times for each scenario and calculate the statistical parameter of results from all runs. Figure 9 graphs the Complementary Cumulative Distribution Function (CCDF) of successful lookup ratio of basic and modified Chord networks which consist of 100, 1000 and 2000 nodes under churn rate of 180s and 360s. Again, we observe higher probability of modified Chord to obtain the same successful lookup ratio in all cases. The reason for this result is the higher

consistency of successor and predecessor pointers of nodes in modified Chord between stabilization activities.

IV. RELATED WORK Our work was inspired by the work of Ali Ghodsi [1]

namely “Atomic ring maintenance”, which again derived from [8]. Differ from their approach, we only modified the join process and consider leave as a failure since this is the common case in mobile wireless environment. In addition, we adopt a hybrid stabilization mechanism where proactive stabilization is activated on join/leave activities and periodical stabilization still run in background. Moreover, Ali Ghodsi only proposed several algorithms and proved it using analytical method without fully evaluates their performance. S. Rhea et al [4] using emulation of 1000 DHT nodes to evaluate several DHT mechanisms under churn. They focused on three issues: reactive versus periodic recovery from neighbor failure, the calculation of timeouts on lookup messages, and proximity neighbor selection. By introducing a timeout mechanism based on virtual coordinates, they tried to improve the mean latency. However, most of these works assume a normal scenario of file-sharing applications, without regard to the characteristics of wireless environments.

Our previous study [2] on DHT performance under churn in wireless environment compared several well-known DHT such as Chord, Kelips and Tapestry using the “Performance versus cost” framework. This study showed that Chord is heavily affected by very high churn rate and even perform the worse among three DHTs under test.

V. CONCLUSION AND FUTURE WORK In this study, we propose a modified Chord protocol with

locking mechanism and new stabilization strategy to ensure the consistency of node pointers under high churn rate. We also successfully implement the modified Chord in OverSim simulator to verify the new protocol. The simulation results show significant performance improvement of modified Chord over basic Chord under various conditions of churn rate and network size. However, the modification on Chord protocol does not affect much on other aspects of Chord such as average hop counts of successful lookup. We also discover a barrier of churn rate around 120 sec that even the modified protocol fails. One interesting result found is that smaller Chord network of hundreds nodes sometime outperform larger network of thousand nodes under churn. This fact leads to a possible solution of mitigating churn effect by clustering large Chord ring into multiple smaller rings connected through a second level overlay. In the next phase of our study, we will study the feasibility of this solution and develop mechanisms for modified Chord to deal with network partitioning under very high churn rate. To further improve Chord performance under churn, we are investigating a DHT proxy mechanism for caching lookup and correcting lookup results. We will also study and evaluate network partitioning phenomenon under high churn rate of others DHT protocols such as Kademlia and Tapestry.

195

ACKNOWLEDGMENT The authors express their gratitude to Panasonic Singapore

Laboratory (PSL) and the national ICT key program KC.01/06-10 for supporting this research.

REFERENCES [1] Ali Ghodsi. “Distributed k-ary System: Algorithms for Distributed Hash

Tables”, PhD dissertation, KTH-Royal Institute of Technology, 2006. [2] Hung Nguyen Chan, Khang Nguyen Van, Giang Ngo Hoang,

“Characterizing Chord, Kelips and Tapestry algorithms in P2P streaming applications over wireless network”, International Conference on Communications and Electronics ICCE 2008, Hoi An, Vietnam

[3] Sameh El-Ansary, “Designs and Analyses in Structured Peer-to-Peer Systems”, Ph.D. Disertation. KTH-Royal Institute of Technology, 2005.

[4] S. Rhea, D. Geels, T. Roscoe, and J. Kubiatowicz, “Handling churn in a DHT,” in Proceedings of the 2004 USENIX Technical Conference, June 2004.

[5] Ion Stoica, Robert Morris, David Karger, Frans Kaashoek, and Hari Balakrishnan. “Chord: A scalable Peer-To-Peer lookup service for internet applications”. In Proceedings of the 2001 ACM SIGCOMM Conference, pages 149–160, 2001.

[6] Omnet ++, http://www.omnetpp.org/

[7] OverSim, http://www.oversim.org [8] N. A. Lynch, D. Malkhi, and D. Ratajczak. “Atomic Data Access in

Distributed Hash Tables”. In Proceedings of the First Interational Workshop on Peer-to-Peer Systems (IPTPS'02), Lecture Notes in Computer Science (LNCS), pages 295-305, London, UK, 2002. Springer-Verlag.

[9] B. Y. Zhao, L. Huang, J. Stribling, S. C. Rhea, A. D. Joseph, and J. D. Kubiatowicz, “Tapestry: A resilient global-scale overlay for service deployment,” IEEE Journal on Selected Areas in Communications, vol. 22, no. 1, pp. 41–53, Jan. 2004.

[10] P. Maymounkov and D. Mazieres, "Kademlia: A Peer-to-Peer Information System based on the XOR Metric," presented at International Workshop on Peer-to-Peer Systems (IPTPS), 2002.

[11] I. Gupta, K. Birman, P. Linga, A. Demers, and R. v. Renesse, "Kelips: Building an Efficient and Stable P2P DHT Through Increased Memory and Background Overhead," presented at Second International Workshop on Peer-to-Peer Systems (IPTPS), Berkeley, CA, USA, 2003.

[12] Vaishnav Janardhan, Henning Schulzrinne, “Peer assisted VoD for set-top box based IP network”, in Workshop of Proc. of ACM SIGCOMM, August 2007.

[13] Eleni Mykoniati, Raul Landa, Spiros Spirou, Richard G. Clegg, Lawrence Latif, David Griffin, “Scalable Peer-to-Peer Streaming for Live Entertainment Content” and Miguel Rio, University College London, IEEE Communications Magazine • December 2008

196