Computer Networks 67 (2014) 141–153

Contents lists available at ScienceDirect

Computer Networks

journal homepage: www.elsevier .com/ locate/comnet

VirtualKnotter: Online virtual machine shuffling for congestionresolving in virtualized datacenter q

http://dx.doi.org/10.1016/j.comnet.2014.03.0251389-1286/� 2014 Elsevier B.V. All rights reserved.

q A shorter version of this paper has been published in the IEEEICDCS’2012.⇑ Corresponding author. Tel.: +86 10 62283412.

E-mail addresses: zoush@bupt.edu.cn (S. Zou), xwe334@eecs.northwestern.edu (X. Wen), kaichen@cse.ust.hk (K. Chen),huangshan19j@gmail.com (S. Huang), ychen@northwestern.edu(Y. Chen), liuyq7809@gmail.com (Y. Liu), xy12180@gmail.com (Y. Xia),huc@ieee.org (C. Hu).

Shihong Zou a,⇑, Xitao Wen b, Kai Chen c, Shan Huang a, Yan Chen b, Yongqiang Liu d, Yong Xia d,Chengchen Hu e

a State Key Laboratory of Networking and Switching, Beijing University of Posts and Telecommunications, Beijing 100876, Chinab Northwestern University, Evanston, IL 60208, USAc Hong Kong University of Science and Technology, Hong Kong, Chinad NEC Labs China, Beijing 100084, Chinae Xi’an Jiaotong University, Xi’an 710049, China

a r t i c l e i n f o a b s t r a c t

Article history:Received 20 December 2013Received in revised form 26 February 2014Accepted 31 March 2014Available online 5 April 2014

Keywords:Link congestionVM placementVM migrationDatacenter

Our measurements on production datacenter traffic together with recently-reportedresults (Kandula et al.) [1] suggest that datacenter networks suffer from long-lived conges-tion caused by core network oversubscription and unbalanced workload placement. In con-trast to traditional traffic engineering approaches that optimize flow routing, in this paper,we explore the opportunity to address the continuous congestion via optimizing VM place-ment in virtualized datacenters. To this end, we present VirtualKnotter to reduce conges-tion with controllable VM migration traffic as well as low migration time, which includesan online VM placement algorithm and an efficient VM migration scheduling algorithm.Our evaluation with both real and synthetic traffic patterns shows that VirtualKnotter per-forms close to the baseline algorithm in terms of link unitization, with only 5–10% migra-tion traffic of the baseline algorithm. Furthermore, VirtualKnotter decreases linkcongestion time by 53% for the production datacenter traffic.

� 2014 Elsevier B.V. All rights reserved.

1. Introduction oversubscription at its aggregation and core layers. How-

Driven by technology advances and economies of scale,datacenters are becoming the mainstream hosting plat-form for a variety of infrastructure services (such asMapReduce [2], GFS [3] and Dryad [4]) and data-intensiveapplications (such as online social networking, searching,scientific computing). Today’s datacenters usually form amulti-root multi-level (typically 3 tiers) tree with

ever, due to the massive nature of communication patternin the datacenter network, it frequently exhibits high linkutilization and even congestion at aggregation or core lay-ers [1]. While high resource utilization is favorable fordatacenter owners, network congestion can cause harmfulqueuing delay and packet loss, and thus affects the net-work throughput. These consequences could significantlydegrade application performance and user experience.Therefore, addressing the congestion problem in datacen-ters is a meaningful goal and is the focus of this paper.

To address this problem, we resort to an increasinglyadopted feature in modern datacenter : virtualization tech-nology. Live virtual machine (VM) migration, as an impor-tant capability of virtualization technology, enables us tomove a live VM from one host to another whilemaintaining near continuous service availability. Live VM

142 S. Zou et al. / Computer Networks 67 (2014) 141–153

migration provides a new dimension of flexibility - rear-ranging VM placement on the fly. Such spatial flexibilityis proved to be effective in several scenarios, including ser-ver consolidation, power consumption saving, fault toler-ance and QoS management [5–8]. In our case, the spatialmobility also creates an opportunity to solve the conges-tion problem. Through a better VM placement, we canlocalize a majority of traffic under ToR switches, balancethe outgoing traffic, and thus resolve congestion.

However, as a limitation, live VM migration usually takestens of seconds to transfer VM states and launch on newhost, which means a new VM placement will not take effectuntil all the transfers complete. Thus, to benefit from VMshuffling, we expect long-term stability in the traffic, so thatwe can predict the future traffic pattern and have time toadjust VM placement. Although no previous measurementdirectly shows the traffic stability in datacenters, the prev-alence and massive nature of data-intensive applicationsindicate the existence of long-term traffic pattern. Forexample, typical applications like search engine indexingand logistic regression tend to exhibit long runtime andlasting traffic pattern as we will discuss in Section 6. Fur-thermore, such long-term traffic pattern is witnessed inour measurement study. As we will show in Section 3, wecollect and analyze an 18-h traffic trace from a productiondatacenter, and observe: (a) highly utilized core and aggre-gation network with lasting congestion pattern; and (b) awell-predictable end-to-end traffic at a time granularity oftens of minutes. Such observations, coupled with the popu-larity of datacenter virtualization technology, point a poten-tial avenue to address congestion via online VM shuffling.

Following this, we propose to tackle the network conges-tion problem through online VM shuffling. We choose tominimize the maximum link utilization, and formulate itas an optimization problem, which is shown to be a varia-tion of the NP-hard quadratic bottleneck assignment prob-lem (QBAP). We therefore design VirtualKnotter, anincremental heuristic algorithm that efficiently optimizesVM placement with controllable VM migration overhead.In addition, we design an efficient VM migration schedulingalgorithm to minimize the time to do the shuffling. We eval-uate the algorithms with various real-world and synthetictraffic patterns. We specifically compare VirtualKnotterwith a clustering-based baseline algorithm that is expectedto produce near-optimal link utilization. Our results suggestthat VirtualKnotter achieves a link utilization performancethat is close to the baseline algorithm, but with only5–10% migration traffic compared with the baseline algo-rithm. Our simulation further evaluates the total congestiontime on each link before and after applying VirtualKnotter.The result shows VirtualKnotter is able to decrease link con-gestion time by 53%, demonstrating the opportunity toexploit the hourly traffic oscillation via online VM shuffling.

We summarize the main contributions of this paper asfollows:

(1) We collect and make an in-depth analysis on thetraffic trace collected from a production datacenter.1

1 The name of the production cluster is anonymized for privacy concern.

Our analysis sheds light on the status quo of conges-tion and stability inside the datacenter, which moti-vates our design.

(2) We formulate the online VM placement problem,prove its NP-hardness, and propose an efficient VMplacement and shuffling algorithm for congestionmitigation with controllable migration traffic andlow time complexity.

(3) We formulate the VM migration scheduling prob-lem, and propose an efficient VM migration schedul-ing algorithm to schedule the migration of the VMpairs selected by the former algorithm.

(4) We conduct extensive evaluation with both realand synthetic traffic patterns to show theoptimization performance and algorithm overheadof VirtualKnotter.

The rest of the paper is organized as follows. In Sec-tion 2, we discuss related studies and background tech-niques. Next we present the measurement result in aproduction datacenter in Section 3. Then we describe theproblem formulation and the algorithm design in Section 4.In Section 5, we evaluate VirtualKnotter via extensive sta-tic and dynamic simulations respectively. We discuss prac-tical issues and limitations in Section 6 before concludingin Section 7.

2. Background and related work

2.1. Traffic engineering in datacenter

Traffic engineering techniques have been investigatedfor decades. In the context of Internet, traffic engineeringis usually performed by optimizing flow routing anddetouring traffic away from congested links, so that thetraffic is balanced and the maximal link utilization is min-imized [9–11]. Most of those sophisticated traffic engineer-ing techniques manipulate route via changing the linkweights and coupling with link state protocols like OSPFand ISIS. While they naturally fit ISP networks with nearlyrandom topologies and high-end routers, they may not begood options for datacenters with relatively regular topol-ogies and commodity switches, where people usuallydeploy simple spanning tree forwarding and ECMP [12].Furthermore, many recently-proposed datacenters suchas BCube [13], DCell [14], and PortLand [15], have well-defined topology and the routing is largely determinedby the base topology. Therefore, we cannot directly applythe existing traffic engineering techniques to these data-center scenarios.

Recently, several traffic engineering solutions havebeen proposed to deal with unbalanced link utilizationproblem in datacenters, such as Hedera [16] and MicroTE[17]. They both proposed to arrange the traffic in flowgranularity with global knowledge of the traffic load.While providing non-trivial advantages in dense structureswith rich path diversity, these approaches would havemarginal use when the network structure of datacenter isoversubscribed, and path diversity is limited. For instance,in a traditional tree-style network, simultaneous flows

S. Zou et al. / Computer Networks 67 (2014) 141–153 143

have to traverse the oversubscribed core or aggregationlinks if the sources and destinations do not locate underthe same top-of-rack switch, thus creating congestion. Inthis scenario, only by relocating the communication corre-spondents can we manage to mitigate the congestion onthe core and aggregation layers. At this point, our designin this paper complements the existing approaches espe-cially when the network is oversubscribed.

2.2. VM live migration and application

VM live migration was first proposed and implementedby Clark et al. [18], providing near continuous service dur-ing VM migration. They reported as short as few hundredsof milliseconds service downtime. Now, most of the popu-lar VM management platforms provide support for livemigration as a standard service, such as VMware vMotion[19], KVM [20], and Microsoft Hyper-V Server [21]. Livemigration technique delivers spatial mobility for VM place-ment strategy in datacenters, along with the cost of extramigration traffic, which could be 1.1� to 1.4� of VM mem-ory footprint, or 0.34� to 0.43� if adopting appropriatecompression [22]. With such extra mobility, VM placementoptimization is found effective on server consolidation,power consumption saving, fault tolerance, easier QoSmanagement and so on [5–8].

Recently, several studies leverage VM migration orplacement to optimize the network metrics like traffic costand end-to-end latency [23,24]. In [23], Shrivastava et al.proposed to rebalance workloads across physical machinesby shifting the VMs away from overloaded physicalmachines. The goal is to minimize the data center networktraffic while satisfying all of the server-side constraints.They particularly tried to minimize the overhead of VMmigration. In [24], Meng et al. proposed to minimize thetraffic cost, which is quantified in terms of traffic volumetimes the communication distance, via VM placement.They proposed a min-cut clustering-based heuristic algo-rithm whose runtime complexity is Oðn4Þ, where n is thenumber of VMs. Worse, their algorithm does not take intoaccount the VM migration traffic, leading to a near com-plete shuffling of almost all VMs in each round. Relativeto these works, VirtualKnotter minimize the continuouscongestion mainly in core and aggregation links with run-time overhead of Oðn2 log nÞ and controllable migrationtraffic, which enables online VM replacement at the gran-ularity of tens of minutes. In [25], Dias et al. proposed tocluster virtual machines by traffic matric among VMs withthe concept of graph community. However, they did nottake into account the cost of migration traffic. The paper[26] presents a system for network aware steady stateVM management that first select a VM heuristically andthen ranks target hosts for a VM migration based on theassociated cost of migration, available bandwidth formigration and the network bandwidth balance achievedby a migration. In this paper, we adopt the idea to swapVM pair in two clusters of a datacenter which simplifiesthe problem and can be closer to optimization. In [27],Wenjie Jiang et al. solved a joint tenant placement androute selection problem with Markov chain approximation.It assumes a dynamic multi-tenants data center where

tenants come and go frequently. In our work, we focuson internal production data centers. In [28], the authorspresent the design, implementation, and evaluation of asystem called XCo, that performs explicit coordination ofnetwork transmissions over a shared Ethernet fabric toproactively prevent Ethernet congestion. A central control-ler issues transmission directives to individual nodes atfine-grained timescales (every few milliseconds) to tempo-rally separate transmissions competing for bottlenecklinks. It focuses on solving the problem of Ethernet conges-tion, not from the view of the whole datacenter. In [29], theauthors proposed a distributed VM migration scheme tominimize the overall communication cost of the DC topol-ogy. Every VM individually tests the candidate servers andmigrate when the benefit overweighs the migration cost.However, we argue that in datacenter there are alreadyextensive central monitoring and management systemsin industry and VM migration can be figured out more effi-ciently with central control.

3. Measurement and motivation

Recent measurement results in datacenter illustrateseveral remarkable traffic properties in datacenternetwork.

� Congestion is ubiquitous in datacenter network. Specif-ically, it is reported not rare to see above 70% link utili-zation at a timescale of 100 s [1]. Such a high utilizationcan cause serious packet drop, significant queuing delayat the congested spots, and thus impacts overallthroughput. Those effects can degrade application per-formance with both large data transfer and smallrequest-response flows.� Datacenter network is frequently the bottleneck to

application-layer performance. For instance, Chowdhu-ry et al. show communication time account for 42–70%running time in MapReduce services [30]. This means adecrease of 10% in communication time will result in4.2–7% performance gain, which is hard to achieve byspeeding up computing.� Link utilization is highly divergent among core and

aggregation links within a datacenter. It is shown thatthe highly utilized hot links usually account for lessthan 10% of all the core and aggregation links, whileother links remain lightly utilized with utilization lessthan 1% [30,31]. Such phenomenon indicates the spa-tially unbalanced utilization of network resources maybe one of the causes of the network congestion. Thisimplies keeping a spatially balanced resource demandmay potentially have the same benefit as provisioningbandwidth capacity at hot spots.

However, despite those observations from previousmeasurement studies, some traffic properties like conges-tion pattern and long-term traffic stability still remainunclear. During our study, we learn that those propertiesare essentially helpful to design a traffic engineeringscheme as well as to determine the parameters for a spe-cific datacenter.

144 S. Zou et al. / Computer Networks 67 (2014) 141–153

In this section, we focus on obtaining quantitativeknowledge of congestion pattern (where and how longdoes congestion occur?) and traffic stability at variousgranularities (how stable is the traffic, viewing in the timescale of seconds, minutes or hours?). We collected trafficmatrices from a production cluster with 395 servers, whichrun MapReduce-like services. This cluster has a hierarchi-cal structure with 4:1 oversubscription ratio. We aggregatethe traffic matrices for every 30 s, as shown in Fig. 1. Thedataset lasts consecutively for about 18 h.

Our measurement reveals three key observations withimplications for the VirtualKnotter design. First, we find amajority of congestion events last for tens of minutes,while the set of congested links evolves over time. Thisobservation demonstrates the long-term communicationpattern of the upper-layer application, implying the poten-tial benefit to conduct traffic engineering at a timescale oftens of minutes. Second, congestion events tend to be local,usually involving less than 30% of links, which indicatestemporarily unbalanced traffic in the datacenter. Finally,we observe that over 60% traffic volume is relatively stableat an hourly granularity. This property allows for the pre-diction of future traffic matrix with the previous ones,which is the key assumption of traffic engineeringtechniques.

3.1. Congestion within datacenter

(a) Traffic Concentration: Fig. 1 shows a typical trafficmatrix in our dataset. It presents a busy trafficmatrix with severe local congestion, which involvessix out of ten racks within the cluster. We can seethe traffic is highly concentrated within and acrossthe upper and lower part of the cluster, which isshown by the four gray blocks in Fig. 1. In fact, withfurther inspection into the link utilization, we findthe links among those racks have an average utiliza-tion of 65% in the core and aggregation layer, withthe highest of 80.3%. In the meantime, links associ-ated with middle four racks remain relatively idle.

Fig. 1. An observed 30-s traffic matrix. Each data point is the trafficvolume from a sender (x axis) to a receiver (y axis). Gray scale reflectstraffic volume in natural log of bytes.

(b) Location and Duration of Congestion Events: To fur-ther understand the spatial distribution and tempo-ral duration of congested links, we locate thecongested links by plotting them into a time seriesfigure, as shown in Fig. 2. We pick 60% utilizationas the congestion threshold, but other thresholdslike 65% or 70% yields qualitatively similar results.We define the term congestion event as the periodof time when the set of congested links keeps thesame without discontinuity of longer than 5 min.Using this definition, we examine the congestionevents in our trace, resulting in two interesting find-ings. First, congestion events exist and tend to belocal during the observation period, with no conges-tion event involving more than half of links. Instead,a typical congestion event just involves about 1/3 ofcore and aggregation links. Moreover, different con-gestion events may consist of quite different sets ofcongested links. This phenomenon indicates thatthe application’s communication demand can dis-tribute highly unevenly within a datacenter, andthat the traffic distribution evolves over time. Sec-ond, a congestion event tends to last for an extendedperiod of time. We totally observe seven congestionevents that last for at least 20 min long. We furtherspeculate such a continuous congestion event mayindicate an application-layer transfer event, such asa MapReduce shuffle between mappers andreducers.

3.2. Traffic stability analysis

Although there are measurement results demonstratingpoor predictability of traffic matrix at the timescale of tensof milliseconds to hundreds of milliseconds [31,32], westill have little knowledge about long-term traffic stabilityin datacenters. In this subsection, we design a stablenessindicator and conduct measurement in the dataset. For-mally, the stableness indicator is defined by

Stablenessðtprev ; tcurrÞ ¼minðtprev ; tcurrÞmaxðtprev ; tcurrÞ

; ð1Þ

where tprev and tcurr stand for traffic volume in the previousepoch and the current epoch respectively. The fundamen-tal idea for stableness indicator is to estimate percentagethe stable part comparing two consecutive traffic states.For a traffic matrix, we calculate a single stableness value

0 2 4 6 8 10 12 14 16 18

05

1015202530

Time (hours)

Aggr

egat

ion

and

Cor

e Li

nk In

dex

Fig. 2. Time and locations of congestion observed in the datacenter. Eachcircle represents a 30-s congestion occurred on a core or aggregation link.

S. Zou et al. / Computer Networks 67 (2014) 141–153 145

for each element. Then, we select 10% percentile, median,and 90% percentile as the indicator of the entire distribu-tion, as shown in Fig. 3(a). Similar procedure is used togenerate Fig. 3(b).

We can simply explain the stableness indicator as thepercentage of stable traffic volume in two consecutiveepochs. Fig. 3(a) illustrates the stableness of pairwise traf-fic varying the timescales from 30 to 4 h. We can see arange of 40% to 70% of traffic volume can be expected sta-ble, peaking at the timescale of 2 h, demonstrating a goodhourly predictability of the traffic matrix in our dataset.Although the traffic stability varies greatly with shorttimescales, the hourly traffic factor tends to concentrateon about 60% with small oscillation. Furthermore,Fig. 3(b) demonstrates even better stability, with the med-ian stable traffic indicator larger than 90%. The better sta-bleness of upper-layer links is a result of trafficaggregation as well as the constantly higher utilization.

Our experiment results reveal highly stable trafficdemand at an hourly granularity in a datacenter. Actually,such high stable trace is not obtained by chance. Data-intensive applications tend to exhibit a similar stability,due to the massive nature of data transfer and long com-puting time on distributed computing node. We will dis-cuss the traffic stability issue later in Section 6.

The above findings motivate us towards an online VMplacement approach for congestion resolving, as the exist-ing of continuous congestion events, evolving congestionpatterns and good traffic predictability. We argue a consid-erable part of continuous link congestion can be elimi-nated, or at least mitigated, by localizing intra-datacentertraffic and evenly distributing outgoing traffic. By exploit-ing the spatial mobility of VM placement in a datacenter,we can potentially achieve both localized and balancedcommunication pattern, and thus benefiting from higherthroughput and lower queuing latency.

24012060301584210.50

0.2

0.4

0.6

0.8

1

Time Granularity (in minutes)

Stab

lene

ss

(a) Stableness indicator of pairwise traffic volume

0.5 1 2 4 8 15 30 60 120 2400

0.2

0.4

0.6

0.8

1

Time Granularity (in minutes)

Stab

lene

ss

(b) Stableness indicator of core and aggregates links

Fig. 3. The fraction of traffic volume remains stable in two successiveperiods. The bar value shows the median and error bar shows 10% and90% percentile.

4. Design

In this section, we first formulate the online VM place-ment problem using integer optimization language andanalyze its complexity. Then, we formulate the VM migra-tion scheduling problem with combinational optimiza-tion. At last, we propose VirtualKnotter, which iscomposed of two corresponding algorithms to solvethe above problems.

4.1. Assumptions

We assume the datacenter is connected with a hierar-chical structure, such as a tree or multi-root tree. Note thatwe aim to address congestion problem, which theoreticallydoes not exist in non-blocking network, such as fat tree orVL2. Thus, we exclude those network structures from thescope of our study. A server’s ability to host VM is con-strained by the server’s physical capacity, such as CPU/memory. Thus, we assume a known number of VM hs canbe hosted on a certain server s, referring as VM slots. Wefurther assume a known single-path routing in the data-center network. In addition, we assume live migration[18] is used to provide near continuous service duringVM migration.

4.2. Online VM placement problem

According to previous discussion, we want to achievefollowing goals in the online VM placement scheme:

(1) The optimized VM placement should minimize thecongestion status measured by a link congestionobjective, such as maximum link utilization.

(2) The migration traffic should be controllable, i.e.,parameters should be provided to control the num-ber of migrated VMs between the current VM place-ment and the optimized VM placement.

(3) The algorithm should be scalable, i.e., the runtimeoverhead should be considerably less than the targetreplacement timescale (tens of minutes) for a typicalsized datacenter.

Given above principles, we formulate the online VMplacement problem as follows.

Input and Output. The online VM placement problemaccepts network routing P, traffic matrix M, external trafficE and current VM placement X 0 as input, and generatesoptimized VM placement X as output. We denote the net-work routing by a binary-value function Ps;dðlÞ, meaningwhether the traffic path from server s to d traversesthrough link l. Similar notation PsðlÞ represents the routingpath going outside the datacenter, meaning whether thetraffic path from server s to the gateway traverses throughlink l. As the datacenter running, we assume the trafficmatrix Mi;j and external traffic Ei for a certain period oftime are also available. Mi;j denotes the traffic volume fromVM i to VM j. Ei denotes externally traffic volume from VM ito the gateway. Note, such traffic statistics can be collectedeither by ToR switches or VM hypervisors on each server

146 S. Zou et al. / Computer Networks 67 (2014) 141–153

without incurring considerable overhead. Moreover, theproblem also takes the current VM placement matrix X 0

as input for incremental VM placement. The output is theoptimized VM placement matrix X. Both Xi;s and X0i;s arebinary-value matrix indicating whether VM i is placed onserver s.

Objective. We choose the maximum link utilization(MLU) as the optimization objective. The MLU is deter-mined by the highest utilized link, which characterizesthe worst congestion status in a network during a periodof time. Given the MLU is widely adopted as the optimiza-tion goal in the context of Internet traffic engineering[9–11], we believe it will also be effective to representthe overall congestion status in a datacenter network. Withthe preceding notations, we formally define the followingobjective function

minX

maxl

Tðl;XÞBðlÞ ; ð2Þ

s:t:

Tðl;XÞ ¼Xs;d

Ps;dðlÞX

i;j

XTi;sMi;jXj;d þ

Xs

PsðlÞEiXi;s

Xj

Xi;j ¼ 1

Xi

Xi;j 6 hj

12

Xi;j

jXi;j � X0i;jj 6 Th

where s (or s; d pair) enumerates all hosts (or host pairs)and i; j pair enumerate all VM pairs, X is a valid VM place-ment matrix, l is a physical link between two switches, BðlÞis the bandwidth of link l. Tðl;XÞ is the traffic traversingthrough link l under the placement of X. The second con-straint means that for one VM can only be placed in oneserver. The third constraint limits the number of VMs tobe placed in one server. The last constraint limits themigrated VM number to be no greater than an inputthreshold Th and so it is feasible to schedule the VM migra-tion in time.

In the objective function, two parts in Tðl;XÞ representrespectively the internal traffic and external traffic travers-ing a given link l. The inner maximum operator enumer-ates all links, so as to seek for the MLU. The outerminimum operator finds the lowest MLU among all validVM placements. Therefore, this objective function is tominimize the MLU.

Complexity. The above optimization problem falls intothe category of Quadratic Bottleneck Assignment Problem(QBAP), which is a known NP-hard problem [33]. It is con-ceivable that the variables Xi;s have quadratic form in theobjective function and the overall problem is an assign-ment problem with a quadratic bottleneck goal (minimumof maximum). We formally give the complexity proof byreducing the QBAP problem to our problem in the Appen-dix. Hence, our problem is also NP-hard.

4.3. VM migration scheduling problem

Input and Output. The VM migration scheduling prob-lem accepts the output of the online VM placement prob-lem as input, and generates optimized VM migrationscheduling order fxig as output. We denote the cost of apair of VM migration to be Ci, which is a little bit more thanthe total of both VMs’ memory footprint. Let Gi representthe bandwidth gain after the migration of ith pair ofVMs. In addition, we assume that at the beginning, thereare A bandwidth available for doing the first swap of VMpair.

Objective. We choose the total time to execute VMmigration as the optimization objective. Hence, we havethe following objective function:

minfxig

Xn

i¼1

txi ð3Þ

s:t:

tx1 ¼ Cx1=A

tx2 ¼ Cx2=ðAþ Gx1Þ. . .

txi ¼ Cxi=ðAþXi�1

k¼1

GxkÞ

where txi denotes the time needed to swap the ith VM pair.The bandwidth can be used to swap the ith VM pair is com-posed of the initial available bandwidth A and the accumu-

lated bandwidth gainPi�1

k¼1Gxk.

4.4. Algorithms

In this subsection, we propose VirtualKnotter, whichincludes a heuristic algorithm to the online VM placementproblem and an efficient algorithm to the VM migrationscheduling problem. We have shown our problems areinherently NP-hard; and no efficient exact solution canscale to the size of a typical datacenter. Therefore, weresort to an intuitive heuristic approach.

Intuition 1: Incremental local search is preferredrather than clustering, in order to satisfy migrationtraffic constraint. So far as our knowledge goes, thereis no clustering algorithm that is able to balance the opti-mality and number of elements that move across clusters.However, for local search, it is inherently easy to keeptrack of the searching depth, which is equivalent tomigration traffic in our case.

Intuition 2: A good initial placement is needed tospeed up local search. Searching usually takes long anduncertain amount of time, which we do not want inonline algorithm. A possible way to speed up searchingis to provide a good estimation as the initial solution.We find that an efficient algorithm that improvestraffic localization may satisfy our requirements.Although not exactly the same, traffic localization sharessimilar objective with the MLU goal. This is because

Fig. 4. Algorithm flowchart of VirtualKnotter. The diamond boxes are input/output data, and the rectangle boxes are functions.

S. Zou et al. / Computer Networks 67 (2014) 141–153 147

congestion often occurs at core and aggregation links,where the localization algorithm aims to offload trafficfrom.

Based on the above intuitions, we propose a two-stepheuristic algorithm: we borrow and adapt the multi-wayKernighan–Lin graph partitioning algorithm to generateinitial VM placement with better localized traffic [34,35].Then, we employ the simulated annealing algorithm to fur-ther optimize the MLU. Fig. 4 shows the high-level logicflow of the algorithms. And we present the pseudo-codein Algorithms 1–4.

4.4.1. Multiway h-Kernighan–Lin AlgorithmThe key idea of the Kernighan–Lin graph partitioning

algorithm is to greedily swap elements across clusters,thereby iteratively reduce the overall weight of a graphcut. We adapt the algorithm by introducing a migrationcoefficient h, in order to constrain the migration cost ofimproved VM placement. This heuristic algorithm is orig-inally used in the layout design of circuits and compo-nents in VLSI [36], where an efficient heuristic solutionfor the minimum graph cut problem is needed. In our sce-nario, we adapt the Kernighan–Lin algorithm for the pur-pose of improving the traffic localization and reduce thetraffic load on core and aggregation layers. The algorithmruns on the original VM placement hierarchically in a top-down manner. In each layer, it bisects the VM clustersand calls h-Kernighan–Lin-Improve procedure for bisec-tion improvement. The procedure swaps elementsbetween two clusters iteratively and greedily accordingto the Gain on the cut weight reduction. Note that thenumber of iterations is limited by migration coefficienth, so as to avoid VM swaps which only bring marginalbenefit. The runtime complexity of Multiway h-Kerni-ghan–Lin Algorithm is Oðn2 log nÞ, where n is the numberof VMs.

Algorithm 1. Multiway h-Kernighan–Lin Procedure

Require: M(Traffic matrix), T(Network topology),X(Current VM placement)for all layer in T do

Sort element set E 2 X on layer by outgoing trafficwhile jEj � 2 do

Split elements into two half interleavingly,resulting in S1 and S2

h-Kernighan–Lin-Improve (M; S1; S2)E S1 and S2 respectively

end whileend forreturn X

Algorithm 2. h-Kernighan–Lin-improve

Require: M(Traffic matrix), S1; S2(VM sets),h(Migration coefficient),C(Migration traffic, which isset as 1.25 times memory footprint), T(Period lengthbetween two shuffling)

CurrGain 0;Gain femptylistgInitialize the migration gain DðiÞ,where DðiÞ ¼

PjRSðiÞMði; jÞ �

Pj2SðiÞMði; jÞ

for s ¼ 1 to 12 h �minðlenðS1Þ; lenðS2ÞÞ do

Swap the VM pair ði; jÞ 2 ðS1; S2Þ, which hasmaximum gainGði; jÞ ¼ DðiÞ þ DðjÞ � 2 �M½i; j� � ðCi þ CjÞ=TCurrGain ¼ CurrGainþ Gði; jÞGain:appendðCurrGainÞUpdate D : DðkÞ ¼ DðkÞ þMðk; jÞ �Mðk; iÞ, if k 2 S1

DðkÞ ¼ DðkÞ þMðk; iÞ �Mðk; jÞ, if k 2 S2

end forreturn maxðGainÞ and corresponding VM sets S01; S

02

Networks 67 (2014) 141–153

4.4.2. Simulated annealing searching

Algorithm 3. Simulated annealing searching

148 S. Zou et al. / Computer

Require: M(Traffic matrix), P(Network routing),X0(Current VM placement), Nmax(Max iterations),h(Migration coefficient)

X;Xbest X0

E; Ebest EnergyðM; P;XÞfor T Nmax to 0 do

Xnew NeighborðXÞEnew EnergyðM; P;XnewÞif PðE; Enew; TÞ > RandðÞ and Diff ðX;X0Þ < hthen

X Xnew; E Enew

end ifif E < Ebest then

Xbest X; Ebest Eend if

end forreturn Xbest

In this step, we need to efficiently search for a fine-grainsolution of minimizing MLU. We employ the simulatedannealing algorithm, which is known efficient in searchingin an immense solution space. The initial VM placementaccepts as input the output of the multiway h-Kerni-ghan–Lin algorithm. The function Energy estimates andreturns the MLU for a given VM placement. In each itera-tion, a neighboring state NeighborðXÞ is generated by swap-ping a VM pair that can offload traffic from the mostcongested link. To find such a VM pair, we conceive a heu-ristic: we seek for two distinct and heavily communicatedpairs over the congested link, pick one VM from each pair,and swap them. Then, we move to a neighboring state witha certain acceptance probability P, which depends on theenergy of current and neighboring placement as well ascurrent temperature T. The acceptance probability we useis defined as

PðE; Enew; TÞ ¼1 if Enew < E

ecðE�EnewÞ=T if Enew P E

�

The temperature is decreased with each iteration untilstopped at zero, allowing a higher probability to move toa worse placement with a high temperature. This behaviorallows simulated annealing algorithm to avoid stuck at thelocal minima. The complexity of the simulated annealinghas two components: the initialization requires Oðn2Þ;each simulated annealing iteration requires OðnÞ. Thus,the overall complexity is Oðn2 þ Nmax � nÞ, where Nmax ismaximum number of iterations.

4.4.3. Simulated annealing schedulingIn this step, we need to efficiently search for an opti-

mized scheduling order for the n pairs of VMs to be swa-ped, which has the least scheduling time. Once again, weemploy the simulated annealing algorithm. The initialscheduling order is set according to the descending order

of Gi=Ci, which is straightforward to schedule those VMpairs with more bandwidth gain and less migration trafficfirst. The function Energy estimates and returns the totalscheduling time for the VM migration. In each iteration, aneighboring state NeighborðXÞ is generated by randomlyswap two items. The complexity of the algorithm hastwo components: the initialization requires Oðn2Þ; eachsimulated annealing iteration requires OðnÞ. Thus, theoverall complexity is Oðn2 þ Nmax � nÞ, where Nmax is maxi-mum number of iterations.

Algorithm 4. Simulated annealing scheduling

Require: C (VM migration traffic vector), G(bandwidth gain vector), A (Initial availablebandwidth),Nmax(Max iterations)

X;Xbest X0;X0 is a permutation of {1, . . . ,n} bydescending Gi=Ci

E; Ebest EnergyðC;G;XÞfor T Nmax to 0 do

Xnew NeighborðXÞEnew EnergyðC;G;XnewÞif PðE; Enew; TÞ > RandðÞ then

X Xnew; E Enew

end ifif E < Ebest then

Xbest X; Ebest Eend if

end forreturn Xbest

5. Evaluation

In this section, we describe our evaluation of VirtualKn-otter in three aspects: static performance, overhead anddynamic performance. The goal of these experiments isto determine the benefit as well as the cost whendeploying VirtualKnotter and the baseline algorithms.

5.1. Methodology

5.1.1. Baseline algorithmsWe compare VirtualKnotter with a clustering-based

placement algorithm. The advantage of clustering algo-rithm lies in the fact that clustering produces near optimaltraffic localization. However, to the best of our knowledge,clustering algorithm cannot be trivially adapted to performan incremental optimization, which implies nearly 100% ofVMs need to be migrated in each round. Also, clusteringalgorithms usually have a runtime complexity no less thanOðn3Þ, where n is the number of VM number. Thus, in ourevaluation, we treat the clustering algorithms as a refer-ence of the optimization performance without limit onruntime and migration traffic. Among many available clus-tering algorithms, we select a variation of hierarchical clus-tering algorithm, which is able to specify the cluster sizeconstraint [37]. We run the clustering algorithm followinga top-down order according to the network topology. Welater map each cluster into a switch and VM into a physicalmachine. The runtime complexity of the algorithm is Oðn3Þ.

0.4 0.5 0.6 0.7 0.8 0.850

0.2

0.4

0.6

0.8

1

Maximum Link Utilization

CD

F

OriginalVirtualKnotterKLSAClustering

(a) CDF of MLU on Real-world Traces (VM# )

0.4 0.5 0.6 0.7 0.8 0.850

0.2

0.4

0.6

0.8

1

Maximum Link Utilization

CD

FOriginalVirtualKnotterKLSAClustering

(b) CDF of MLU on Measurement-based Patterns (VM#=10K)

0.4 0.5 0.6 0.7 0.80

0.2

0.4

0.6

0.8

1

Maximum Link Utilization

CD

F

OriginalVirtualKnotterKLSAClustering

(c) CDF of MLU on Hotspot Patterns (VM#=10K)

Fig. 5. Static algorithm performance in terms of maximum linkutilization.

S. Zou et al. / Computer Networks 67 (2014) 141–153 149

The detailed description is referred to the original paper[37].

We also select each single step of VirtualKnotter,namely the Multiway h-Kernighan–Lin algorithm (KL)and the simulated annealing algorithm (SA) as baselinealgorithms. The purpose is to show how the combinationof algorithms actually benefits compared with individualsteps.

5.1.2. Communication suites

� Real-world Traces: We use the collected traffic tracedescribed in Section 3. The traffic trace is collected froma production cluster with 395 servers and an oversub-scription ratio of 4:1 in the core layer. The collecteddata has a time granularity of 30 s, and lasts for nearly18 h.� Measurement-based Patterns: In order to test the

scalability of the algorithm, we derive the measure-ment-based patterns from the measurement results byKandula et al. [1]. First, we derive host communicationpattern from both the inter-rack and intra-rack corre-spondent distribution. Then, we assign traffic volumeto each pair of hosts, according to the traffic volumedistribution. The VM number is 10 K, and the physicaltopology is assumed hierarchical with an oversubscrip-tion ratio of 10:1.� Hotspot Patterns: Recent measurement revealed

highly skewed traffic patterns often exist in productiondatacenter, known as hotspot pattern [31,38]. We syn-thesize such traffic pattern by randomly select ToRswitches as hotspots, connect the individual serversunder hotspots with a number of servers under normalToRs, and assign a constant large traffic volume to eachconnection. The VM number is 10 K, and the physicaltopology is assumed hierarchical with an oversubscrip-tion ratio of 10:1.

5.1.3. Metrics and settingsFirst, we evaluate the static algorithm performance by

measuring the maximum link utilization. We compare Vir-tualKnotter against KL, SA and clustering algorithm, as wellas the original VM placement without any optimization.Then, we evaluate the algorithm overhead in the sense ofadditional migration traffic, migration time and algorithmruntime. Finally, we simulate the real scenario and evalu-ate the overall algorithm performance on dynamic conges-tion resolving. We replay the time-series traffic and run thealgorithm on current traffic pattern, resulting in an opti-mized VM placement. Then, we apply the optimized VMplacement on the next traffic pattern, and schedule VMmigration according to the order produced by Algorithm4. We compare the number of congested links varyingthe replacement timescale. Note, the migration coefficienth is set to 0.1 in both KL and SA, the maximum number ofiterations Nmax is set to 1000 in SA.

5.2. Static performance

Fig. 5 shows the maximum link utilization before andafter applying the algorithms. Every data point represents

a communication pattern of a collected or synthetic trafficmatrix. We try to minimize the maximum link utilization;thus the curve close to the upper left corner is favorable.From the figures, we find VirtualKnotter significantly out-performs both KL and SA, and has a similar static perfor-mance as Clustering algorithm, which serves as areference to the upper bound. This result illustrates thatthrough combination VirtualKnotter provides qualitativeimprovement over both KL and SA. It is worth to note thatVirtualKnotter requires significantly less VM migrationcompared with Clustering-based algorithm (5–10% vs.�100%), which will be analyzed in detail in the followingsubsection.

Fig. 7. Total time for executing VM migration.

500 1K 2K 4K 8K 16K0

50

100

150

200

250

300

350

Size of Datacenter

Run

time

(in s

econ

ds)

VirtualKnotterClustering

Fig. 8. Algorithm runtime overhead varying the size of a datacenter.

150 S. Zou et al. / Computer Networks 67 (2014) 141–153

5.3. Overhead

VM migration introduces considerable bulk data trans-fer into the datacenter network. To understand the coun-ter-effect, we need to quantitatively measure how largevolume of the additional migration traffic we shouldexpect for each algorithm and how long it takes to do themigration. We model the VM migration as bulk data trans-fer. We assume each VM has a memory footprint rangedfrom 1 to 8 gigabytes. For a VM with 2 gigabtyes, it willresult in 2.2–2.8 gigabytes bulk transfer using live migra-tion [39]. Thus, we take the median 1.25 times memoryfootprint as the extra traffic volume for each migratedVM in the simulation. .

5.3.1. Migration trafficWe plot the relative traffic volume for both VirtualKnot-

ter and the baseline algorithm in Fig. 6. From the figure, weobserve that the baseline algorithm introduces around 10%traffic volume of goodput with the timescale of 30 min. Thetraffic overhead of VirtualKnotter is over one order of mag-nitude less than Clustering algorithm, which rangesbetween 0.2% and 1% of goodput.

5.3.2. Migration timeWe implement a straightforward VM migration sched-

uling algorithm named Gain, which simply schedule theVMs pair with most bandwidth gain first. The initial avail-able bandwidth is set as 200Mbit/s, and the bandwidthgain after each swap of VMs pair is set from 10 Mbit/s to200 Mbit/s. The memory footprint of VM and the band-width gain are randomly chosen in each trial run and anaverage of 100 trial runs is used for a data point in Fig. 7.From it we can see that VirtualKnotter uses almost 30% lesstime than Gain.

5.3.3. Algorithm runtimeThe runtime overhead of VirtualKnotter is shown in

Fig. 8. It is evident from the figure, that VirtualKnotter con-sumes tens of seconds for a typical virtual datacenter ortenant with thousands of VMs, and scales much better thanthe baseline algorithm. Also the runtime overhead of Vitu-alKnotter is considerably less than the target replacement

0 5 10 15 20 25 300.001

0.01

0.1

1

Time (hours)

Rel

ativ

e M

igra

tion

Volu

me

(vs.

Goo

dput

)

Goodput VolumeVirtualKnotterClustering

Fig. 6. Relative migration traffic compared with network goodput. Thetimescale is 30 min.

granularity which is tens of minutes, thereby enablingthe online VM replacement.

5.4. Dynamic simulation

We conduct simulation to evaluate the dynamic perfor-mance of both VirtualKnotter and the baseline algorithmconsidering migration traffic. We replay the real-worldtraces and run both algorithms in a variety of timescales.The optimized VM placement resulted from previous per-iod of time is deployed on the next period with the VMmigration scheduling algorithm.

8 15 30 60 1200

1500

3000

4500

5500

Migration Timescale (in minutes)

Link

Con

gest

ion

Tim

e (li

nk*m

inut

e)

OriginalVirtualKnotterClustering

Fig. 9. Link congestion time varying replacement timescale. Congestionthreshold is 0.6. All the congested time of core and aggregation links areincluded.

0 2 4 6 8 10 12 14 16 18

0

5

10

15

20

25

30

Time (hours)Ag

greg

ate

and

Cor

e Li

nk In

dex

(a) Congestion timeseries before VirtualKnotter

0 2 4 6 8 10 12 14 16 18

0

5

10

15

20

25

30

Time (hours)

Aggr

egat

e an

d C

ore

Link

Inde

x

(b) Congestion timeseries after VirtualKnotter

Fig. 10. Algorithm performance in term of congestion timeseries. Each circle represents a 30-s congestion occurred on a core or aggregation link.

S. Zou et al. / Computer Networks 67 (2014) 141–153 151

The migration traffic is modeled exactly the same as inSection 5.3. Fig. 9 shows the total congestion time of allcore and aggregation links varying replacement granular-ity. The figure demonstrates that, even considering migra-tion overhead, VirtualKnotter still manages to harvest thebenefit of over one half less link congestion time at time-scales of 30 min, one hour or two hours. On the contrary,the baseline algorithm, due to the migration traffic, exhib-its a worse congestion status than original placement, witha 7.8–230.6% higher link congestion time compared withoriginal placement.

Fig. 10 presents the locations and durations of conges-tion before and after applying VirtualKnotter. From the fig-ure, we can see that most of the continuous congestionevents are resolved by VM replacement right afterdetected, demonstrating VirtualKnotter is effective onresolving long-lived congestion event. The benefit is har-vested at the cost of dispersed short-lived congestionevents caused by the burst of migration traffic. Such con-gestion events are normally aligned with VM replacementevents, and last for only one to two minutes. We believesuch ephemeral congestion events are more tolerable byapplications in datacenter and yield far less negativeeffects compared with the continuous congestion events.

6. Discussion

6.1. Traffic stability

VirtualKnotter manifests good performance on resolv-ing static congestion with VM placement shuffling. How-ever, in a real-world setting, we have to predict the futuretraffic pattern using history records; thereby the traffic sta-bility may have considerable influence on the accuracy ofprediction. We admit that VirtualKnotter is not suitablefor datacenters with highly dynamic traffic. Although we

observe an average proportion of 40–70% of traffic remainsstable within our target timescales, we understand the traf-fic properties highly depend on the application layer. Weargue that data-intensive applications would most likelyexhibit a similar stability with our measurement resultdue to the massive nature of data pre-fetch, intermediateresult shuffle and result transfer. For example, the distrib-uted indexing time for a search engine is estimated over10,000 s and 4000 s in the map phase and the reduce phaserespectively by Mccreadie et al. [40]. During the whole per-iod, network can suffer from high utilization with a contin-uous traffic pattern. Logistic regression, as a representativemachine learning algorithm, involves tens of MapReduceiterations, which lasts for an hour to train the model on a29 GB dataset [41]. Again, traffic pattern among iterationsis not likely to change greatly. Therefore, we believe thereare a part of data-intensive applications that will exhibit asimilar long-term stability in datacenters.

6.2. One-time placement scheduling vs. dynamic replacement

We propose a dynamic VM replacement scheme in thispaper. For those datacenter whose traffic pattern evolvesover time, we have shown that dynamic replacementscheme can effectively harvest from the traffic oscillation.However, we notice that there exist applications with rel-atively constant traffic patterns. For those applications, it isprobably sufficient to conduct one-time placement sched-uling rather than dynamic replacement. We leave the iden-tification of such constant traffic pattern anddetermination of best replacement timescale as our futurework.

7. Conclusion

In this paper, we present VirtualKnotter, a novel onlineVM placement algorithm and an efficient VM migration

152 S. Zou et al. / Computer Networks 67 (2014) 141–153

scheduling algorithm for resolving link congestion in data-center network. VirtualKnotter exploits the flexibility ofVM placement in virtualized datacenter, for the purposeof improving traffic localization and balance link utiliza-tion. VirtualKnotter strikes a good balance between resolv-ing congestion and introducing migration traffic, resultingin an efficient and practical VM placement algorithm. Inaddition, VirtualKnotter efficiently schedules VM migra-tion to shorten the migration time and avoid extra conges-tion. We evaluate VirtualKnotter via static experimentsand extensive dynamic simulation. Our results suggest thatVirtualKnotter delivers over 50% of congested time reduc-tion with negligible overhead.

Acknowledgements

This work was partially supported by National High-tech R&D Program of China (863 Program) (No.2011AA01A101, No. 2013AA013303).

Appendix A. Proof of problem complexity

We prove the complexity of the online VM placementproblem (OVMPP) by reducing quadratic bottleneckassignment problem (QBAP) to OVMPP [33].

The QBAP problem is formally described as following:given three n-by-n matrices A ¼ ðaijÞ;B ¼ ðbklÞ andC ¼ ðcikÞ with non-negative real values. One wants to findthe optimal permutation /ðiÞ, which satisfies a one-to-one mapping from N to N (N ¼ f1;2; . . . ;ng). The objectivefunction is

min/

max16i;j6n

aijb/ðiÞ/ðjÞ þ ci/ðiÞ:

Now, we want to map objective Function 2 to the abovegoal, to show for any QBAP instance we can transform it toan OVMPP in polynomial time. In fact, we can rewriteassignment matrix X to permutation /ðiÞ by assign differ-ent index for each VM slot. We further assume a full-meshed network, where each source destination pairðs; dÞ uniquely connected by a physical link lðs; dÞ. Thus,we have the following reduction mapping

aij ¼X

i;j

Pi;jðlði; jÞÞ

¼ Pi;jðlði; jÞÞ; ðsince ðs;dÞ maps to a unique linkÞb/ðiÞ/ðjÞ ¼

X/ðiÞ;/ðjÞ

XT/ðiÞ;iM/ðiÞ;/ðjÞX/ðjÞ;j

ci/ðiÞ ¼X

i

PiðlðiÞÞE/ðiÞX/ðiÞ;i:

The above mapping enables us to transform any QBAPto an OVMPP, implying QBAP is no harder than OVMPP.Since QBAP is known NP-hard, we conclude the OVMPPis also NP-hard.

References

[1] S. Kandula, S. Sengupta, A. Greenberg, P. Patel, R. Chaiken, The natureof data center traffic: measurements & snalysis, in: IMC ’09.

[2] J. Dean, S. Ghemawat, Mapreduce: simplified data processing onlarge clusters, Commun. ACM (2008) 107–113.

[3] S. Ghemawat, H. Gobioff, S.-T. Leung, The google file system, in: SOSP’03.

[4] M. Isard, M. Budiu, Y. Yu, A. Birrell, D. Fetterly, Dryad: distributeddata-parallel programs from sequential building blocks, SIGOPSOper. Syst. Rev. (2007) 59–72.

[5] R. Nathuji, K. Schwan, Virtualpower: Coordinated powermanagement in virtualized enterprise systems, in: SOSP ’07.

[6] N. Bobroff, A. Kochut, K. Beaty, Dynamic placement of virtualmachines for managing SLA violations, in: Integrated NetworkManagement, 2007.

[7] A.B. Nagarajan, F. Mueller, C. Engelmann, S.L. Scott, Proactive faulttolerance for hpc with xen virtualization, in: ICS ’07.

[8] B. Li, J. Li, J. Huai, T. Wo, Q. Li, L. Zhong, Enacloud: an energy-savingapplication live placement approach for cloud computingenvironments, in: CLOUD ’09.

[9] H. Wang, H. Xie, L. Qiu, Y.R. Yang, Y. Zhang, A. Greenberg, Cope:traffic engineering in dynamic networks, in: SIGCOMM ’06.

[10] D. Awduche, A. Chiu, A. Elwalid, I. Widjaja, X. Xiao, Overview andPrinciples of Internet Traffic Engineering. RFC Editor, 2002.

[11] D. Applegate, E. Cohen, Making intra-domain routing robust tochanging and uncertain traffic demands: understandingfundamental tradeoffs, in: SIGCOMM ’03.

[12] A. Dixit, P. Prakash, R. Rao Kompella, On the efficacy of fine-grainedtraffic splitting protocols in data center networks, in: SIGCOMM ’11.

[13] C. Guo, G. Lu, D. Li, H. Wu, X. Zhang, Y. Shi, C. Tian, Y. Zhang, S. Lu,BCube: A high performance, server-centric network architecture formodular data centers, in: SIGCOMM, 2009.

[14] C. Guo, H. Wu, K. Tan, L. Shi, Y. Zhang, S. Lu, DCell: a scalable andfault tolerant network structure for data centers, in: SIGCOMM,2008.

[15] R.N. Mysore, A. Pamboris, N. Farrington, N. Huang, P. Miri, S.Radhakrishnan, V. Subramanya, A. Vahdat, PortLand: a scalablefault-tolerant layer 2 data center network fabric, in: SIGCOMM,2009.

[16] M. Al-fares, S. Radhakrishnan, B. Raghavan, N. Huang, A. Vahdat,Hedera: dynamic flow scheduling for data center networks, in: NSDI’10.

[17] T. Benson, A. Anand, A. Akella, M. Zhang, The case for fine-grainedtraffic engineering in data centers, in: INM/WREN ’10.

[18] C. Clark, K. Fraser, S. Hand, J.G. Hansen, E. Jul, C. Limpach, I. Pratt, A.Warfield, Live migration of virtual machines, in: NSDI ’05.

[19] VMware, Vmware Vmotion for Live Migration of Virtual Machines,2011, November. <http://www.vmware.com/products/vmotion/overview.html>.

[20] KVM, Kvm Migration, 2011, November. <http://www.linux-kvm.org/page/Migration>.

[21] Microsoft, Hyper-v Live Migration, 2011, November. <http://technet.microsoft.com/en-us/library/dd446679>(WS.10).aspx.

[22] H. Jin, L. Deng, S. Wu, X. Shi, X. Pan, Live virtual machine migrationwith adaptive, memory compression, in: 2009 IEEE InternationalConference on Cluster Computing and Workshops, 2009, pp. 1–10.

[23] V. Shrivastava, P. Zerfos, K.-W. Lee, H. Jamjoom, Y.-H. Liu, S. Banerjee,Application-aware virtual machine migration in data centers, in:INFOCOM ’11.

[24] X. Meng, V. Pappas, L. Zhang, Improving the scalability of data centernetworks with traffic-aware virtual machine placement, in:INFOCOM ’10.

[25] D.S.C. Dias, L.H. MK, Online traffic-aware virtual machine placementin data center networks, in: Global Information Infrastructure andNetworking Symposium (GIIS), 2012, IEEE, 2012, pp. 1–8.

[26] A.G. Vijay Mann, P. Dutta, A. Vishnoi, P. Bhattacharya, R. Poddar, A.Iyer, Remedy: Network-aware steady state vm management for datacenters, in: Networking, 2012 Proceedings IFIP, 2012.

[27] M.C. Wenjie Jiang, T. Lan, S. Ha, M. Chiang, Dynamic vm placementand routing in a data center network, in: MiniINFOCOM, 2012Proceedings IEEE, 2012.

[28] J.A. Rajanna Vijay, S. Smit, G. Kartik, Explicit coordination to preventcongestion in data center networks, Cluster Comput. 15 (2012) 183–200. 10.1007/s10586-011-0156-9, <http://dx.doi.org/10.1007/s10586-011-0156-9>.

[29] G.H. Fung Po Tso, K. Oikonomou, D.P. Pezaros, Implementingscalable, network-aware virtual machine migration for cloud datacenters, in: CLOUD, 2013 Proceedings IEEE, 2013.

[30] M. Chowdhury, M. Zaharia, J. Ma, M. Jordan, I. Stoica, Managing datatransfers in computer clusters with orchestra, in: SIGCOMM ’11.

[31] T. Benson, A. Akella, D.A. Maltz, Network traffic characteristics ofdata centers in the wild, in: IMC ’10.

S. Zou et al. / Computer Networks 67 (2014) 141–153 153

[32] A. Greenberg, N. Jain, S. Kandula, C. Kim, P. Lahiri, D. Maltz, P. Patel, S.Sengupta, VL2: a scalable and flexible data center network, in:SIGCOMM, 2009.

[33] R.E. Burkard, Selected topics on assignment problems, Discrete Appl.Math. 123 (1-3) (2002) 257–302.

[34] B.W. Kernighan, S. Lin, An efficient heuristic procedure forpartitioning graphs, Bell Syst. Technical J. 49 (1970) 291.

[35] M.W. Sung, S. Kyu, L. Jason, C.M. Sarrafzadeh, Multi-way partitioningusing bi-partition heuristics, in: ASPDAC, 2000.

[36] C.P. Ravikumar, Parallel Methods for VLSI Layout Design, GreenwoodPublishing Group Inc., Westport, CT, USA, 1995.

[37] C. Pluempitiwiriyawej, A New Hierarchical Clustering Model forSpeeding up the Reconciliation of xml-based, Semistructured Data inMediation Systems, Ph.D. Dissertation, 2001.

[38] D. Halperin, S. Kandula, J. Padhye, P. Bahl, D. Wetherall, Augmentingdata center networks with multi-gigabit wireless links, in:SIGCOMM ’11.

[39] C. Clark, K. Fraser, S. Hand, J.G. Hansen, E. Jul, C. Limpach, I. Pratt, A.Warfield, Live migration of virtual machines, in: NSDI ’05.

[40] R. Mccreadie, C. Mcdonald, I. Ounis, Comparing distributed indexing:to MapReduce or not? in: 7th Workshop on Large-Scale DistributedSystems for Information Retrieval, 2009.

[41] M. Zaharia, M. Chowdhury, M.J. Franklin, S. Shenker, I. Stoica, Spark:Cluster computing with working sets, in: HotCloud, 2010.

Shihong Zou received his Bachelor of Engi-neering degree in Computer Engineering fromNanjing University of Posts and Telecommu-nications (Nanjing, China) in 1999, and hisPh.D. degree in communication and informa-tion systems from Beijng University of Postsand Telecommunications (BUPT) in 2004. Heis an Associate Professor at State Key Labora-tory of Networking and Switching Technologyin Beijing University of Posts and Telecom-munications. His research interests includemobile security, wireless networking, and

network performance analysis. He has published over 40 papers in topconferences and journals, and has applied over 20 patents.

Xitao Wen received his Bachelor of Sciencedegree in Computer Science from PekingUniversity (Beijing, China) in 2010, and is nowpursuing his Ph.D degree in NorthwesternUniversity, US. His interests span the area ofnetworking and security in networked sys-tems, with a current focus on software-defined network security and data centernetworking.

Yan Chen received my Ph.D. in ComputerScience from University of California atBerkeley in December 2003, now is an asso-ciate professor in Northwestern University,US. His research interests are in computernetworking and large-scale distributed sys-tems, network security, measurement, anddiagnosis. He won the DOE Early CAREERAward in 2005, the DOD (Air Force of Scien-tific Research) Young Investigator Award in2007, and the Microsoft Trustworthy Com-puting Awards in 2004 and 2005 with my

colleagues. Based on Google Scholar, his papers have been cited over 6500times, and the h-index of his publications is 31 as of December 2013.

Kai Chen is an Assistant Professor with the Department of ComputerScience and Engineering, Hong Kong University of Science and Technol-ogy, Hong Kong. He received his PhD from Northwestern University,Evanston IL in 2012. His research interests include networked systemsdesign and analysis, data center networks, and cloud computing. He isinterested in finding simple yet deep and elegant solutions to real-worldnetworking and systems problems.

Shan Huang received her Bachelor of Engineering degree in Communi-cation Engineering from Xi’an University of Posts and Telecommunica-tions (Xi’an, China) in 2013, now is a postgraduate student incommunication and information systems from Beijing University of Postsand Telecommunications (BUPT).Her research interests include wirelessnetworking and network performance analysis.

YongqiangLiu received his Bachelor of Engineering degree in ComputerScience from Harbin Institute of Technology (Harbin, China) in 2001,andhis Ph.D.degree in Computer Networking from Peking University in2006.He worked as Researcher, Research Manager at NEC LaboratoriesChina from Jul.2011 to Feb.2012.Now is a Senior Research Scientist atHewlett-Packard Laboratories China.His research interests includewire-less ad hoc network and wireless mesh network, data center networking,parallel computing, and android networking research

Yong Xia is currently a principal engineer at Microsoft, Bellevue, WA,USA. His research interests include computer networks and distributedsystems. He received a B.E. degree from Huazhong University of Scienceand Technology, Wuhan, China, a M.E. degree from the Institute ofAutomation, Chinese Academy of Sciences, Beijing, and a Ph.D. degreefrom Rensselaer Polytechnic Institute, Troy, NY, in 1994, 1998, and 2004,respectively. He is a Senior Member of IEEE and a Member of ACM.

Chengchen Hu received his B.S. degree from the Department of Auto-mation, Northwestern Polytech-nicalUniversity, Xi’an, China, and his Ph.D. degree from the Department of Computer Science and Technology,Tsinghua University, in 2003 and 2008 respectively. He worked as anassistant research professor in Tsinghua University from Jun. 2008 to Dec.2010 andis now an associate professor in the MOE key lab for IntelligentNetworks and Network Security, Department of Computer Science andTechnology, in Xi’an Jiaotong University. His recent research interests iscomputer networking systems, including network measurement andmonitoring, cloud data center networks, software defined networking. Heservers at the organization committee and technique program committeeof several conferences, e.g., INFOCOM, IWQoS, GLOBECOM, ICC, etc.