Accepted Manuscript

Optimization of live virtual machine migration in cloud computing: A survey and futuredirections

Mostafa Noshy, Abdelhameed Ibrahim, Hesham Arafat Ali

PII: S1084-8045(18)30083-3

DOI: 10.1016/j.jnca.2018.03.002

Reference: YJNCA 2083

To appear in: Journal of Network and Computer Applications

Received Date: 11 November 2017

Revised Date: 18 February 2018

Accepted Date: 5 March 2018

Please cite this article as: Noshy, M., Ibrahim, A., Ali, H.A., Optimization of live virtual machine migrationin cloud computing: A survey and future directions, Journal of Network and Computer Applications(2018), doi: 10.1016/j.jnca.2018.03.002.

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service toour customers we are providing this early version of the manuscript. The manuscript will undergocopyediting, typesetting, and review of the resulting proof before it is published in its final form. Pleasenote that during the production process errors may be discovered which could affect the content, and alllegal disclaimers that apply to the journal pertain.

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

Optimization of Live Virtual Machine Migration inCloud Computing: A Survey and Future Directions

Mostafa Noshya, Abdelhameed Ibrahimb, Hesham Arafat Alic

aComputer Engineering and Systems Dept., Faculty of Engineering, Mansoura University,Egypt (e-mail: mostafanoshy@mans.edu.eg)

bComputer Engineering and Systems Dept., Faculty of Engineering, Mansoura University,Egypt (e-mail: afai79@mans.edu.eg)

cComputer Engineering and Systems Dept., Faculty of Engineering, Mansoura University,Egypt (e-mail: h arafat ali@mans.edu.eg)

Abstract

In the growing age of cloud computing, shared computing and storage resources

can be accessed over the Internet. Conversely, the infrastructure cost of the

cloud reaches an incredible limit. Therefore, virtualization concept is applied in

cloud computing systems to help users and owners to achieve better usage and

efficient management of the cloud with the least cost. Live migration of virtual

machines(VMs) is an essential feature of virtualization, which allows migrating

VMs from one location to another without suspending VMs. This process has

many advantages for data centers such as load balancing, online maintenance,

power management, and proactive fault tolerance. For enhancing live migration

of VMs, many optimization techniques have been applied to minimize the key

performance metrics of total transferred data, total migration time and down-

time. This paper provides a better understanding of live migration of virtual

machines and its main approaches. Specifically, it focuses on reviewing state-

of-the-art optimization techniques devoted to developing live VM migration ac-

cording to memory migration. It reviews, discusses, analyzes and compares

these techniques to realize their optimization and their challenges. This work

also highlights the open research issues that necessitate further investigation to

∗Corresponding authorEmail address: mostafanoshy@mans.edu.eg (Mostafa Noshy)

Preprint submitted to Journal of Network and Computer Applications March 6, 2018

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

optimize the process of live migration for virtual machines.

Keywords: Virtualization, Hypervisor, Virtual machine, Live virtual machine

migration, Downtime

1. Introduction

Cloud computing depends basically on a major technology called virtual-

ization. That technology was started in the 1960s by IBM as a transparent

way to provide interactive access to mainframe computers, where time-sharing

and resource-sharing allow to multiple users (and applications) to use huge5

size and highly expensive hardware concurrently [1, 2]. Nowadays, rapid tech-

nological development of processing power and storage has made computing

resources more and more abundant, cheaper and powerful than before. Thus,

cloud computing trend has emerged due to this development, where comput-

ing and storage resources can be delivered to multiple users over the Internet10

in an on-demand fashion [3]. Therefore, modern cloud computing environments

can exploit virtualization technology to increase resources utilization and reduce

both computational and energy costs. Virtualization technology allows running

multiple operating systems (OSs) on a single physical machine with high perfor-

mance. Each OS runs on a separate Virtual Machine (VM), which is controlled15

by a hypervisor.

Live migration of VMs is a major advantage of virtualization, which is a

powerful tool to manage cloud environment resources. A VM can be migrated

seamlessly and transparently from one physical machine (source host) to an-

other (destination host), while the VM is still running during migration. Load20

balancing [4] is the main benefit for live VM migration to migrate VMs from

over-loaded servers to light-loaded ones to relieve loads on congested hosts. An-

other benefit for live VM migration is power management [5], where VMs that

run light-load jobs can be consolidated into fewer numbers of servers to minimize

IT operation expenses and power consumption by shutting down some servers25

after migration completion from these servers. Proactive fault tolerance and

2

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

online maintenance [6] are also benefits for live VM migration, where VMs can

be migrated to another server to be able to make some upgrade or maintenance

for physical machines or to avoid expected faults before their occurrence. VM

migration can also increase application performance by migrating some VMs30

which need more resources from their limited resources servers to rich resources

ones. Today, many hypervisors support live VM migration such as Xen [7],

KVM [8], VMware [9] and Microsoft Hyper-V [10].

Live migration of VMs has solved the residual dependency problem, which

process migration [11] suffers from. In process migration, the source host must35

remain available and network-accessible after process migration. In contrast,

the source host can be decommissioned after VM migration. Live VM migration

has become a hot research topic since its appearance [12]. Thus, optimization

of live VM migration has attracted the attention of academic and industrial

researchers. They try to improve migration process by minimizing total data40

transferred, total migration time and downtime. Most of the researches focus on

local area network (LAN) domain where the source machine and the destination

machine are on the same LAN with almost shared storage between the source

host and the destination one. Thus, their main focus is to migrate memory

with minimal time and without performance disruption. VM should also retain45

the same IP address after migration by generating unsolicited ARP reply to

advertising that the IP has moved to another location [12]. There are also

many researches that focus on migrating VMs in Wide Area Network (WAN)

domain [13, 14, 15, 16, 17], where storage is not shared and must be transferred

in addition to memory. These researches use some techniques such as dynamic50

DNS, IP tunneling, and mobile IP.

Although many kinds of literature have been devoted to VM migration area;

we focus on studying optimization techniques that have been proposed in the

context of memory migration. Storage is almost shared in data centers; thereby

memory migration is the main bottleneck in live VM migration due to frequent55

dirtying of VM memory pages during migration. Therefore, the main contri-

bution of this article is to review existing optimization techniques in memory

3

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

migration. It explores pre-copy, post-copy and hybrid approach, which are the

main approaches to live migrate VMs. The key performance metrics that used

to evaluate the performance of migration are presented. We discuss the man-60

ner of each technique to minimize total data transferred, total migration time

or downtime. We also classify the optimization techniques into compression

techniques, deduplication techniques, checkpointing techniques and other ones.

Optimization techniques in each category are discussed, analyzed and compared

to understand their contribution and limitation better. This survey also seeks65

to introduce some hot research directions that deserve more research efforts

and points out to challenges of these directions. These research directions in-

clude achieving more optimization in memory migration, facing challenges of

migration over WAN links, studying power cost during live VM migration, live

migration of multiple VMs and studying security issues to protect the migrated70

VMs against attacks.

The rest of the paper is organized as follows: Section 2 explores virtualization

and VM migration process. This section also discusses the two main approaches

to live VM migration process, which are pre-copy and post-copy, in addition to

the hybrid one. Section 3 states the key performance metrics which used to75

assess live VM migration techniques. Then, different optimization techniques

that seek more efficient migration are discussed. Section 4 introduces research

direction issues. Finally, section 5 concludes this work.

2. Background

Live VM migration process depends basically on virtualization technology.80

This section will first briefly overview virtualization technology. Then, we will

explore the process of live VM migration and its main approaches.

2.1. Virtualization

Virtualization technology facilitates sharing hardware resources to increase

resource utilization and decrease operation cost, where some operating systems85

4

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

can use the same hardware. It enables each operating system to run on isolated

and secured virtual environment, which is called virtual machine, to provide

an illusion for them as if they are running on actual hardware. Virtual Ma-

chines are instances of the physical machine that created and managed by a

software layer above the physical machine, which is called a hypervisor or a90

Virtual Machine Monitor (VMM). Figure 1 depicts the relation between virtual

machines and the physical machine. Hypervisors guarantee required resources

for each virtual machine including a virtual processor, a virtual memory, and

other virtual resources. They ensure for administrators of data centers that if

any of virtual machines is crashed or hijacked, other virtual machines on the95

same physical machine are not affected. Hypervisors have two types, Type 1

and Type 2. In Type 1, the hypervisor runs directly on top of the bare-metal

hardware. In contrast, hypervisors run on a host operating system in Type 2.

One of the most important advantages of virtualization, which is provided by

most hypervisors, is live virtual machine migration. Thus, we will focus on this100

feature to reach a better understanding of its different optimization techniques.

Figure 1: Virtualization [1]

2.2. Virtual Machine Migration

VM migration techniques seek to significantly improve manageability of data

centers and clusters by transferring a complete state of the VM from source host

5

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

to the destination host. It is necessary to refer to the network architecture for105

data centers in the context of VM migration as shown in Fig 2. VMs can be

migrated from one server to another one in the same rack through Top-of-Rack

(ToR) switches. They can be also migrated to a server in another rack through

data center core network. To accomplish the migration process of the VM, its

states need to be migrated which include (memory, storage, and virtual device110

states). Figure 3 shows the migration process for a VM from one host to another.

This migration process can be live or non-live.

Figure 2: Network architecture for data center [18]

Non-live VM migration approaches [19, 20] depend on suspending the virtual

machine at the source host, and then the VM’s state is migrated. Finally, the

VM at the destination host is resumed. Non-live VM migration does not achieve115

transparency, where the user notices an interruption during migration.

In contrast, VM continues to run during migration without disrupting service

in live VM migration techniques. Thus, users should not be aware of the mi-

gration due to the transparency of live VM migration techniques. Now, we will

discuss live VM migration techniques in more detail. Although VM’s full state120

6

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

Figure 3: Virtual Machine Migration

must be transferred to the new location, the bottleneck is transferring mem-

ory pages. There are two main approaches to live VM migration, pre-copy and

post-copy, which are commonly used in live VM migration techniques. There-

fore, we will explore them in addition to the hybrid approach in the following

subsections. These main approaches are shown in Fig 4.125

Figure 4: Live Virtual Machine Migration Approaches

2.2.1. Pre-copy

Pre-copy [12, 21, 22] is the predominant approach in live VM migration.

Most hypervisors, such as Xen [7], KVM [8] and VMware [9] hypervisors, use

this approach as the default migration one. In this migration approach, the

memory state of VM is transferred in iterations from the source host to the130

destination host, whereas the VM is still running on the source host. In the

first iteration, the entire memory pages of the VM are transferred to the desti-

7

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

nation host. The transmitted memory page is resent to the destination in the

next iteration if it is dirtied. Thus, only modified pages by the running VM are

transferred to the destination host in several iterations. This iterative transfer-135

ring of dirtied pages continues until either the number of dirtied pages reaches

a predefined threshold or the number of iterations reaches a preset number.

Then, the VM is suspended at the source host, and all the remaining memory

pages in addition to CPU state such as register values are migrated to the des-

tination host. Finally, VM is restarted at the destination host, and the copy140

at the source host is destroyed. Pre-copy seeks to reduce the amount of data

needs to be migrated during downtime, which is the time during which the VM

is suspended. Therefore, this approach reduces VM downtime and application

degradation. However, transferring of duplicated memory pages prolongs the

total time of migration and consumes the network bandwidth.145

In pre-copy approach, the source host keeps the newest state of the VM’s

memory until the migration process is completed. Therefore, this approach

is reliable because the migration can be aborted and VM at source host can

continue running when the destination host crashes.

2.2.2. Post-copy150

Post-copy approach [23, 24] is suggested to decrease total migration time.

First, the VM which is running on source host is suspended. Then, this approach

transmits only the processor state to the destination host. Finally, the VM at

destination host is started, and then memory pages are actively pushed from the

source host to the destination host. A page fault will be generated if the VM155

at the destination host wants to read a not transferred memory page. Thus,

this faulted page is fetched from the source host. This migration approach

thus ensures that each memory page is transferred at most once by avoiding the

duplicate transmission overhead of pre-copy approach. Therefore, it reduces the

total migration time, especially for write-intensive applications. Unfortunately,160

this strategy may increase downtime and suffer from performance degradation,

when there are numerous page faults. These page faults will cause the VM to

8

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

wait for fetching these pages after VM’s start on the destination host.

This approach is less reliable than pre-copy approach. If the VM at the

destination host crashes, then the source host has not the newest copy of the165

VM since the destination host has the most up-to-date copy of the VM. On

the other hand, if the VM at source host crashes, then the VM at destination

host cannot continue executing the VM with only a part of the whole state.

Therefore, the migration, in contrast to pre-copy, cannot be aborted during

migration.170

2.2.3. Hybrid

Hybrid approach [25] seeks to reduce downtime and performance degrada-

tion which post-copy approach suffers from due to faulted pages. It also tries

to decrease total migration time and total data transferred in comparison to

pre-copy approach. Consequently, the most frequently used of VM memory175

pages are transferred. Then, it suspends the VM at source host to transfer the

minimum state of the VM to the destination host. Finally, the VM is resumed

at the destination host. The remaining memory pages are retrieved from source

host to the destination host after the VM is resumed at the destination host.

This approach is also like post-copy not reliable.180

3. Optimization Techniques

Before discussing the different techniques to optimize live VM migration

process, we must mention the main performance metrics which used to evaluate

the performance of migration techniques. There are three major metrics, which

are used in the evaluation:185

1. Total migration time: It is the duration between starting the migration

process and the time when the VM is no longer available on the source

host. Migration techniques seek to reduce this duration as possible.

2. Downtime: It is defined as the period during which the VM is unrespon-

sive due to suspending the VM execution. Migration researches also seek190

9

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

to increase the performance of the migration by reducing downtime as

possible, which make the migration process more transparent.

3. Total data transferred: It is defined as the overall transferred data during

the migration process. The more this data is reduced, the more network

consumption is reduced.195

Now, we will review state-of-the-art existing optimization techniques as

shown in Fig 5. These techniques are summarized in Tables 1, 2 and 3.

Figure 5: Taxonomy of live VM migration optimization techniques

3.1. Compression

Many optimization techniques tend to use compression algorithms to de-

crease total data transferred, thereby decrease total migration time. These200

techniques utilize regularities in memory pages to encode them at the source

host into fewer data before they are transferred. On the other hand, the re-

flected operation, which is called decompression, is applied at the destination

host to retrieve the original data. Although compression algorithms cause some

CPU overhead, utilizing the available network bandwidth and minimizing total205

migration time have the first priority. Jin et al., 2009 [26] proposed a technique

10

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

to decrease the transferred data during migration based on memory compres-

sion, which is called (MECOM) for short. This technique is considered the first

use of memory compression to accelerate the migration process. Data being

transferred in each round are first compressed at the source host by MECOM’s210

algorithm and then decompressed at the destination host. Therefore, this tech-

nique also minimizes downtime and total migration time. To balance the VM

migration cost and performance, an algorithm was designed based on compres-

sion which consists of the two following parts:

1. Determining pages which contain many zero bytes or high similarity, which215

are called strong regularities pages.

2. Choosing an appropriate compression algorithm. It chooses a fast and sim-

ple compression algorithm for strong regularities pages and an algorithm

with high compression ratio for weak regularities pages (low similarity).

Their experiment results for MECOM, compared with the default pre-copy tech-220

nique of Xen, show that the technique can minimize the total migration time

with 34.93%, the downtime with 27.1%, and the total data transferred on aver-

age with 68.8%. However, there is some CPU overhead, about 30%, due to the

compression and decompression processes. This overhead may be acceptable in

practice due to the availability of CPU resource today with the development of225

multi-core technology. This technique can achieve better results with workloads

that contain zero bytes or high similarity which leads to less total data trans-

ferred, less total migration time, more network bandwidth utilization with less

processing overhead.

Svard et al., 2011 [27] modified KVM hypervisor, to use delta compression,230

to optimize the throughput of migration and decrease downtime. They use a

live migration algorithm that is called (XBRLE), which applies an Exclusive-OR

binary operation with Run Length Encoding. This algorithm does not send the

dirtied page, but it sends only the delta between the old memory page, which

is cached, and the new dirtied one by applying the Exclusive-OR operation.235

Then, RLE is used to compress the delta page before it is transferred. This

11

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

compressed delta page is decompressed at the destination host. Finally, an

Exclusive-OR operation is applied between the delta page and the old memory

page to reconstruct the new memory page. Workloads whose memory pages are

dirtied repeatedly and environments with low network bandwidth will benefit240

more from this technique. Although avoiding to transfer full dirtied pages can

achieve good results, the proposed technique needs a large cache memory to

store memory pages.

Ma et al., 2012 [28] adopted a technique to reduce the transferred memory

pages to minimize the total migration time and downtime. Their technique is245

based on Memory Exploration and Memory Encoding, so it is named (ME2)

for short. ME2 explores the memory to identify the unallocated memory pages

and transfers only the used memory pages. It compresses the allocated memory

pages before transferring with Run Length Encoding algorithm (RLE). Their

experiment results for ME2, compared with the default pre-copy technique of250

Xen, show that their technique can reduce on average the downtime with 47.6%,

the total transferred data with 50.5% and the total migration time with 48.2%.

Although scanning the memory pages to decide the allocated pages that should

be transferred causes some time overhead, it is compensated by a reduction

in total migration time. Compression is also applied in another theses such255

as [29, 30, 31]. In general, applying compression algorithms in optimization

techniques of live VM migration can have an effective role in decreasing total

data transferred and total migration time that almost outperforms both time

and processing cost for compression/decompression.

3.2. Deduplication260

Some optimization techniques tried also to reduce amount of migrated data

but relying on deduplication of data. Based on deduplication, they can save net-

work bandwidth and reduce total migration time by getting rid of transferring

identical or similar memory pages. Zhang et al., 2010 [31] proposed a technique

based on deduplication of data, which is called Migration with Data Dedupli-265

cation (MDD). This proposed technique can reduce network consumption and

12

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

Table 1: Live VM migration optimization techniques based on compression.

Optimization

technique

Approach VMM Technique Optimization Limitation

MECOM

[26]

Pre-copy XEN Use simple and fast

compression algo-

rithm for memory

pages with high

similarity and high

compression ratio

algorithms for ones

with low similarity.

Reduce downtime

with 27.1%, total

data transferred

with 68.8% and

total migration

time with 34.93%.

About 30% CPU

overhead due to

compression and

decompression.

Delta com-

pression

[27]

Pre-copy KVM Send only delta

page between old

memory page and

the new one. Use

RLE for compress-

ing delta pages.

Decrease down-

time and total

migration time.

Compression and

decompression

introduce CPU

overhead. Ade-

quate cache for

memory pages is

required.

ME2 [28] Pre-copy XEN Identifying used

memory pages,

then sending them

after compression

by RLE.

Reduce downtime

with 47.6%, total

data transferred

with 50.5%, and

total migration

time with 48.2%.

CPU overhead due

to compression

and decompres-

sion. Overhead

due to scanning

memory.

reduce also both total migration time and downtime by minimizing the total

data transferred during migration. It depends on hash-based fingerprints tech-

nology to determine similarity among memory pages. It utilizes self-similarity

of the memory to decrease the transferred data by applying Exclusive-OR op-270

eration between identical or similar pages to convert the redundant data to

continuous zeros blocks. Then, it can eliminate the redundant data by encoding

the parity with the simplest encoding algorithm (RLE).

Experiment results for MDD, compared with the default pre-copy technique

of Xen, show that this technique can reduce the total migration time with275

32%, the downtime with 26.16%, and the total data transferred with 56.60%.

13

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

However, MDD requires 47.21% extra CPU than Xen. In addition to the pro-

cessing cost, it also consumes memory resource to cache memory pages during

migration. MDD applied deduplication in the context of migrating memory

within a cluster where storage is shared. Therefore, deduplication can be uti-280

lized to achieve good results over low bandwidth links in WAN environments

where a large date is migrated especially in such workloads with more similar-

ity. Deduplication is also exploited in inter-rack live migration (IRLM)[32] to

avoid transferring identical memory pages, depending on inter-rack similarity,

not self-similarity when migrating multiple VMs across racks. There are an-285

other theses such as [33, 34, 35, 36, 37] applied deduplication to optimize live

VM migration.

3.3. Checkpointing

Liu et al., 2009 [38] introduced a technique that adopts the technology of

Checkpointing/Recovery and Trace/Replay, which is called for short CR/TR-290

Motion, to make VM migration faster and more transparent. This technique

transfers a checkpoint of the working VM while the VM is still running on the

source host. Then, it transfers iteratively the small system execution log of non-

deterministic events (time and external input) instead of the large size of VM’s

memory pages. These logs are replayed in the destination host to reconstruct a295

consistent VM’s memory image. Finally, the VM at the source host is suspended

when the log size is reduced to a threshold value. Therefore, the remaining log

file is transferred to the destination host to be replayed there, where VM can

normally run at the destination host instead of the source host.

Experiment results compared to pre-copy approach show that this technique300

can reduce up to 72.4% of downtime, 31.5% of total migration time and 95.9%

of total transferred data on average. However, this technique causes about

8.54% overhead due to migration. This overhead is small compared to the great

improvement in total migration time, downtime and total data transferred. This

technique may achieve good results if it is used in WAN environments which305

have low-bandwidth, but it must be taken into account difference between log

14

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

Table 2: Live VM migration optimization techniques based on deduplication such as MDD

[31] and based on checkpointing such as CR/TR-Motion [38] and Vecycle [39].

Optimization

technique

Approach VMM Technique Optimization Limitation

MDD [31] Pre-copy XEN Reduce transferred

data using XOR

between similar

pages. Redundant

data is converted

to continuous zeros

blocks. RLE is

used for encoding.

Reduce the total

data transferred

with 56.60%, the

downtime with

26.16%, and the

total migration

time with 32%.

About 47.21% on

average CPU over-

head due to data

deduplication.

CR/TR-

Motion

[38]

Pre-copy XEN Transferring a

checkpoint of the

VM. Then, trans-

ferring iteratively

small system exe-

cution logs which

are replayed at the

destination.

Reduce on average

72.4% of down-

time, 31.5% of to-

tal migration time

and 95.9% of total

transferred data.

8.54% on average

application perfor-

mance overhead

due to checkpoint-

ing and logging.

Vecycle [39] N/A KVM Store a checkpoint

of the transferred

VM at the source

host, which can be

used if the VM is

transferred back to

the source host.

Reduce trans-

ferred data and

total migration

time if migrated

VM is migrated

again to its source

host.

Storage overhead

due to storing

checkpoints. CPU

overhead due to

calculating check-

sums. Only useful

when VM is mi-

grated again to its

source.

growth rate and the network transfer rate.

Vecycle, 2015 [39] depends on storing a checkpoint of the transferred VM to

the local disk of the source host. If the VM is transferred again to the source

host, this checkpoint will decrease total transferred data and total migration310

time. In this case, only memory pages which are updated need to be migrated.

These updated pages are determined by using the checksum. Thus, this tech-

15

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

nique decreases total migration time, where retrieving memory pages from local

storage is faster than retrieving them from slow networks. However, this tech-

nique may cause only storage overhead, due to storing checkpoints, if the VM is315

not transferred back to the source host or if VM memory is changed drastically.

3.4. Other Optimization Techniques

3.4.1. Reuse Distance

Alamdari and Zamanifar, 2012 [40] used an algorithm which is called reuse

distance. This algorithm minimizes the number of dirty memory pages which320

are sent in each iteration. It measures the distance between the memory page

and its next update in each iteration. The memory pages with low distances are

updated (dirtied) frequently, these pages are called skipped pages, and they are

not transferred in the iteration. Experiment results show that this technique

can reduce total migration time, downtime and total transferred data for VMs325

with write-intensive applications because it reduces the transfer of frequently

dirtied pages.

3.4.2. Three Phase Optimization

Sharma and Chawla, 2016 [41] proposed a technique that depends on three

phases to optimize live virtual machine migration, which is called (TPO) for330

short. In the first phase, it reduces the transferred memory pages at the first it-

eration by transferring only un-updated memory pages. In the second phase, it

also reduces the transferred memory pages from the second iteration to the one

before last by using historical statistics to predict frequently updated memory

pages and avoid transferring them. In the last one, it transfers the remaining335

memory pages. If these remaining memory pages are large, they are compressed

before transferring using (RLE) algorithm. Experiment results for TPO, com-

pared with the default pre-copy technique of Xen, show that this technique

can reduce the total data transferred for high workloads with 71%, the total

migration time with 70%, and the downtime with 3%.340

16

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

3.4.3. SonicMigration

Akane Koto et al., 2012 [42] proposed a technique to decrease network traf-

fic and total migration time by transferring only the necessary memory pages.

This technique avoids transferring soft pages, which can be free pages or file

cache pages, to decrease migration noise. It also uses a shared memory between345

VMM and the guest kernel to store and update addresses of soft memory pages.

To keep data consistent, the kernel is updated at the VM-suspend phase. At

this stage, VMM interrupts the VM before suspending it. After the kernel data

is updated, the guest sends a hypercall to VMM to start the stop and copy

phase. This hypercall introduces CPU and memory overhead. Experiment re-350

sults compared to Xen show that SonicMigration can reduce the total migration

time with 68.3% of and the network traffic with 83.9%.

3.4.4. Agile Migration

Deshpande et al., 2016 [43] proposed a hybrid technique of pre-copy and

post-copy for live VM migration to solve quickly the resource pressure problem355

based on agility. This technique reduces the number of transferred memory

pages. It tracks the working set memory pages (hot memory pages) of each VM

to transfer them from the source to the destination whereas the non-working

set (cold memory pages) are evicted to a portable per-VM swap device. After

resuming execution at the destination host, the dirtied working set memory360

pages can be requested by the destination or actively pushed from the source.

The migration of memory pages from source to destination is shown in Fig 6.

3.4.5. Introspection-Based Memory Pruning

Wang et al., 2017 [44] proposed a technique to reduce total data transferred

and thereby total migration time based on classifying memory pages into five365

categories. These categories are anonymous, inode, kernel, cache and free mem-

ory pages. Thus, they transfer only the first three categories of memory pages,

as they are necessary for kernel execution. Although free memory pages do

not affect normal execution, avoiding to transfer cache memory pages may lead

17

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

Table 3: Other live VM migration optimization techniques.

Optimization

technique

Approach VMM Technique Optimization Limitation

Reuse dis-

tance [40]

Pre-copy XEN Measure distance

between memory

page and its next

update to skip

transferring Mem-

ory pages with low

distances.

Reduce total migra-

tion time, down-

time and total data

transferred for VMs

with write-intensive

applications.

CPU overhead due

to distance calcu-

lation for memory

pages.

TPO [41] Pre-copy XEN Reduce transfer-

ring of memory

pages based on

three phases.

Reduce downtime

with 3%, total data

transferred with

71%, and total

migration time

with 70% for high

workloads.

0.05% space over-

head and CPU

processing over-

head to predict

frequently updated

pages.

Sonic Mi-

gration[42]

Pre-copy XEN Avoid transferring

soft pages.

Reduce 68.3% of to-

tal migration time

and 83.9% of net-

work traffic.

CPU and memory

overhead in VM-

suspend phase.

Agile mi-

gration

[43]

hybrid KVM Transferring hot

memory pages of

VM, whereas cold

memory pages

are evicted to a

portable per-VM

swap device.

Reduce memory

pressure, total mi-

gration time and

total data trans-

ferred. Eliminate

transferring of cold

pages without caus-

ing residual state at

the source.

Retrieving faulted

pages may increase

the downtime.

Introspection-

Based

Memory

Pruning

[44]

Pre-copy KVM Avoid transferring

both free and

cache memory

pages.

Reduce 72% on av-

erage of total migra-

tion time.

Time overhead

due to introspec-

tion of pages and

handling missing

cache pages.

18

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

Figure 6: Agile VM migration: The working set is only transferred directly on the TCP

connection where the destination host can retrieve cold pages from the per-VM swap device

on-demand [43]

to performance degradation due to inconsistency between actual transferred370

memory pages and the kernel state. Therefore, they use a kernel notification

mechanism to handle missing pages by making the kernel aware of them. Ex-

periment results compared with the default pre-copy of KVM showed that this

technique could reduce 72% on average of the total migration time.

4. Research Directions375

This section refers to the main research directions in live VM migration and

their challenges. These research directions include memory migration optimiza-

tion which is the major bottleneck in live migrating VMs over LAN links espe-

cially when migrating VMs with memory-intensive workloads, facing challenges

of migration over WAN links to achieve more optimization in the migration380

process, studying power cost due to live VM migration which should be small

as possible, live migration of multiple VMs and security issues to ensure that

migrated VMs are protected against attacks. Figure 7 shows the main research

directions that need more researches for more optimization. The aforemen-

tioned research issues and their challenges will be explored in more detail in the385

following subsections.

19

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

Figure 7: Main research directions in live VM migration

4.1. Optimization in Memory Migration

As we explored, there are many techniques to optimize live VM migration

process. These techniques try to optimize the migration by decreasing the total

migration time, total data transferred or downtime. The main focus of this390

survey is to discuss different techniques to enhance migration within the same

LAN, where the storage does not need to be transferred from the source host

to the destination host because it is shared.

Thus, memory migration is the main bottleneck of migration over LAN.

Although different explored techniques optimized memory migration, it is still395

a hot research topic to reach to better optimization by utilizing available re-

sources. This optimization process can be achieved by designing a technique

applying one, or more than one, of the main optimization, approaches such as

(compression, deduplication and checkpointing) or using a technique to predict

dirty pages of memory in advance such as [41, 45]. These proposed techniques400

may lead to more reduction in the migrated memory which results in more

optimization in total migration time, downtime or both.

4.2. Migration over WAN

Although live migration over LAN links is the major interest for many re-

searches in this field, there are also many researches such as [13, 14, 15, 16, 17]405

that study migration over WAN links. This research area is more complex than

20

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

migration over LAN links and needs more research efforts. The main difficulty

is the large size of data that need to be transferred, where the storage is not

shared and must be migrated. This large-sized data suffers from the heterogene-

ity of network architecture and migration over intermittent and low-bandwidth410

links to a distant host. Therefore, finding a way to decrease the total data

transferred over WAN links will increase utilization of network links and will

also decrease the total migration time. Also, using high-speed links can reduce

total migration time and downtime which increase migration performance. As

we deal with migration of the storage over long distances, reliability should be415

taken into account to ensure data consistency. Although replication [46] is the

common solution to achieve reliability, erasure codes [47] can be a good alter-

native solution with less storage overhead [48]. There are also many researches

[49, 50, 51] seek to efficiently use erasure codes to achieve more reliability in

data storage.420

4.3. Power consumption of live VM migration

Introducing energy-aware contributions in cloud computing [52, 53, 54, 55]

has become a hot research topic due to both economic and environmental rea-

sons. As we previously pointed out that power management is one of the major

applications for live VM migration in data centers, where VMs can be con-425

solidated into less physical servers to minimize power consumption. However,

studying consumed power by live VM migration process itself is a significant

research point that was overlooked previously. Thus, more researches have been

devoted recently to study energy overhead of this process. The additional con-

sumed power has two parts, one at the source host and the other at the des-430

tination host. This power cost is due to using more computing and network

resources for accomplishing the migration process. Huang et al. [56] demon-

strated power cost of live VM migration based on a linear model depends on

CPU utilization. On the other hand, Liu et al. [57] proposed another linear

model to study this cost based on the amount of transferred data. Strunk [58]435

proposed a third linear model as a function of VM memory size and the net-

21

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

work bandwidth between the source host and the destination host. Dhanoa and

Khurmi [59] analyzed the effect of VM size and network bandwidth on migration

time and consumed power during live VM migration. Prediction of energy cost

due to live VM migration are introduced based on resource utilization of the440

servers in [60]. The power cost of VM migration is modeled in [18], where CPU

overhead and data transfer due to migration are considered. Thus, reaching to a

more accurate prediction model for the power cost of live VM migration will help

to take better migration decisions in the context of data centers power manage-

ment scenarios. Also, developing optimization techniques for live VM migration445

with less migration time and processing overhead will help in minimizing power

cost during the migration process.

4.4. Live migration of multiple VMs

Many research efforts have been devoted to optimizing the live migration

of a single VM, and also there is a need to live migrate multiple VMs in real450

applications within data centers. Some optimization techniques such as [32,

34, 61, 62, 63] have been proposed in recent years to enhance live migration

of multiple VMs. The main concern for these techniques is to reduce the total

migration time, which is the total time from starting migration of the first VM to

finishing migration of the last one, and the total downtime which is the total time455

of periods during which any VM is unresponsive. Thus, scheduling the multiple

VM migrations is the main challenge to optimize the performance. Another

important challenge that should be taken into account is the interference among

VMs, where migrating some of the correlated VMs may degrade service due

to network latency especially over WAN links. XU et al. [64] analyzed VM460

migration interference and co-location interference during and after migration on

both source servers and destination servers. In general, introducing techniques

that efficiently utilize available bandwidth to migrate multiple VMs with least

service degradation is an attractive research direction.

22

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

4.5. Security465

Security, especially for WAN links, is a major challenge in VM migration.

Migrated VMs must be isolated and secured enough during the migration pro-

cess, because they may contain encryption keys, passwords, or any other sen-

sitive data [65, 66]. In WAN links, VMs are transferred to their destination

hosts on shared links over long distances and through heterogeneous network470

architecture which makes them vulnerable to attacks. Therefore, the source

host and the destination host should be trusted. Also, applying a suitable en-

cryption algorithm will help to protect VMs against malicious activities. The

hypervisor, communication channel, and end-users should be secured enough

against attacks. The security has some dimensions [67], which are summarized475

in Table 4.

23

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

Table 4: Dimensions of security.

Dimension Issues

Access Control Access control lists are defined to prevent unau-

thorized activities from attackers.

Authentication Source and destination hosts should be authenti-

cated, users should be trusted, the firewall is ap-

plied, and system administrator roles should be

used.

Non-Repudiation Activities of hypervisors and users should be

monitored.

Data Confidentiality Encryption is used to protect migrated data.

Communication Transmission channel must be protected against

active and passive attacks. VPN tunnels are one

of the solutions to these attacks.

Data Integrity Attackers may inject malicious codes, manipulate

control messages of the migration protocols or mi-

grate a VM to be able to control its users and pro-

cesses. Encryption, digital signatures, and check-

sums can mitigate these attacks.

Availability Unauthorized attackers may migrate many VMs

to degrade performance and decrease availability.

Packet filtering, access control lists, authentica-

tion, and monitoring can control these attacks.

Privacy VMs, which store users’ data, maybe live mi-

grated to another physical host or a third party

without users’ knowledge. Also, Attackers may

migrate malicious VMs to the target host to sniff

information of a VM there. VLAN can be a so-

lution to these issues.

24

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

5. Conclusion

Live VM migration is a powerful tool that helps administrators of cloud data

centers to manage their resources effectively. This paper summarized the con-

cept of live VM migration, its advantages, and its approaches. It introduced the480

main performance metrics to assess live VM migration. It focused on survey-

ing the state-of-the-art optimization techniques according to memory migration,

which in general try to minimize total migration time, total data transferred and

downtime. It classified the discussed optimization techniques according to the

technique they adopt. Optimization techniques in each category was studied485

and compared to reach to their strengths and weaknesses. The research direc-

tion issues which they need more researches to optimize the migration process

were also discussed.

References

[1] R. Rose, Survey of system virtualization techniques, Oregon State Univer-490

sity (2004) 1–15.

URL http://www.ece.cmu.edu/~ece845/docs/rose-virtualization.

pdf

[2] S. Nanda, T. cker Chiueh, A survey on virtualization technologies,

Technical Report, Department of Computer Science, SUNY at Stony495

brook (2005) 1–42.

URL http://kabru.eecs.umich.edu/papers/publications/2011/

TR179.pdf

[3] Q. Zhang, L. Cheng, R. Boutaba, Cloud computing: state-of-the-art and

research challenges, Journal of Internet Services and Applications 1 (1)500

(2010) 7–18. doi:https://doi.org/10.1007/s13174-010-0007-6.

[4] T. Wood, P. Shenoy, A. Venkataramani, M. Yousif, Black-box and gray-

box strategies for virtual machine migration, in: Proceedings of the

25

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

4th USENIX conference on Networked systems design & implementation,

NSDI’07, USENIX Association, 2007, pp. 17–17.505

URL http://dl.acm.org/citation.cfm?id=1973430.1973447

[5] R. Nathuji, K. Schwan, Virtualpower: Coordinated power management in

virtualized enterprise systems, ACM SIGOPS Operating Systems Review

41 (6) (2007) 265–278. doi:10.1145/1323293.1294287.

[6] A. B. Nagarajan, F. Mueller, C. Engelmann, S. L. Scott, Proactive fault510

tolerance for hpc with xen virtualization, in: Proceedings of the 21st An-

nual International Conference on Supercomputing, ICS ’07, ACM, 2007,

pp. 23–32. doi:10.1145/1274971.1274978.

[7] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neuge-

bauer, I. Pratt, A. Warfield, Xen and the art of virtualization, ACM515

SIGOPS Operating Systems Review 37 (5) (2003) 164–177. doi:10.1145/

1165389.945462.

[8] A. Kivity, Y. Kamay, D. Laor, U. Lublin, A. Liguori, Kvm: the linux vir-

tual machine monitor, in: Proceedings of the Linux Symposium, Vol. 1 of

OLS ’07, 2007, pp. 225–230.520

URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.

488.2278

[9] Vmware, http://www.vmware.com, accessed: 2018-2-2.

[10] Microsoft, https://www.microsoft.com/en-us/cloud-platform/

virtualization, accessed: 2018-2-2.525

[11] D. S. Milojicic, F. Douglis, Y. Paindaveine, R. Wheeler, S. Zhou, Process

migration, ACM Computing Surveys 32 (3) (2000) 241–299. doi:10.1145/

367701.367728.

[12] C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt,

A. Warfield, Live migration of virtual machines, in: Proceedings of the 2Nd530

26

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

Conference on Symposium on Networked Systems Design & Implementa-

tion - Volume 2, NSDI’05, USENIX Association, 2005, pp. 273–286.

URL http://dl.acm.org/citation.cfm?id=1251203.1251223

[13] F. Travostino, P. Daspit, L. Gommans, C. Jog, C. de Laat, J. Mambretti,

I. Monga, B. van Oudenaarde, S. Raghunath, P. Y. Wang, Seamless live535

migration of virtual machines over the man/wan, Future Generation Com-

puter Systems 22 (8) (2006) 901–907. doi:10.1016/j.future.2006.03.

007.

[14] R. Bradford, E. Kotsovinos, A. Feldmann, H. Schioberg, Live wide-area mi-

gration of virtual machines including local persistent state, in: Proceedings540

of the 3rd International Conference on Virtual Execution Environments,

VEE ’07, ACM, 2007, pp. 169–179. doi:10.1145/1254810.1254834.

[15] E. Harney, S. Goasguen, J. Martin, M. Murphy, M. Westall, The efficacy

of live virtual machine migrations over the internet, in: Proceedings of the

2Nd International Workshop on Virtualization Technology in Distributed545

Computing, VTDC ’07, ACM, 2007, pp. 8:1–8:7. doi:10.1145/1408654.

1408662.

[16] A. J. Mashtizadeh, M. Cai, G. Tarasuk-Levin, R. Koller, T. Garfinkel,

S. Setty, Xvmotion: Unified virtual machine migration over long distance,

in: Proceedings of the 2014 USENIX Conference on USENIX Annual Tech-550

nical Conference, USENIX ATC’14, USENIX Association, 2014, pp. 97–

108.

URL http://dl.acm.org/citation.cfm?id=2643634.2643645

[17] M. Arif, A. K. Kiani, J. Qadir, Machine learning based optimized live vir-

tual machine migration over wan links, Telecommunication Systems 64 (2)555

(2016) 245–257. doi:10.1007/s11235-016-0173-3.

[18] C. Canali, R. Lancellotti, M. Shojafar, A computation- and network-aware

energy optimization model for virtual machines allocation, in: Proceed-

ings of the 7th International Conference on Cloud Computing and Services

27

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

Science - Volume 1: CLOSER,, INSTICC, SciTePress, 2017, pp. 71–81.560

doi:10.5220/0006231400710081.

[19] M. Kozuch, M. Satyanarayanan, Internet suspend/resume, in: Proceedings

of the Fourth IEEE Workshop on Mobile Computing Systems and Appli-

cations, WMCSA ’02, IEEE Computer Society, 2002, pp. 40–.

URL http://dl.acm.org/citation.cfm?id=832315.837557565

[20] A. Whitaker, R. S. Cox, M. Shaw, S. D. Grible, Constructing services with

interposable virtual hardware, in: Proceedings of the 1st Conference on

Symposium on Networked Systems Design and Implementation - Volume

1, NSDI’04, USENIX Association, 2004, pp. 13–13.

URL http://dl.acm.org/citation.cfm?id=1251175.1251188570

[21] M. Nelson, B.-H. Lim, G. Hutchins, Fast transparent migration for virtual

machines, in: Proceedings of the Annual Conference on USENIX Annual

Technical Conference, ATEC ’05, USENIX Association, 2005, pp. 25–25.

URL http://dl.acm.org/citation.cfm?id=1247360.1247385

[22] K. Z. Ibrahim, S. Hofmeyr, C. Iancu, E. Roman, Optimized pre-copy live575

migration for memory intensive applications, in: Proceedings of 2011 Inter-

national Conference for High Performance Computing, Networking, Storage

and Analysis, SC ’11, ACM, 2011, pp. 40:1–40:11. doi:10.1145/2063384.

2063437.

[23] M. R. Hines, U. Deshpande, K. Gopalan, Post-copy live migration of virtual580

machines, SIGOPS Operating Systems Review 43 (3) (2009) 14–26. doi:

10.1145/1618525.1618528.

[24] M. R. Hines, K. Gopalan, Post-copy based live virtual machine migration

using adaptive pre-paging and dynamic self-ballooning, in: Proceedings of

the 2009 ACM SIGPLAN/SIGOPS International Conference on Virtual585

Execution Environments, VEE ’09, ACM, 2009, pp. 51–60. doi:10.1145/

1508293.1508301.

28

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

[25] S. Sahni, V. Varma, A hybrid approach to live migration of virtual ma-

chines, in: IEEE International Conference on Cloud Computing in Emerg-

ing Markets, CCEM ’12, IEEE, 2012, pp. 1–5. doi:10.1109/CCEM.2012.590

6354587.

[26] H. Jin, L. Deng, S. Wu, X. Shi, X. Pan, Live virtual machine migration with

adaptive, memory compression, in: 2009 IEEE International Conference on

Cluster Computing and Workshops, 2009, pp. 1–10. doi:10.1109/CLUSTR.

2009.5289170.595

[27] P. Svard, B. Hudzia, J. Tordsson, E. Elmroth, Evaluation of delta com-

pression techniques for efficient live migration of large virtual machines,

SIGPLAN Notices 46 (7) (2011) 111–120. doi:10.1145/2007477.1952698.

[28] Y. Ma, H. Wang, J. Dong, Y. Li, S. Cheng, Me2: Efficient live migration

of virtual machine with memory exploration and encoding, in: 2012 IEEE600

International Conference on Cluster Computing, 2012, pp. 610–613. doi:

10.1109/CLUSTER.2012.52.

[29] L. Hu, J. Zhao, G. Xu, Y. Ding, J. Chu, Hmdc: Live virtual machine

migration based on hybrid memory copy and delta compression, Applied

Mathematics & Information Sciences 7 (2L) (2013) 639–646.605

URL www.naturalspublishing.com/Article.asp?ArtcID=3263

[30] S. Hacking, B. Hudzia, Improving the live migration process of large enter-

prise applications, in: Proceedings of the 3rd International Workshop on

Virtualization Technologies in Distributed Computing, VTDC ’09, ACM,

2009, pp. 51–58. doi:10.1145/1555336.1555346.610

[31] X. Zhang, Z. Huo, J. Ma, D. Meng, Exploiting data deduplication to accel-

erate live virtual machine migration, in: Proceedings of the 2010 IEEE In-

ternational Conference on Cluster Computing, CLUSTER ’10, IEEE Com-

puter Society, 2010, pp. 88–96. doi:10.1109/CLUSTER.2010.17.

29

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

[32] U. Deshpande, U. Kulkarni, K. Gopalan, Inter-rack live migration of mul-615

tiple virtual machines, in: Proceedings of the 6th International Workshop

on Virtualization Technologies in Distributed Computing Date, VTDC ’12,

ACM, 2012, pp. 19–26. doi:10.1145/2287056.2287062.

[33] P. Riteau, C. Morin, T. Priol, Shrinker: Improving live migration of virtual

clusters over wans with distributed data deduplication and content-based620

addressing, in: Proceedings of the 17th International Conference on Parallel

Processing - Volume Part I, Euro-Par’11, Springer-Verlag, 2011, pp. 431–

442.

URL http://dl.acm.org/citation.cfm?id=2033345.2033391

[34] U. Deshpande, X. Wang, K. Gopalan, Live gang migration of virtual ma-625

chines, in: Proceedings of the 20th International Symposium on High Per-

formance Distributed Computing, HPDC ’11, ACM, 2011, pp. 135–146.

doi:10.1145/1996130.1996151.

[35] S. Al-Kiswany, D. Subhraveti, P. Sarkar, M. Ripeanu, Vmflock: Virtual

machine co-migration for the cloud, in: Proceedings of the 20th Interna-630

tional Symposium on High Performance Distributed Computing, HPDC

’11, ACM, 2011, pp. 159–170. doi:10.1145/1996130.1996153.

[36] U. Deshpande, D. Chan, S. Chan, K. Gopalan, N. Bila, Scatter-gather live

migration of virtual machines, IEEE Transactions on Cloud Computing

PP (99) (2017) 1–1. doi:10.1109/TCC.2015.2481424.635

[37] F. Zhang, X. Fu, R. Yahyapour, Cbase: A new paradigm for fast vir-

tual machine migration across data centers, in: Proceedings of the 17th

IEEE/ACM International Symposium on Cluster, Cloud and Grid Com-

puting, CCGrid ’17, IEEE Press, 2017, pp. 284–293. doi:10.1109/CCGRID.

2017.26.640

[38] H. Liu, H. Jin, X. Liao, L. Hu, C. Yu, Live migration of virtual machine

based on full system trace and replay, in: Proceedings of the 18th ACM

30

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

International Symposium on High Performance Distributed Computing,

HPDC ’09, ACM, 2009, pp. 101–110. doi:10.1145/1551609.1551630.

[39] T. Knauth, C. Fetzer, Vecycle: Recycling vm checkpoints for faster migra-645

tions, in: Proceedings of the 16th Annual Middleware Conference, Middle-

ware ’15, ACM, 2015, pp. 210–221. doi:10.1145/2814576.2814731.

[40] J. F. Alamdari, K. Zamanifar, A reuse distance based precopy approach to

improve live migration of virtual machines, in: 2012 2nd IEEE International

Conference on Parallel, Distributed and Grid Computing, 2012, pp. 551–650

556. doi:10.1109/PDGC.2012.6449880.

[41] S. Sharma, M. Chawla, A three phase optimization method for precopy

based vm live migration, SpringerPlus 5 (1) (2016) 1022. doi:10.1186/

s40064-016-2642-2.

[42] A. Koto, H. Yamada, K. Ohmura, K. Kono, Towards unobtrusive vm655

live migration for cloud computing platforms, in: Proceedings of the

Asia-Pacific Workshop on Systems, APSYS ’12, ACM, 2012, pp. 7:1–7:6.

doi:10.1145/2349896.2349903.

[43] U. Deshpande, D. Chan, T. Y. Guh, J. Edouard, K. Gopalan, N. Bila, Agile

live migration of virtual machines, in: 2016 IEEE International Parallel and660

Distributed Processing Symposium (IPDPS), 2016, pp. 1061–1070. doi:

10.1109/IPDPS.2016.120.

[44] C. Wang, Z. Hao, L. Cui, X. Zhang, X. Yun, Introspection-based memory

pruning for live vm migration, International Journal of Parallel Program-

ming 45 (6) (2017) 1298–1309. doi:10.1007/s10766-016-0471-0.665

[45] T.-Y. Wu, N. Guizani, J.-S. Huang, Live migration improvements by re-

lated dirty memory prediction in cloud computing, Journal of Network

and Computer Applications 90 (Supplement C) (2017) 83 – 89. doi:

10.1016/j.jnca.2017.03.011.

31

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

[46] K. K. Ramakrishnan, P. Shenoy, J. Van der Merwe, Live data center migra-670

tion across wans: A robust cooperative context aware approach, in: Pro-

ceedings of the 2007 SIGCOMM Workshop on Internet Network Manage-

ment, INM ’07, ACM, 2007, pp. 262–267. doi:10.1145/1321753.1321762.

[47] E. Mugisha, G. Zhang, Reliable multi-cloud storage architecture based on

erasure code to improve storage performance and failure recovery, Interna-675

tional Journal of Advanced Cloud Computing and Applied Research 1 (2).

doi:10.23953/cloud.ijaccar.260.

[48] K. Kralevska, D. Gligoroski, R. E. Jensen, H. Øverby, Hashtag erasure

codes: From theory to practice, IEEE Transactions on Big Data PP (99)

(2017) 1–1. doi:10.1109/TBDATA.2017.2749255.680

[49] C. Huang, M. Chen, J. Li, Pyramid codes: Flexible schemes to trade space

for access efficiency in reliable data storage systems, Trans. Storage 9 (1)

(2013) 3:1–3:28. doi:10.1145/2435204.2435207.

[50] K. Kralevska, D. Gligoroski, H. Øverby, Balanced locally repairable codes,

in: 2016 9th International Symposium on Turbo Codes and Iterative Infor-685

mation Processing (ISTC), 2016, pp. 280–284. doi:10.1109/ISTC.2016.

7593121.

[51] R. Li, Y. Hu, P. P. C. Lee, Enabling efficient and reliable transition from

replication to erasure coding for clustered file systems, IEEE Transactions

on Parallel and Distributed Systems 28 (9) (2017) 2500–2513. doi:10.690

1109/TPDS.2017.2678505.

[52] E-eco: Performance-aware energy-efficient cloud data center orchestration,

Journal of Network and Computer Applications 78 (2017) 83 – 96. doi:

10.1016/j.jnca.2016.10.024.

[53] M. Shojafar, C. Canali, R. Lancellotti, J. Abawajy, Adaptive computing-695

plus-communication optimization framework for multimedia processing in

32

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

cloud systems, IEEE Transactions on Cloud Computing PP (99) (2016)

1–1. doi:10.1109/TCC.2016.2617367.

[54] M. Shojafar, N. Cordeschi, E. Baccarelli, Energy-efficient adaptive resource

management for real-time vehicular cloud services, IEEE Transactions on700

Cloud Computing PP (99) (2016) 1–1. doi:10.1109/TCC.2016.2551747.

[55] N. S. More, R. B. Ingle, Research issues for energy-efficient cloud com-

puting, in: Intelligent Computing and Information and Communica-

tion, Springer Singapore, Singapore, 2018, pp. 265–272. doi:10.1007/

978-981-10-7245-1_27.705

[56] Q. Huang, F. Gao, R. Wang, Z. Qi, Power consumption of virtual ma-

chine live migration in clouds, in: 2011 Third International Conference

on Communications and Mobile Computing, 2011, pp. 122–125. doi:

10.1109/CMC.2011.62.

[57] H. Liu, H. Jin, C.-Z. Xu, X. Liao, Performance and energy modeling for710

live migration of virtual machines, Cluster Computing (2013) 249–264doi:

10.1007/s10586-011-0194-3.

[58] A. Strunk, A lightweight model for estimating energy cost of live migration

of virtual machines, in: 2013 IEEE Sixth International Conference on Cloud

Computing, 2013, pp. 510–517. doi:10.1109/CLOUD.2013.17.715

[59] I. S. Dhanoa, S. S. Khurmi, Analyzing energy consumption during vm live

migration, in: International Conference on Computing, Communication

Automation, 2015, pp. 584–588. doi:10.1109/CCAA.2015.7148475.

[60] K. Rybina, A. Schill, Estimating energy consumption during live migration

of virtual machines, in: 2016 IEEE International Black Sea Conference on720

Communications and Networking (BlackSeaCom), 2016, pp. 1–5. doi:

10.1109/BlackSeaCom.2016.7901567.

33

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

[61] U. Deshpande, K. Keahey, Traffic-sensitive live migration of virtual ma-

chines, Future Generation Computer Systems 72 (Supplement C) (2017)

118–128. doi:10.1016/j.future.2016.05.003.725

[62] G. Sun, D. Liao, V. Anand, D. Zhao, H. Yu, A new technique for efficient

live migration of multiple virtual machines, Future Generation Computer

Systems 55 (Supplement C) (2016) 74–86. doi:10.1016/j.future.2015.

09.005.

[63] K. Onoue, S. Imai, N. Matsuoka, Scheduling of parallel migration for mul-730

tiple virtual machines, in: 2017 IEEE 31st International Conference on

Advanced Information Networking and Applications (AINA), 2017, pp.

827–834. doi:10.1109/AINA.2017.136.

[64] F. Xu, F. Liu, L. Liu, H. Jin, B. Li, B. Li, iaware: Making live migration

of virtual machines interference-aware in the cloud, IEEE Transactions on735

Computers 63 (12) (2014) 3012–3025. doi:10.1109/TC.2013.185.

[65] T. Garfinkel, M. Rosenblum, When virtual is harder than real: Security

challenges in virtual machine based computing environments, in: Proceed-

ings of the 10th Conference on Hot Topics in Operating Systems - Volume

10, HOTOS’05, USENIX Association, 2005, pp. 20–20.740

URL http://dl.acm.org/citation.cfm?id=1251123.1251143

[66] D. Perez-Botero, A brief tutorial on live virtual machine migration from a

security perspective, University of Princeton, USA, 2011.

URL https://pdfs.semanticscholar.org/b562/

a31a55998bab7fc416622e697844ae72f318.pdf745

[67] M. Aiash, G. Mapp, O. Gemikonakli, Secure live virtual machines migra-

tion: issues and solutions, in: Proceedings of the 28th International Con-

ference on Advanced Information Networking and Applications Workshops,

WAINA’14, IEEE, 2014, pp. 160–165. doi:10.1109/WAINA.2014.35.

34

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

Mostafa Noshy is a teaching assistant at Computer Engineering and Control Systems Department, Mansoura University, Egypt. He received his B.Sc. in 2014 with an overall grade of excellent with honors from Mansoura University. His research interests include cloud computing, virtualization and live virtual machine migration.

Abdelhameed Ibrahim was born in Mansoura city, Egypt, in 1979. He attended the Faculty of Engineering, Mansoura University, in Mansoura, where he received Bachelor and Master Degrees in Engineering from the electronics (Computer Engineering and Systems) department in 2001 and 2005, respectively. He was with the Faculty of Engineering, Mansoura University, from 2001 through 2007. In April 2007, he joined the Graduate School of Advanced Integration Science, Faculty of Engineering, Chiba University, Japan, as a doctor student. He received Ph.D. Degree in Engineering in 2011. His research interests are in the fields of computer vision and pattern recognition, with a special interest in material classification based on reflectance information.

Hesham Arafat Ali is a Prof. in Computer Eng. & Sys.; An assoc. Prof. in Info. Sys. and computer Eng. He

was assistant prof. at the Univ. of Mansoura, Faculty of Computer Science in 1997 up to 1999. From

January 2000 up to September 2001, he was joined as Visiting Professor to the Department of Computer

Science, University of Connecticut. From 2002–2004 he was a vice dean for student affair the Faculty of

Computer Science and Inf., Univ. of Mansoura. He was awarded with the Highly Commended Award

from Emerald Literati Club 2002 for his research on network security. He is a founder member of the

MANUSCRIP

T

ACCEPTED

ACCEPTED MANUSCRIPT

IEEE SMC Society Technical Committee on Enterprise Information Systems (EIS). He has many book

chapters published by international press and about 150 published papers in international (conf. and

journal). He has served as a reviewer for many high quality journals, including Journal of Engineering

Mansoura University. His interests are in the areas of network security, mobile agent, Network

management, Search engine, pattern recognition, distributed databases, and performance analysis.