1

Dynamic Flow Migration for Embedded Services inSDN/NFV-Enabled 5G Core Networks

Kaige Qu, Student Member, IEEE, Weihua Zhuang, Fellow, IEEE, Qiang Ye, Member, IEEE,Xuemin (Sherman) Shen, Fellow, IEEE, Xu Li, and Jaya Rao

Abstract—Software defined networking (SDN) and networkfunction virtualization (NFV) are key enabling technologies infifth generation (5G) communication networks for embeddingservice-level customized network slices in a network infrastruc-ture, based on statistical resource demands to satisfy long-termquality of service (QoS) requirements. However, traffic loads indifferent slices are subject to changes over time, resulting inchallenges for consistent QoS provisioning. In this paper, a dy-namic flow migration problem for embedded services is studied,to meet end-to-end (E2E) delay requirements with time-varyingtraffic. A multi-objective mixed integer optimization problemis formulated, addressing the trade-off between load balancingand reconfiguration overhead. The problem is transformed to atractable mixed integer quadratically constrained programming(MIQCP) problem. It is proved that there is no optimality gapbetween the two problems; hence, we can obtain the optimum ofthe original problem by solving the MIQCP problem with somepost-processing. To reduce time complexity, a heuristic algorithmbased on redistribution of hop delay bounds is proposed to find anefficient solution. Numerical results are presented to demonstratethe aforementioned trade-off, the benefit from flow migration interms of E2E delay guarantee, as well as the effectiveness andefficiency of the heuristic solution.

Index Terms—Service function chaining (SFC), network slic-ing, dynamic flow migration, end-to-end (E2E) delay, VNF statetransfer, SDN/NFV-enabled 5G networks.

I. INTRODUCTION

The service-oriented fifth generation (5G) networks willsupport new use cases and diverse services with multi-dimensional performance requirements, which cannot be sup-ported by the legacy one-size-fits-all network architecture [2].Network slicing is a promising solution to accommodate thebroad range of services over a common network infrastructure,and provides service-level performance guarantees [3]. Theemerging software defined networking (SDN) and networkfunction virtualization (NFV) technologies create new oppor-tunities for a flexible and programmable network architecture,which supports network slicing [4]–[7]. SDN decouples net-work control from data plane into a centralized control module,

This work was financially supported by research grants from HuaweiTechnologies Canada and from the Natural Sciences and Engineering ResearchCouncil (NSERC) of Canada.

Kaige Qu, Weihua Zhuang, and Xuemin (Sherman) Shen are withthe Department of Electrical and Computer Engineering, University ofWaterloo, Waterloo, ON, Canada, N2L 3G1 (emails: {k2qu, wzhuang,sshen}@uwaterloo.ca).

Qiang Ye is with the Department of Electrical and Computer Engineeringand Technology, Minnesota State University, Mankato, MN 56001 USA (e-mail: qiang.ye@mnsu.edu). Corresponding author: Qiang Ye.

Xu Li and Jaya Rao are with Huawei Technologies Canada Inc., Ottawa,ON, Canada, K2K 3J1 (emails: {Xu.LiCA, jaya.rao}@huawei.com).

This work is presented in part at IEEE ICC 2019 [1].

which enables a global network view and facilitates centralizednetwork management. End-to-end (E2E) data delivery pathscan be established by an SDN controller, by configuring for-warding rules in SDN-enabled switches via southbound proto-cols such as Openflow [8]. NFV separates network functionsfrom dedicated hardware to software instances, referred to asvirtual network functions (VNFs), hosted in NFV nodes suchas commodity servers and data centers. NFV enables cost-effective VNF placement and elastic VNF capacity scaling.

With new business models introduced in 5G networks, atenant such as a service provider requests a set of networkservices in the form of service function chains (SFCs), froman infrastructure provider (InP). An SFC is composed ofmultiple VNFs in a predefined order, to fulfill a compositeservice in E2E data delivery. Each VNF supports a dedicatedpacket processing functionality such as intrusion detectionsystem (IDS) and network address translation (NAT). ManyVNFs are state-dependent, and states are updated together withpacket header or payload processing, to guarantee accuratepacket processing. For example, a virtual IDS belonging toan SFC keeps track of pattern matchings for accurate attackdetection in subsequent packets. VNF states are stored andupdated locally in associate VNFs. Packets processed by aVNF are transmitted to next VNF in the same SFC for furtherprocessing, generating traffic between consecutive VNFs, untilthe last VNF. We refer to an SFC flow as the aggregate trafficflow traversing an SFC, and an inter-VNF subflow as the trafficbetween two consecutive VNFs in the same SFC.

The InP customizes network services over the network in-frastructure, generating service-level network slices. For eachservice (SFC), VNFs are embedded on NFV nodes, and inter-VNF subflows are routed over physical paths between the cor-responding upstream and downstream VNFs. Here, a physicalpath is a series of SDN-enabled switches and links. The VNFsand subflows are allocated certain processing resources andtransmission resources respectively, according to long-termtraffic statistics and quality of service (QoS) requirements.This process is referred to as SFC embedding [9]–[11]. We calla logical abstraction of all embedded physical paths betweentwo NFV nodes as a virtual link. After SFC embedding, pro-cessing resources of NFV nodes are virtualized and distributedamong VNFs, and virtual links are established among NFVnodes and are allocated with transmission resources.

We consider a single tenant with multiple network slicesunder its management [3]. The slices are required to bemutually isolated from each other in terms of service per-formance, e.g., E2E delay [12], [13]. However, during the

2

operation of network slices, traffic arrivals of each servicefluctuate over time, due to dynamic user subscription andtime-varying traffic flows from source nodes, which resultsin imbalanced resource usage on NFV nodes and virtual linksfrom time to time. The mismatch between traffic load andresource availability is detrimental to both service performanceand resource utilization. Allocating dedicated resources toeach slice according to peak demands guarantees performanceisolation at the cost of resource overprovisioning. For efficientresource utilization, resource sharing among slices is desired,i.e., processing resources at NFV nodes and transmissionresources on virtual links are shared among traffic flows ofmultiple services. Due to the chaining nature of SFCs, oneservice can share resources with different services at differentNFV nodes and virtual links traversed along its E2E path,making performance isolation challenging to achieve. Withinan NFV-MANO reference architecture, a resource orchestrator(RO) is responsible for orchestrating resources among slices,on behalf of the tenant [3]. Each slice is subject to resourcecapacity scaling via a network service orchestrator (NSO) [3].Hence, a dynamic resource management scheme is requiredfor the RO to accommodate traffic dynamics and providecontinuous QoS performance guarantees for multiple services.

In this paper, we study a delay-aware flow migrationproblem for delay-sensitive services in a processing resourcelimited network, to guarantee average delay isolation amongservices within maximal tolerable service downtime. Theplacement of VNFs is adjusted over time, with scaled pro-cessing resources allocated to each VNF. The associate statesat migrated VNFs are transfered to target NFV nodes forconsistency. For a subflow, if either its upstream or down-stream VNF migrates to an alternative NFV node, it should beremapped to an alternative virtual link accordingly. The goal isto achieve efficient utilization of processing resources at NFVnodes with minimum transmission resource overhead incurredby state transfers. With the consideration that not every twoNFV nodes are directly connected by virtual links, the numberof extra virtual links required for flow rerouting is minimizedto reduce signaling overhead. The problem is formulated asa multi-objective mixed integer optimization problem. Fordelay awareness, under the assumption of prior knowledgeof time-varying traffic rates, average E2E delay requirementsare included in constraints based on delay modeling. Dueto several quadratic constraints, an optimal solution to theproblem is difficult to obtain using solvers such as Gurobi [14].We transform the original problem into a tractable mixedinteger quadratically constrained programming (MIQCP) prob-lem. Although the two problems are not equivalent, it isproved that there is a zero optimality gap between them.Given an MIQCP optimum, the optimum of the original prob-lem is obtained through a proposed mapping algorithm. TheMIQCP transformation, together with the mapping algorithm,gives an optimal solution, but time complexity is high dueto NP-hardness of the MIQCP problem. Therefore, a low-complexity heuristic algorithm based on redistribution of hopdelay bounds is proposed to obtain a sub-optimal solution tothe original problem.

The rest of this paper is organized as follows. Section II

gives an overview of background and related work. Thesystem model is presented in Section III, and a delay-awareflow migration problem is formulated in Section IV. SectionV presents the MIQCP problem transformation, and derivesthe optimality gap between the transformed problem and theoriginal problem. A low-complexity heuristic algorithm isproposed in Section VI. Performance evaluation for both theMIQCP and heuristic solutions is presented in Section VII,and conclusions are drawn in Section VIII. A list of importantnotations is given in Table I.

II. BACKGROUND AND RELATED WORK

Traffic engineering (TE) has been extensively investigated,to find paths for date delivery from source to destination withinlink capacity [15], [16]. A cost function, such as a piece-wiselinear increasing and convex function of link utilization, canbe used to penalize high link utilization near capacity. Thetraditional TE ensures that no packets get sent across over-loaded links, by minimizing link utilization costs. Flow mi-gration, i.e., steering traffic flows of embedded SFCs throughalternative NFV nodes and virtual links, is a TE approach forelastic SFC provisioning [17]. Similarly, maximum loadingon NFV nodes can be minimized, to achieve load balancingover processing resources. However, traditional TE methodscannot be directly applied due to the following reasons. First,candidate paths for an SFC flow must traverse through NFVnodes for processing. In traditional TE problems, a flow isa source-destination pair without a predefined sequence ofintermediate processing nodes. Second, the transfer of VNFstates should be considered, since simply rerouting in-progressflows on a state-dependent VNF to an alternative NFV nodeleads to state inconsistency, causing processing inaccuracy.Some frameworks such as OpenNF are proposed to solve thestate inconsistency problem, by not only migrating packetsof the rerouted flow but also transferring the associate VNFstates [18]–[20]. In Co-Scalar [21], parallel state transfer isproposed for an SFC with multiple state-dependent VNFs.Instead of sequentially transferring the states of each VNF,Co-Scalar transfers all VNF states in parallel, thus greatlyreducing latency at the cost of transmission resources.

Dynamic VNF operations, including horizontal scaling, ver-tical scaling, and migration, are widely employed to provideelastic processing resource provisioning [22]. In this paper,both vertical scaling and migration are employed to provideelasticity under the assumption that the total number of VNFinstances is unchanged. In existing studies on dynamic SFCembedding, the time-varying processing and transmission re-source demands are assumed known a priori, based on whichVNFs are placed on alternative NFV nodes, and inter-VNFsubflows are rerouted to different physical paths [23]–[25].QoS requirements are expressed in such a way that the time-varying resource demands are satisfied without violating theresource capacity. In [26], both resource capacity constraintsand delay constraints are included in problem formulation,without load dependent queueing delay. In our study, queueingdelay is considered and the delay isolation issue with time-varying traffic load is addressed. Existing studies take into con-sideration of the reconfiguration overhead for flow migration,

3

Physical

Link

SDN-Enabled

Switch

NFV Node with

Multiple VNFs

Physical Network with

embedded SFCs

SFC 1

SFC 2

SFC 3

SDN-Enabled

Edge Switch

Fig. 1: A physical network with embedded SFCs.

which is modeled as a weighted number of reconfigured NFVnodes and physical links [24], or the total revenue loss dueto throughput loss within a constant service downtime [25],or the time duration for all state transfers associated withflow migration [27]. One performance metric for migrationis the maximum allowable downtime within certain timeduration [28]. Under the assumption that the time intervalfor flow migration is much larger than state transfer time, weconsider the total transmission resource overhead incurred bystate transfers, within a maximal tolerable service downtimein one service interruption.

III. SYSTEM MODEL

A. Physical Network

The physical network consists of links and SDN-enablednodes, including switches and NFV nodes. Switches forwardtraffic from incoming links to outgoing links. Some switchesact as edge switches for service access. NFV nodes haveboth forwarding and processing capabilities, each supportingmultiple VNFs subject to resource constraints. There are anumber of SFCs embedded in the physical network, each forone service request, as shown in Fig. 1.

B. Service Requests

A time-slotted system is considered, with integer k denotingthe k-th time interval over which a flow migration plan remainsunchanged. LetR denote the set of embedded service requests.A service request, r ∈ R, is represented in the form of SFC.It originates from source node a(r) and traverses through Hr

VNFs in sequence towards destination node b(r), with averageE2E delay requirement D(r), maximal tolerable downtime∆(r) in one service interruption, average packet size σ(r) inbit, and time-varying traffic rate λ(r)(k) in packet/s. Underthe assumption that flow migration is not frequent and thetime interval is sufficiently large, traffic arrival of SFC r ∈ Rduring time interval k is modeled as Poisson with rate λ(r)(k)packet/s. Under the assumption that the time interval is muchlarger than maxr∈R∆(r), the experienced service downtime ismuch shorter than stable service operation time for any service.Let Hr = {1, · · · , Hr}, and denote the h-th (h ∈ Hr) VNFin SFC r as V (r)

h . Let V (r)0 and V

(r)Hr+1 be dummy VNFs in

SFC r, locating at source node a(r) and destination node b(r)

respectively. Let V be a set containing all VNFs belongingto different SFCs, with (r, h) ∈ V denoting the h-th VNF inSFC r. Let A be a set of edge switches hosting all dummy

TABLE I: List of important notations

Parameters

Ci Processing capacity of NFV node i ∈ I in cycle/sD(r) Average E2E delay requirement for SFC r ∈ RHr Number of VNFs in SFC r ∈ RL

(r)h The h-th subflow in SFC r ∈ RP rh

i Processing density of NFV node i ∈ I for VNF V (r)h

Ti CPU polling period of the CPU in NFV node i ∈ IV

(r)h The h-th VNF in SFC r ∈ RW Context switching time overhead in a CPU polling period

Z(r)h (k) State size of VNF V (r)

h in time interval kβli Whether i ∈ I ∪ A is the starting point of virtual link l ∈ L

∆(r) Maximal tolerable service downtime for SFC r ∈ Rλ(r)(k) Time-varying traffic rate in packet/s for SFC r ∈ Rσ(r) Average packet size in bit for SFC r ∈ Rϕl

i Whether i ∈ I ∪ A is the ending point of virtual link l ∈ L

Decision variables

Brhi,j(k) Transmission resource overhead for transferring the state of VNF V (r)

hfrom NFV node i to NFV node j

drhi (k) The (dummy) processing delay on VNF V (r)h at NFV node i

mrhi,j(k) Whether the mapped NFV node for VNF V (r)

h changes from NFVnode i to NFV node j

xrhi (k) Whether VNF V (r)

h is mapped to NFV node iyrhij (k) Whether subflow L

(r)h is mapped to an extra virtual link between

i ∈ I ∪ A and j ∈ I ∪ Awi(k) Whether context switching happens at NFV node iη(k) The maximum NFV node loading factorµrhi (k) Processing rate allocated to VNF V (r)

h by NFV node iτrhi,j (k) The (dummy) delay to transfer the state of VNF V

(r)h from NFV

node i to NFV node j

VNFs. The h-th (h ∈ {0}∪Hr) inter-VNF subflow in SFC r,i.e., the subflow between upstream (dummy) VNF V

(r)h and

downstream (dummy) VNF V(r)h+1, is denoted as L(r)

h .

C. Abstraction of Virtual Resource Pool

In SFC embedding, we assume that each VNF is embeddedto a single NFV node, every dummy VNF is hosted at oneedge switch, and every subflow is allowed to be embedded tomultiple physical paths between the locations of its upstreamand downstream (dummy) VNFs. Fig. 1 shows a physicalnetwork with three embedded SFCs in single-path routing.We call a logical abstraction of all embedded physical pathsbetween two NFV nodes or between one NFV node andone edge switch as a virtual link. A virtual resource pool isabstracted from the physical network with embedded SFCs,represented as a directed graph G = {I ∪ A,L}, where I isa set of all NFV nodes, A is a set of edge switches hostingdummy VNFs, and L is a set of virtual links. For virtual linkl ∈ L, we use binary parameters, {βli} and {ϕli}, to describeits location and direction, with βli = 1 if i ∈ I ∪ A is itsstarting point and ϕli = 1 if i ∈ I ∪ A is its ending point. Itis possible that G is not fully connected. Assume that thereare sufficient transmission resources in the physical network.We can increase resources on existing virtual links, and findpaths with enough resources for extra virtual links. Hence, weconsider a processing resource limited virtual resource pool.

D. Processing Resource Sharing

The processing resource capacity Ci of NFV node i ∈ I isits maximum supporting CPU processing rate in cycle/s. Forone packet/s of processing rate, the CPU resource demand ona certain NFV node depends on many factors, including the

4

t1 t2

W

Ti

… …

Ti: polling period W: context switching time overhead

t1: time share for flow 1 t2: time share for flow 2 tR: residual time share

W W WW

tR

Ti

tR

t1 t2 t1 t2

Fig. 2: A CPU polling scheme with two flows.

f1 f3 f4f2

f3

f2

①

② State Transfers

Old Paths

New Paths

source destination

Fig. 3: An illustration for flow migration and state transfer.

packet size, the type of function, the packet I/O scheme, andthe virtualization technology [29]–[31]. We summarize all thefactors into two categories: VNF dependent, and NFV nodedependent. We define processing density of NFV node i forVNF V

(r)h as P rhi (in cycle/bit), which is the CPU resource

demand (in cycle/s) of VNF V (r)h on NFV node i for one bit/s

of processing rate. Accordingly, P rhi σ(r) is the processingdensity of NFV node i for VNF V

(r)h in cycle/packet. A

CPU polling scheme is employed for resource sharing amongmultiple flows, as illustrated in Fig. 2, in which two flow-specific processing queues are polled for service. Each queuegets a portion of CPU resources which is linear with itsallocated CPU time share in a polling period. The pollingscheme introduces multi-task context switching time overhead,due to extra CPU time spent on saving and loading contextsbetween every two consecutive tasks [32]. Here, processingpackets from a certain processing queue is a task. The pollingperiod Ti and the context switching time overhead W in NFVnode i are constants. Under the assumption that resourcesare infinitely divisible, generalized processor sharing (GPS)is a benchmark resource allocation scheme to achieve QoSisolation and multiplexing gain among flows [12], [13]. Basedon GPS discipline, the two flows are guaranteed minimum pro-cessing rates (in packet/s) of µ1 = t1Ci

TiP 1i σ

(1) and µ2 = t2Ci

TiP 2i σ

(2)

respectively, where t1 + t2 + 2W = Ti − tR, and tR is theresidual time share in a polling period. Define loading factorof NFV node i, denoted by ηi, as the percentage of allocatedtime shares plus context switching time overhead in a pollingperiod of NFV node i.

E. Reconfiguration Overhead

When an SFC flow migrates at a state-dependent VNF,the VNF is remapped to an alternative NFV node, with theassociate states transferred to the target NFV node. Fig. 3illustrates the flow migration and associate state transfers,where two VNFs are remapped, and two state transfers aretriggered correspondingly. Packet processing is halted duringstate transfer, incurring a service downtime. Let Z(r)

h (k) (inbit) be the time-varying state size of VNF V (r)

h , whose value attime interval k is monitored by the SDN controller. For a state

transfer at VNF V(r)h , Z(r)

h (k) is the product of state transferdelay and consumed transmission resources (in bit/s) [27]. Fora remapped SFC with multiple state transfers, we use parallelstate transfer in data plane in which all state transfers can takeplace simultaneously [21]. Then, the service downtime, whichis the maximum state transfer delay along the E2E path, ismuch less than that of sequential state transfer, at the costof transmission resources. Within a maximal tolerable servicedowntime, the total transmission resource overhead incurredby state transfers should be minimized. We assume that thetransmission resource overhead for each state transfer is notless than Bmin =

min(Z(r)h (k))

max(∆(r)).

For a subflow, if its upsteam or downstream VNF migratesto an alternative NFV node, the subflow should be remappedto an alternative virtual link accordingly. After a successfulremapping, transmission resources allocated to the subflow arereleased from the old virtual link. However, it is possible thatNFV nodes in the virtual resource pool are not fully connected.Assume that the SDN controller can find physical paths withenough resources for an extra virtual link. Forwarding ruleconfiguration along the physical paths incurs signaling over-head between the SDN controller and SDN-enabled switches.

In summary, the reconfiguration overhead due to flow migra-tion is described in two parts: the total transmission resourceoverhead incurred by state transfers, and the total signalingoverhead for configuring extra virtual links required for flowrerouting. The latter is assumed to be linear with the totalnumber of extra virtual links required for flow rerouting.

IV. PROBLEM FORMULATION

Problem 1. Consider a processing resource limited virtualresource pool. Assume that packet processing time at anNFV node for an SFC is exponentially distributed [33], [34].During time interval k, traffic arrival of SFC r ∈ R is Poissonwith rate λ(r)(k) packet/s. The rate can be predicted at the endof time interval (k−1) based on measurements and historicalinformation [17], [35]. A delay-aware flow migration problemis to 1) find the remapping between VNFs and NFV nodes intime interval k, based on the old mapping in time interval(k − 1), and 2) scale the processing resources allocated toVNFs, to satisfy average E2E delay requirements withoutviolating processing resource constraints. The objective, fortime interval k, is to achieve load balancing among NFVnodes, with minimal reconfiguration overhead:

O(k) = α1η(k) + α2

∑(r,h)∈V

∑i,j∈I

Brhi,j(k)

Bmin

+ α3

∑r∈R

∑h∈{0}∪Hr

∑i,j∈I∪A

yrhij (k).(1)

In objective function (1), there are several decision variablesfor time interval k: 1) continuous variable η(k) ∈ [0, 1] formaximum loading factor among all NFV nodes during intervalk; 2) nonnegative continuous variable set B(k) = {Brhi,j(k)},with Brhi,j(k) being the transmission resource overhead totransfer the state of VNF V

(r)h from NFV node i ∈ I during

interval (k − 1) to NFV node j ∈ I during interval k;3) binary variable set y(k) = {yrhij (k)}, with yrhij (k) = 1

5

if subflow L(r)h is mapped to an extra virtual link between

i, j ∈ I∪A during interval k, and yrhij (k) = 0 otherwise. Notethat α1, α2, α3 are tunable weights to control the priority of thethree components, with α1+α2+α3 = 1. In the right hand sideof (1), the first term is the cost for imbalanced loading amongNFV nodes since minimizing η(k) achieves load balancingamong all the NFV nodes, the second term is the cost for theoverall normalized transmission resource overhead due to statetransfers with a normalization ratio of 1

Bmin, the third term

is the cost for extra virtual links required by flow rerouting.The normalization makes the three components in objectivefunction (1) comparable, based on which α1, α2, α3 can beselected on the same order of magnitude. A component in (1)is ignored if the corresponding weight is set to 0. If all weightsare positive, all components in (1) are jointly optimized.

In terms of constraints, we start from node mapping con-straints. Define binary decision variable set x(k) = {xrhi (k)}for interval k, with xrhi (k) = 1 if (dummy) VNF V

(r)h is

mapped to node i ∈ I ∪A during interval k, and xrhi (k) = 0

otherwise. As VNF V(r)h should be mapped to exactly one

NFV node in I, we have∑i∈I

xrhi (k) = 1, ∀(r, h) ∈ V. (2)

For dummy VNFs, i.e., source and destination nodes, theirphysical locations are fixed and confined by

xr0a(r)(k) = 1, r ∈ R (3a)

xr0i (k) = 0, r ∈ R, i ∈ I ∪ A\a(r) (3b)

xr(Hr+1)

b(r)(k) = 1, r ∈ R (3c)

xr(Hr+1)i (k) = 0, r ∈ R, i ∈ I ∪ A\b(r). (3d)

The next constraints are related to transmission resourceoverhead for state transfer. From interval (k−1) to interval k,the set representing VNF to NFV node mapping changes fromx(k−1) = {xrhi (k−1)} to x(k) = {xrhi (k)}, where x(k−1)is known in interval k. We introduce binary decision variableset m(k) = {mrh

i,j(k)} for interval k, with mrhi,j(k) = 1 if the

mapped NFV node for VNF V(r)h changes from NFV node

i ∈ I during interval (k − 1) to NFV node j ∈ I duringinterval k, and mrh

i,j(k) = 0 otherwise. Hence, there is arelationship constraint among {mrh

i,j(k)} (i 6= j), {xrhi (k−1)}and {xrhj (k)}, given by

mrhi,j(k)=xrhi (k−1)xrhj (k), ∀(r,h)∈V,∀i∈I,∀j∈I\{i}. (4)

Also, we have

mrhi,i(k) = 0, ∀(r, h) ∈ V, ∀i ∈ I. (5)

According to the definition of Brhi,j(k), we have

0 ≤ Brhi,j(k) ≤ mrhi,j(k)M, ∀(r, h) ∈ V, ∀i, j ∈ I (6)

where M is a big-M constant to guarantee that Brhi,j(k)=0 ifmrhi,j(k) = 0. Let τ (k) = {τ rhi,j (k)} be a positive continuous

decision variable set for interval k, with τ rhi,j (k) denoting the(dummy) delay to transfer state of VNF V (r)

h from NFV nodei ∈ I during interval (k − 1) to NFV node j ∈ I during

interval k. It follows that

τ rhi,j (k) =Z

(r)h (k)

Brhi,j(k) + ε, ∀(r, h) ∈ V, ∀i, j ∈ I (7)

where 0 < ε � 1 is a constant to avoid τ rhi,j (k) being unde-termined, and τ rhi,j (k) is a dummy delay only if mrh

i,j(k) = 0.Moreover, τ rhi,j (k) has an upper bound

0<τrhi,j(k)≤mrhi,j(k)∆(r)+[1−mrhi,j(k)

]Z(r)h (k)

ε,∀(r,h)∈V,∀i,j∈I. (8)

If mrhi,j(k) = 1, the upper bound is the corresponding maximal

tolerable service downtime ∆(r); otherwise, it is Z(r)h (k)

ε .For constraints related to processing resource scaling, let

µ(k) = {µrhi (k)} be a nonnegative continuous decision vari-able set for interval k, with µrhi (k) being the processing ratein packet/s allocated to VNF V (r)

h by NFV node i ∈ I duringinterval k. It should be lower bounded by xrhi (k)λ(r)(k) due tothe queue stability requirement and upper bounded by xrh

i (k)Ci

P rhi σ(r)

due to the limited processing capacity, given by

xrhi (k)λ(r)(k)≤µrhi (k)≤ xrhi (k)CiP rhi σ(r)

, ∀(r, h)∈V, ∀i ∈ I. (9)

Let wi(k) be a binary decision variable with wi(k) = 1 ifcontext switching happens at NFV node i during interval k,i.e., there are at least two VNFs mapped to NFV node i, andwi(k) = 0 otherwise. The loading factor of NFV node i duringinterval k, denoted by ηi(k), consists of two parts, with amaximum value equal η(k), which is upper bounded by apredefined constant ηU (0 < ηU ≤ 1), given by∑(r,h)∈V

[P rhi σ(r)µrhi (k)

Ci+wi(k)x

rhi (k)W

Ti

]≤η(k)≤ηU ,∀i∈I (10)

where both useful time and context switching time overhead ofCPU resources in a polling period are taken into consideration.The left hand side of (10) is the expression for ηi(k). The valueof wi(k) is confined by an inequality constraint of∑

(r,h)∈V xrhi (k)−1

|V|≤wi(k)≤

∑(r,h)∈V x

rhi (k)

A, ∀i ∈ I (11)

where A is an arbitrary value from (1, 2].For average E2E delay constraints, let d(k) = {drhi (k)}

be a positive continuous decision variable set for interval k,with drhi (k) denoting the average (dummy) delay on the queueassociated with VNF V (r)

h at NFV node i. With Poisson trafficarrival and exponential packet processing time, the processingsystem is an M/M/1 queue. Then, drhi (k) is given by

drhi (k)=1

µrhi (k)−xrhi (k)λ(r)(k)+ε, ∀(r, h)∈V, ∀i∈ I (12)

where drhi (k) is a dummy delay only if xrhi (k) = 0. There isan upper bound constraint for drhi (k), explicitly showing itsrelationship with xrhi (k):

0<drhi (k)≤xrhi (k)D(r)+[1−xrhi (k)

]1ε, ∀(r,h)∈V,∀i∈I. (13)

For QoS satisfaction, the average E2E delay of SFC r ∈ Rshould not exceed upper bound D(r),

6

∑h∈Hr

∑i∈I

xrhi (k)drhi (k) ≤ D(r), ∀r ∈ R. (14)

To decide whether a subflow should be mapped to an extravirtual link, we consider two cases. In the first case, we have

yrhij (k) = 1−∑l∈L

βliϕlj x

rhi (k)x

r(h+1)j (k),

∀r ∈ R,∀h ∈ {0} ∪ Hr,∀i ∈ I ∪ A,∀j ∈ I ∪ A\{i}(15)

to ensure that yrhij (k) equal 0 if (dummy) VNF V(r)h and

(dummy) VNF V(r)h+1 are mapped to node i ∈ I ∪ A and

node j ∈ I ∪ A\{i} between which a virtual link exists. Inthe second case, we have

yrhii (k)=1−xrhi (k)xr(h+1)i (k),∀r∈R,∀h∈{0}∪Hr,∀i∈I∪A (16)

to ensure that yrhii (k) equal 0 if (dummy) VNF V(r)h and

(dummy) VNF V (r)h+1 are mapped to the same node i ∈ I ∪A.

In summary, the optimization problem ismin

η(k),B(k),y(k),x(k),m(k),τ (k),µ(k),w(k),d(k)

O(k)(17a)

s.t. (2)− (16) (17b)d(k), τ (k) > 0, (17c)x(k),w(k),m(k),y(k) ∈ {0, 1}. (17d)

Remark 1. Problem (17) is non-convex due to constraints (7),(10), (12), (14), (15), and (16).

V. OPTIMAL MIQCP SOLUTION

In problem (17), quadratic constraints (10), (14) (15), and(16) can be transformed to equivalent linear forms using thebig-M method. Quadratic constraints (7) and (12) cannot belinearized due to product terms of two continuous variables,but they can be replaced by combinations of linear constraintsand rotated quadratic cone constraints. The new problem withtransformed and replaced constraints is an MIQCP problem,which is not equivalent to the original problem. In this section,we discuss the relationship between both problems.

By introducing an auxiliary nonnegative continuous deci-sion variable set, θ(k) = {θi(k)}, we linearize constraint (10)based on the big-M method with M = |V|, given by∑(r,h)∈V

P rhi σ(r)µrhi (k)

CiTi + θi(k)W ≤ η(k)Ti, ∀i ∈ I (18a)

∑(r,h)∈V

xrhi (k)−|V|[1−wi(k)]≤θi(k)≤∑

(r,h)∈V

xrhi (k), ∀i∈I (18b)

0 ≤ θi(k) ≤ |V|wi(k), ∀i ∈ I. (18c)

By introducing an auxiliary nonnegative continuous deci-sion variable set, γ(k) = {γrhi (k)}, we linearize constraint(14) based on the big-M method with M = 1

ε as∑h∈Hr

∑i∈I

γrhi (k) ≤ D(r), ∀r ∈ R (19a)

drhi (k)− 1

ε

[1−xrhi (k)

]≤γrhi (k)≤drhi (k), ∀(r,h)∈V,∀i∈I (19b)

0 ≤ γrhi (k) ≤ 1

εxrhi (k), ∀(r, h) ∈ V,∀i ∈ I. (19c)

By introducing an auxiliary binary decision variable set,ξ(k) = {ξrhij (k)}, we get an equivalent linear form ofconstraint (15) and constraint (16) for ∀r ∈ R,∀h ∈ {0}∪Hr,and ∀i ∈ I ∪ A, given by

ξrhij (k) ≤ xrhi (k), ∀j ∈ I ∪ A (20a)

ξrhij (k) ≤ xr(h+1)j (k), ∀j ∈ I ∪ A (20b)

ξrhij (k) ≥ xrhi (k) + xr(h+1)j (k)− 1, ∀j ∈ I ∪ A (20c)

yrhij (k) = 1−∑l∈L

βliϕlj ξrhij (k), ∀j ∈ I ∪ A\{i} (20d)

yrhii (k) = 1− ξrhii (k). (20e)

Proposition 1. With linearized constraints (18), (19) and (20),problem (17) can be transformed to an MIQCP problem, ifconstraint (7) is replaced by

ρrhi,j(k) = Brhi,j(k) + ε, ∀(r, h) ∈ V, i, j ∈ I (21a)

ρrhi,j(k) ≥ ε, ∀(r, h) ∈ V, i, j ∈ I (21b)

τ rhi,j (k)ρrhi,j(k) ≥ z(r)h (k)2, ∀(r, h) ∈ V, i, j ∈ I (21c)

z(r)h (k) =

√Z

(r)h (k), ∀(r, h) ∈ V (21d)

where ρrhi,j(k) = {ρrhi,j(k)} and z(r)h (k) = {z(r)

h (k)} areauxiliary continuous decision variable sets, and if constraint(12) is replaced by

πrhi (k)=µrhi (k)−xrhi (k)λ(r)(k)+ε, ∀(r,h)∈V,∀i∈I (22a)πrhi (k) ≥ ε, ∀(r, h) ∈ V,∀i ∈ I (22b)drhi (k)πrhi (k) ≥ c2, ∀(r, h) ∈ V,∀i ∈ I (22c)c = 1 (22d)

where π(k) = {πrhi (k)} is an auxiliary continuous decisionvariable set and c is an auxiliary continuous decision variable.The optimality gap between the two problems is zero, i.e., anoptimum of problem (17) is either a unique optimum or oneof multiple optimal solutions to the MIQCP problem. Givenan MIQCP optimum (“?”), an optimum of problem (17) (“∗”)can be obtained by Algorithm 1.

Proof. The fundamental difference between the MIQCP prob-lem and problem (17) lies in “≥” signs in rotated quadraticcone constraints (21c) and (22c). If both constraints are activein an MIQCP optimum, i.e., all the “≥” signs achieve equality,the MIQCP optimum is also an optimum of problem (17).Next, we discuss how the “≥” signs affect the optimum.

First, assume that there is an inactive constraint (21c) inan MIQCP optimum, i.e., τ rhi,j

?(k)[Brhi,j

?(k) + ε

]> Z

(r)h (k).

If Brhi,j?(k) = 0, it does not affect the objective value.

Thus, we consider only the case with Brhi,j?(k) > 0. If

Brhi,j?(k) is replaced by Brhi,j

◦(k), with Brhi,j

◦(k)<Brhi,j

?(k)

and τ rhi,j?(k)[Brhi,j

◦(k)+ε

]=Z

(r)h (k), all constraints are still

satisfied. The objective value is unchanged if α2=0, in whichcase [τ rhi,j

?(k), Brhi,j

◦(k)] is an optimal pair in another MIQCP

optimum. Otherwise (α2 > 0), the objective value can befurther reduced, inferring that the assumption must be false.

Second, assume that there is an inactiveconstraint (22c) in an MIQCP optimum, then

7

Algorithm 1: Post-processing to MIQCP optimum

1 Input: η?, B?, y?, x?, m?, τ ?, µ?, w?, d?.2 Initialization (∗ = ?).3 for (r, h) ∈ V , i, j ∈ I do4 if τ rhi,j

?(k)ρrhi,j

?(k) > (z

(r)h (k)?)2 and

Brhi,j?(k) > 0, then Brhi,j

∗(k) =

Z(r)h (k)

τrhi,j

?(k)− ε

5 for (r, h) ∈ V , i ∈ I do6 if drhi

?(k)πrhi

?(k) > (c?)2 and xrhi

?(k) == 1,

then7 if α1 > 0, then8 drhi

∗(k) = 1

µrhi

?(k)−xrhi

?(k)λ(r)(k)+ε

9 if α1 == 0, then10 µrhi

∗(k) = 1

drhi?(k)

+ xrhi?(k)λ(r)(k)− ε

11 if α1 == 0, then calculate η∗(k)12 Output: η∗, B∗, y∗, x∗, m∗, τ ∗, µ∗, w∗, d∗.

drhi?(k)[µrhi

?(k)− xrhi

?(k)λ(r)(k) + ε

]> 1. Similarly,

we consider only the constraint with xrhi?(k) = 1. There are

four cases, depending on α1 and ηi(k).Case 1: α1 > 0, and NFV node i is the only one with

a loading factor of η?(k) (i.e., dominating NFV node). Ifµrhi

?(k) is replaced by µrhi

◦(k), with drhi

?(k)[µrhi

◦(k) −

xrhi?(k)λ(r)(k)+ε]=1, all constraints are satisfied but η?(k)

can be further reduced. Hence, the assumption must be false;Case 2: α1 > 0 and ηi(k) < η?(k) (i.e., non-dominating

NFV node). If drhi?(k) is replaced by drhi

◦(k), with

drhi◦(k)[µrhi

?(k)−xrhi

?(k)λ(r)(k)+ε] = 1, all constraints are

satisfied with the objective value unchanged. Thus, [drhi◦(k),

µrhi?(k)] is an optimal pair in another MIQCP optimum;

Case 3: α1 > 0, and there is more than one NFV nodeincluding NFV node i with loading factor η?(k). There mustbe at least one of them satisfying an active constraint (22c).One such NFV node is selected as the dominating NFV node,and others are seen as non-dominating NFV nodes;

Case 4: α1 = 0, and η(k) is not optimized. If we replaceµrhi

?(k) by µrhi

◦(k), all constraints are satisfied and the

objective value is unchanged. Thus, [drhi?(k), µrhi

◦(k)] is an

optimal pair in another MIQCP optimum.In summary, an MIQCP optimum with inactive constraints

in (21c) and (22c) can always be mapped to another MIQCPoptimum with active constraints in (21c) and (22c), withoutaffecting other constraints and the objective value. The mappedMIQCP optimum is also the optimum of problem (17). Themapping algorithm is provided in Algorithm 1.

Remark 2. The MIQCP problem is NP-hard.

Proof. To prove the NP-hardness, it is sufficient to consider aspecial case in which services with D(r) →∞ are embeddedin a fully-connected virtual resource pool. We also considerzero VNF state size, zero context switching time overhead,and sufficiently large processing resource capacity for eachNFV node holding all VNFs without overloading [23], [25].

In such a case, the MIQCP problem can be reduced from amultiprocessor scheduling problem [36]. The multiprocessorscheduling problem minimizes the maximum load among anumber of processors which are assigned with a number oftasks with different loads, which is proved to be NP-hard.

VI. HEURISTIC SOLUTION

Although problem (17) can be solved by the optimalMIQCP solution according to Proposition 1, the computationaltime is high due to NP-hardness of the MIQCP problem. Inthis section, we propose a low-complexity modular heuristicsolution to problem (17). We consider only the case whereall components in objective function (1) are jointly optimized,i.e., α1, α2, α3 > 0. In this case, we assume that one VNFmigration is penalized more than imbalanced loading (i.e.,η(k) reaching its upper bound ηU ). Then, the condition ofα1ηU < α2 should be satisfied in the worst case, if all VNFmigrations incur the same transmission resource overhead forstate transfer and require no extra virtual links for flow rerout-ing. Accordingly, in the proposed algorithm, we first minimizethe number of overloaded NFV nodes with loading factorsgreater than ηU , and make migration decisions at overloadedNFV nodes, after which η(k) is equal to ηU . Afterwards,η(k) is further reduced for load balancing. The algorithm isinsensitive to α1 but sensitive to α2

α3, due to reconfiguration

overhead aware migration decisions. Therefore, it provides asub-optimal solution to problem (17) with α1ηU < α2.

A. Overview

The heuristic algorithm is to determine a migration andresource allocation plan for interval k in the presence of pre-dicted traffic variations (i.e., from {λ(r)(k−1)} to {λ(r)(k)}).We first find if and where NFV node resource overloadingwould happen due to traffic variations, based on three factors.The first is node mapping, denoted by {xrhi (k)}; the secondis hop (VNF) delay bounds, denoted by {Drh(k)}, with∑h∈Hr

Drh(k) = D(r); the third is NFV node loadingfactor threshold, denoted by ηth. With current node mappingand hop delay bounds, i.e., xrhi (k) = xrhi (k − 1) andDrh(k) =

∑i∈I x

rhi (k − 1)drhi (k − 1), we calculate NFV

node loading factors with traffic rates, {λ(r)(k)}, based on theM/M/1 queueing model. By comparing the NFV node loadingfactors with threshold ηth (initial value set as ηU ), a set ofoverloaded NFV nodes is identified as potential bottlenecks.

1) Reconfiguration overhead reduction: Even if potentialbottlenecks are identified, it is possible that migration is notnecessary. For a given ηth value, how an E2E delay require-ment is decomposed into hop delay bounds makes a differenceon the number of overloaded NFV nodes. By making hopdelay bounds less stringent on overloaded NFV nodes andmore stringent on underloaded NFV nodes, it is possibleto reduce the number of overloaded NFV nodes. The basicidea is as follows. If an SFC traverses both overloaded andunderloaded NFV nodes, loading factors of the underloadedones are increased to ηth, by shrinking corresponding hopdelay bounds, and loading factors of the overloaded onesare decreased, by enlarging corresponding hop delay bounds.

8

The strategy is referred to as delay scaling. Delay scaling isperformed iteratively, until there is no SFC traversing bothoverloaded and underloaded NFV nodes. The iterative delayscaling procedure with given threshold, ηth, is referred to asredistribution of hop delay bounds.

If the number of overloaded NFV nodes is reduced tozero after redistribution of hop delay bounds, no migrationis required. Otherwise, migration is necessary to overcome re-source overloading. Migration decisions are made sequentially,i.e., only a pair of variables in set {xrhi (k)} is updated in onemigration decision, each followed by a redistribution of hopdelay bounds, until no more migration is required.

With alternate migration decision and redistribution of hopdelay bounds, reconfiguration overhead is greedily reduced intwo ways. One is the potential reduction of overloaded NFVnodes by redistribution of hop delay bounds. The other is con-sideration of reconfiguration overhead in migration decision.

2) Load balancing: If no potential bottlenecks are detectedor all detected potential bottlenecks are removed by migrationand redistribution of hop delay bounds, load balancing isthe only remaining objective. NFV node loading factors aregradually balanced by iterative redistribution of hop delaybounds with threshold updating. The threshold, ηth, is updatedfrom binary search, until it reaches sufficient precision.

More details on redistribution of hop delay bounds are givenin Subsection VI-B, with pseudo code presented in Algo-rithm 2. Migration decision is discussed in Subsection VI-C,and threshold updating is discussed in Subsection VI-D. Fi-nally, the heuristic algorithm is presented in Algorithm 3.

B. Redistribution of hop delay bounds

Classification. With given threshold ηth and a given setof {Drh(k)}, the loading factor of NFV node i ∈ I in thepresence of traffic variations is calculated as

ηi(k) =∑

(r,h)∈V

(P rhi σ(r)µrhi (k)

Ci+wi(k)xrhi (k)W

Ti

)(23)

where wi(k) is calculated from (11) and µrhi (k) is given by

µrhi (k) =

(λ(r)(k) +

1

Drh(k)

)xrhi (k). (24)

Three sets of NFV nodes are identified: O = {i ∈ I |ηi(k) >ηth} consisting of overloaded NFV nodes, U = {i ∈I |ηi(k) < ηth} for underloaded NFV nodes, and Q =

{i ∈ I |ηi(k) = ηth}. Let binary variable X(r)i (k) indicate

whether SFC r traverses NFV node i during interval k,with X

(r)i (k) = 1 if

∑h∈Hr

xrhi (k) > 0, and X(r)i (k) = 0

otherwise. Let f (r)1 be a binary flag indicating whether SFC

r traverses any overloaded NFV nodes, with f(r)1 = 1 if∑

i∈OX(r)i (k) > 0 , and f

(r)1 = 0 otherwise. Set U is

divided into two subsets, i.e., U = UU ∪ UO, where UU ={i ∈ U |

∑r∈RX

(r)i (k)f

(r)1 = 0} is a set of underloaded

NFV nodes on which no SFCs traverse other overloadedNFV nodes, and UO = {i∈ U |

∑r∈RX

(r)i (k)f

(r)1 > 0} is a

set of underloaded NFV nodes on which at least one SFCtraverses other overloaded NFV nodes. Let f (r)

2 be a binaryflag indicating whether SFC r traverses any NFV nodes in UO,

SFC category I

SFC category II

SFC category III

SFC category IV

Fig. 4: Four SFC categories based on NFV node loading factors.

with f (r)2 = 1 if

∑i∈UO X

(r)i (k) > 0, and f (r)

2 =0 otherwise.Accordingly, SFCs are classified into four categories: SFCcategory I with f (r)

1 = 1 and f (r)2 = 0, SFC category II with

f(r)1 = f

(r)2 = 1, SFC category III with f (r)

1 =0 and f (r)2 =1,

and SFC category IV with f (r)1 =f

(r)2 =0, as shown in Fig. 4.

Update and iteration. Define two sets of delay scalingfactors, with initial values of 1, vertical delay scaling factors{δi(k)} and horizontal delay scaling factors {β(r)(k)}. A two-step delay scaling strategy is proposed as follows.

1) Step I - Delay scaling for SFC category III: Hop delaybounds for SFC category III are relaxed on NFV nodes in UO,to release resources for SFC category II, by making hop delaybounds more stringent on NFV nodes in UU .

First, the loading factor of NFV node i ∈ UU is increasedto ηth, by shrinking hop delay bounds for SFC category III onNFV node i by a positive factor, δi(k), less than 1, as derivedin Appendix and given by

δi(k) =

∑(r,h)∈V

P rhi σ(r)

Drh(k)xrhi (k)f

(r)2

[ηth−ηi(k)]Ci+∑

(r,h)∈VP rh

i σ(r)

Drh(k)xrhi (k)f

(r)2

. (25)

The preceding delay scaling, called vertical delay scaling, isapplied to multiple SFCs belonging to category III on NFVnode i ∈ UU . Then, hop delay bounds for SFC r in categoryIII on NFV nodes in UO are relaxed by a factor, β(r)(k), largerthan 1, given by

β(r)(k) =D(r) −

∑h∈Hr

(Drh(k)

∑i∈UU∪Q x

rhi (k)

)D(r) −

∑h∈Hr

(Drhp (k)

∑i∈UU∪Q x

rhi (k)

) (26)

where Drhp (k) is the old value of Drh(k) before vertical delay

scaling. The preceding delay scaling, called horizontal delayscaling, is applied to multiple hops in an SFC affected byvertical delay scaling. Based on (23), {ηi(k)} is updated.

2) Step II - Delay scaling for SFC category II: Moreresources available at NFV nodes in UO from Step I areallocated to SFC category II. First, vertical delay scaling isapplied to NFV nodes in UO to increase their loading factorsto ηth, through scaling hop delay bounds for SFC category IIon it by a vertical scaling factor. Then, horizontal delay scalingis applied to SFC category II, by relaxing hop delay boundsfor SFC r in category II on NFV nodes in O by a horizontalscaling factor. The scaling factors are similar to that in StepI, and details are omitted.

After the delay scaling procedures, NFV node loading fac-tors, NFV node classification and SFC categories are updated.If UO 6= ∅ and

∑r∈R f

(r)1 > 0, i.e., there is at least

one SFC in category II, it is possible to further reduce thenumber of overloaded NFV nodes through Step II, so Step

9

Algorithm 2: Redistribution of hop delay bounds

1 Input: ηth, {xrhi (k)}, {Drh(k)}2 Calculate {ηi(k)}, O, UU , UO, Q, {f (r)

1 }, {f(r)2 }.

3 Loop index n = 0.4 while UO 6= ∅ and

∑r∈R f

(r)1 > 0 do

5 {δi(k)} = 1; {β(r)(k)} = 1.6 if n == 0, then7 Vertical delay scaling at NFV nodes in UU .8 Horizontal delay scaling for SFC category III

at NFV nodes in UO.9 Update {ηi(k)}.

10 Vertical delay scaling at NFV nodes in UO.11 Horizontal delay scaling for SFC category II at

NFV nodes in O.12 Update {ηi(k)}, O, UU , UO, Q, {f (r)

1 }, {f(r)2 }.

13 n = n+ 1

14 Output: {Drh(k)}, {ηi(k)}, O, O1, {f (r)1 }.

II is performed iteratively until the condition is violated.The outputs of Algorithm 2 are shown in line 14, whereO1 = {i ∈ O|

∑r∈RX

(r)i (k) = 1} denotes a set of

overloaded NFV nodes traversed by a single SFC.

C. Migration Decision

When one migration is required, a migration decision pro-cedure is to select one bottleneck NFV node, one SFC tomigrate, and one target NFV node. Migration decisions aremade greedily to reduce the reconfiguration overhead. First,a candidate bottleneck NFV node set, B, with |B| = |R|, isdetermined. For an SFC, the traversed NFV node with largesthop delay bound is selected as a candidate bottleneck. Then,bottleneck NFV node, ib, is determined in three cases. In thefirst case with B ∩ O 6= ∅, an NFV node in B ∩ O with thelargest number of SFCs is selected, given by

ib = argmaxi∈B∩O

∑r∈R

X(r)i (k). (27)

In the second case with B ∩ O =∅ and O\O1 6=∅, ib is anNFV node in O\O1 with the largest loading factor,

ib = argmaxi∈O\O1

ηi(k). (28)

In the third case with B ∩ O = O\O1 = ∅, an NFV nodein B whose SFCs traverse the largest number of overloadedNFV nodes is selected, given by

ib = argmaxi∈B

∑r∈R

X(r)i (k)

∑j∈O

X(r)j (k)

. (29)

Next, an SFC to migrate from ib and a target NFV nodeto accommodate the migrated SFC are jointly selected tominimize the reconfiguration overhead, i.e., the weighted sumof normalized transmission resource overhead for state transferand number of extra virtual links for flow rerouting. In thisway, α2 and α3 are considered in the heuristic algorithm. Ifthere are multiple choices, an SFC with the largest resourcedemand is migrated to the closest target NFV node.

Algorithm 3: Heuristic algorithm for problem (17)

1 Input: Step size ζ0, precision ε2 Initialize: {xrhi (k)}, {Drh(k)}3 Let: κ = 0, ηth = ηU , ζ = ζ0

4 while κ == 1 or ζ > ε do5 if κ == 1, then6 Update {xrhi (k)} according to the migration

decision making procedure.

7 Update {Drh(k)}, {ηi(k)}, O, O1, {f (r)1 }

according to Algorithm 2.8 if

∑r∈R f

(r)1 == 0, then

9 if ζ 6= ζ0, then ζ = ζ/210 ηth = ηth − ζ, κ = 0

11 else if ηth == ηU , then κ = 112 else ζ = ζ/2, ηth = ηth + ζ, κ = 0

13 Output: Sub-optimal solution to problem (17).

D. Coordination with Threshold Update

Let binary variable, κ, indicate whether migration is re-quired to overcome resource overloading. It is set as 0 initiallyand updated iteratively. Let ζ be a step size to update ηth,with initial value ζ0 and updated before each ηth update. Aprecision, ε, for ζ is set as a stop condition.

After initialization in Algorithm 3 (lines 2-3), a redistri-bution of hop delay bounds is performed to check whethermigration is required. Based on outputs of Algorithm 2, κ,ηth and ζ are updated in three cases, as shown in lines 8-12.

1) Update for κ and ηth: In the first case, there are noremaining overloaded NFV nodes, i.e.,

∑r∈R f

(r)1 = 0. Then

κ = 0, and ηth is reduced by a step. In the other two cases,there are still overloaded NFV nodes but UO = ∅, meaningthat delay scaling is not sufficient to deal with resourceoverloading on NFV nodes, but either at least one migrationor increasing ηth by a step is required, depending on theηth value. In the second case with ηth = ηU , at least onemigration is needed, i.e., κ = 1, and ηth should remain ηU tocheck whether more migrations are required after a migrationdecision is made. In the third case with ηth < ηU , no moremigrations are required since ηth has been reduced by at leastone step in previous updates, thus κ = 0 and ηth is increasedby a step. With the updates for κ and ηth, redistribution of hopdelay bounds is performed iteratively until no more migrationsare required and the precision of ζ reaches ε.

2) Update for ζ: Step size ζ plays a key role in guar-anteeing algorithm convergence. For example, a constant ζequal to ε guarantees precision but makes the algorithm slowto converge due to a potential oscillation of ηth around itsoptimal value. Therefore, we employ the following strategy toupdate ζ. If the outputs of Algorithm 2 fall into the second casewhere a migration is required, ζ remains a constant equal to ζ0.After all migrations are performed, the outputs of Algorithm 2correspond to the first case, and ηth should be reduced by aconstant step size ζ equal to ζ0. Until the first time that theoutputs of Algorithm 2 fall into the third case, ζ starts to bereduced by half before each ηth update.

10

E. Complexity Analysis

We first analyze the time complexity of Algorithm 2. Delayscaling for SFC category III is performed once, using atmost O(

∑i∈I |V|) time. Delay scaling for SFC category II

is performed iteratively until there are no new NFV nodesin UO or there are no overloaded SFCs. The worst casehappens when each round of delay scaling for SFC category IItransforms a single NFV node in O to a new NFV node in UO,consuming O(

∑i∈O

∑i∈I |V|) time. Thus, the complexity

of Algorithm 2 is O(∑i∈O

∑i∈I |V|), upper bounded by

O(|I|2|V|). The complexity of the migration decision proce-dure is dominated by the selection of SFC to migrate in thethird case, which requires a running time of O(|I||R|2). InAlgorithm 3, at most |V| sequential migration decisions areperformed followed by [ 1

ζ0 + log2( ζ0

ε )] iterations of thresholdupdating. In each iteration, hop delay bounds are readjusted.Therefore, the worst case running time of Algorithm 3 is|V|[O(|I||R|2) + O(|I|2|V|)] + [ 1

ζ0 + log2( ζ0

ε )]O(|I|2|V|),which is simplified to O(|I|2|V|2) when |R|2 < |I||V|.

VII. PERFORMANCE EVALUATION

In this section, simulation results are presented to evalu-ate the MIQCP and heuristic solutions for the delay-awareflow migration problem. Two time intervals are considered:(k−1) and k, representing the current and next time intervalsrespectively. We use two mesh networks with 64 NFV nodesand 256 NFV nodes to represent the virtual resource pool.Virtual links exist only between neighboring NFV nodes. Inthe 64-node network, we consider fixed SFC mapping fortime interval (k − 1), with three SFCs initially mapped tothe virtual resource pool. Specifically, SFC 3 shares two NFVnodes with SFC 1 and one of them also with SFC 2. In the256-node network, we consider different numbers of SFCs,with [3, 5] VNFs in each one, randomly distributed in thenetwork during time interval (k − 1). In the CPU pollingscheme, we set a ratio of 0.01 between the context switchingtime and the polling period. The upper bound, ηU , for η(k),is 0.95. The average E2E delay requirement for each SFCis 0.02 s, and the maximal tolerable service downtime is0.005 s. For VNF states, the size is a constant, equal to 10bytes, thus requiring at least a transmission rate Bmin of 16kbit/s for a state transfer. Under the simulation setup, the totalnormalized transmission resource overhead for state transfer isequal to the total number of migrations. For the weights, weset α2 = 2α3. Then, α1 <

23ηU+2 = 0.4123 should be satisfied

to penalize migration more than imbalanced loading. In thiscase, 0.4123 is the worst-case boundary for α1, to guaranteethe penalization preference if α1 is less than the boundary.We implement both the MIQCP and heuristic solutions inpython. We use NetworkX to simulate the network scenario,and Gurobi python interface to solve the MIQCP problem.

A. Load Balancing and Reconfiguration Overhead Trade-off

We use the 64-node network with three initially mappedSFCs to evaluate the performance of the MIQCP solutionwith varying traffic load under three sets of weights in (1),and investigate the trade-off between load balancing and

reconfiguration overhead. For traffic load during interval k,we have λ(1)(k) = 600 packet/s, λ(2)(k) = 200 packet/s,and vary λ(3)(k) from 200 packet/s to 740 packet/s. Beyond740 packet/s, the problem becomes infeasible due to process-ing resource constraints and average E2E delay constraints.Performance metrics are the maximum NFV node loadingfactor, η(k), the number of migrations, N(k), and the numberof extra virtual links, S(k), for flow rerouting. We explorethree sets of weights. For {α1, α2, α3} = {1, 0, 0}, thereconfiguration overhead is not optimized but load balancingis the focus, corresponding to a load balancing flow migration(LBFM) strategy. For {α1, α2, α3} = {0, 2

3 , 13}, η(k) is not

optimized but reconfiguration overhead reduction is empha-sized, corresponding to a minimum overhead flow migration(MOFM) strategy. For {α1, α2, α3} = {0.4, 0.4, 0.2}, bothload balancing and reconfiguration overhead reduction areimportant, corresponding to a hybrid flow migration (HFM)strategy. Fig. 5 shows performance of three strategies with theincrease of λ(3)(k).

LBFM strategy: It is observed that η(k) is dominated bySFC 1 when λ(3)(k) is relatively small, showing a flat trendfirst, but turns to be dominated by SFC 3 with the increaseof λ(3)(k). Both N(k) and S(k) are high and vary with thetraffic load, since they are not optimized. SFCs separate fromeach other even at a relatively low traffic load to balance trafficloads in the virtual resource pool.

MOFM strategy: Both N(k) and S(k) show a step-wiseincreasing trend with the increase of λ(3)(k). However, η(k)is fixed at ηU , since it is not optimized.

HFM strategy: A trade-off among performance metrics isobserved. With the increase of λ(3)(k), η(k) drops sharplywhen N(k) or S(k) is increased by 1. When N(k) and S(k)stay stable, η(k) shows either a linear increasing or a flat trend.Compared with LBFM and MOFM strategies, HFM strategyapproaches to the lower bounds of N(k) and S(k) determinedby the MOFM strategy, while keeping η(k) at a medium level.

B. Average End-to-End Delay Performance

We carry out packet-level simulations using network simu-lator OMNeT++ to evaluate average E2E delay of SFC 3 withand without flow migration, under the same network and SFCsettings as in Subsection VII-A. For average traffic rates, weset λ(1)(k) = λ(2)(k) = 200 packet/s and increase λ(3)(k)from 200 packet/s. To verify the effectiveness and accuracy ofour flow migration model in the presence of traffic burstiness,not only Poisson but also MMPP packet arrivals are simulated.For each traffic arrival pattern, we collect sufficient packetdelay information to estimate the average E2E delay. We usea two-state MMPP model with same transition rate betweenstates and an average rate of λ(3)(k). We use “MMPP-q1-q2” to represent the MMPP traffic model, where q1 and q2

are ratios between state-dependent rates and λ(3)(k), withq1 + q2 = 2. For example, for “MMPP-1.6-0.4” traffic modelwith λ(3)(k) = 500 packet/s, the state-dependent rates are 800packet/s and 200 packet/s respectively.

Fig. 6 shows the average E2E delay without flow migration,in which a flat trend is observed, followed by an exponentialincreasing trend, for Poisson traffic arrival with increasing rate

11

200 300 400 500 600 700

Packet Arrival Rate of SFC-3 (packet/s)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1(k

)

0

1

2

3

4

5

6

7

8

9

10

11

N(k

) or

S(k

)

(k)

N(k)

S(k)

(a) LBFM strategy

200 300 400 500 600 700

Packet Arrival Rate of SFC-3 (packet/s)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(k)

0

1

2

3

4

5

6

7

8

9

10

11

N(k

) or

S(k

)

(k)

N(k)

S(k)

(b) MOFM strategy

200 300 400 500 600 700

Packet Arrival Rate of SFC-3 (packet/s)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(k)

0

1

2

3

4

5

6

7

8

9

10

11

N(k

) or

S(k

)

(k)

N(k)

S(k)

(c) HFM strategy

Fig. 5: Performance of three flow migration strategies with the increase of traffic load λ(3)(k), for ηU = 0.95.

200 225 250 275 300 325 350 375 400 425

Average Packet Arrival Rate (packet/s)

0

0.02

0.04

0.06

0.08

0.1

E2E

dela

y (

s)

Poisson

Fig. 6: Average E2E delay without flow migration.

200 300 400 500 600 700Average Packet Arrival Rate (packet/s)

0.00

0.01

0.02

0.03

0.04

E2E de

lay (s)

PoissonMMPP-1.2-0.8

MMPP-1.4-0.6MMPP-1.6-0.4

Fig. 7: Average E2E delay comparison with flow migration.

from 200 packet/s to 425 packet/s. The flat trend correspondsto feasible traffic rates for E2E delay guarantee with localprocessing resource scaling. Beyond 360 packet/s, local re-sources are not sufficient, resulting in an exponential increaseof E2E delay. Fig. 7 shows the average E2E delay with flowmigration. We observe that the E2E delay requirement issatisfied for Poisson traffic with rate in [200, 700] packet/s,inferring that more traffic can be accommodated from serviceswhich originally share some NFV nodes on their E2E paths,with joint flow migration and processing resource scaling. Ata certain average rate, the E2E delay performance degradeswith more traffic burstiness. However, even “MMPP-1.6-0.4”for average rate in [400, 700] packet/s with flow migrationperforms much better than Poisson traffic arrival for averagerate larger than 360 packet/s without flow migration, indicatingthat our flow migration model can accommodate some trafficburstiness without a significant degradation on E2E delay.

C. Comparison between MIQCP and heuristic solutions

0.01 0.11 0.21 0.31 0.41 0.51 0.61 0.71 0.81 0.91

Weight 1

0.7

0.9

1.1

1.3

1.5

1.7

1.9

2.1

2.3

Va

lue

0.4123

Worst-case boundary for 1

MIQCP: (k)

Heuristic: (k)

MIQCP: Reconfiguration overhead

Heuristic: Reconfiguration overhead

MIQCP: Total cost

Heuristic: Total cost

Fig. 8: Costs with respect to weight α1 in objective function, for α2 = 2α3.

1) Cost sensitivity to different weights: Under the 64-nodenetwork setup with three SFCs, we compare the MIQCP andheuristic solutions in terms of their cost sensitivity to differentweights in (1). With α2 = 2α3 and α1 + α2 + α3 = 1, threecost metrics including the maximum NFV node loading factor,η(k), the reconfiguration overhead, 2N(k)+S(k), and the totalcost, α1η(k)+(1−α1)(2N(k)+S(k)), are evaluated. The firsttwo costs are partial costs. Although the heuristic solution is inprinciple insensitive to α1, we use the same definition of totalcost for a fair comparison. The three cost metrics with respectto α1 for both the MIQCP and heuristic solutions are givenin Fig. 8. In both solutions, the total cost approaches to thereconfiguration overhead, for α1 close to 0, and approaches tothe maximum NFV node loading factor, for α1 close to 1. Forthe heuristic solution, we observe constant partial costs withrespect to α1, which is consistent with the design principle.For the MIQCP solution, both partial costs show a stable trendfor small and medium values of α1 in a range larger thanthe theoretical worst-case range (0, 0.4123). For large valuesof α1, the reconfiguration overhead of the MIQCP solutionincreases with α1, while the maximum NFV node loadingfactor decreases with α1, since much more penalization isplaced on imbalanced loading than migrations.

2) Cost efficiency and time efficiency: We use a 256-node mesh network to compare the cost and time efficiencybetween the MIQCP and heuristic solutions. The comparisonis performed with fixed weights in (1), under the conditionof α1ηU < α2. Specifically, we have {α1, α2, α3} = {0.4,0.4, 0.2}. Eight groups of experiments are implemented with

12

0

5

10

15

20

25

30

10 15 20 25 30 35 40 45

To

tal

co

st

Number of SFCs

MIQCP Heuristic

Fig. 9: Total cost with respect to the number of SFCs.

Fig. 10: Running time with respect to the number of SFCs.

10, 15, 20, 25, 30, 35, 40 and 45 SFCs respectively. The totalcost and running time for each group are evaluated at differenttraffic rates from 200 to 740 packet/s. In each experiment,all SFCs have the same traffic rate, denoted by λ(k). Theinitial step size, ζ0, and the precision, ε, are set to 0.1 and0.0001 respectively. Fig. 9 shows the average total cost withrespect to the number of SFCs (|R|). It shows that the total costobtained from both solutions increases with |R|. Adding moreSFCs tends to increase the number of overloaded NFV nodes,especially when the traffic rate is high and the added SFCsshare some NFV nodes with others. Hence, more migrationstend to be triggered with more SFCs added, incurring morecost. Fig. 10 shows the average running time with respect to|R|. We can see an almost exponential increasing trend forthe running time of the MIQCP solution. In comparison, thetime complexity of the heuristic solution is much less, and theincreasing trend is much less significant.

D. Convergence of heuristic algorithm

To evaluate convergence of the proposed heuristic algo-rithm, we plot the updating process of the threshold ηth, in the45-SFC experiments with different traffic rates λ(k), as shownin Fig. 11, in which η(k) is the maximum NFV node loadingfactor after convergence. On each threshold updating curvecorresponding to a specific traffic rate, we see that ηth firstremains ηU due to several sequential migration decisions at thebeginning and then drops with the initial step size of 0.1 untila turning point at the lower bound. After the turning point, thestep size is reduced by half with each iteration until it is belowthe required precision 0.0001. With the increase of traffic rateto 500 packet/s, more migrations happen to gradually decouple

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4

Running Time (s)

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

th

(k) = 200, N(k) = 4, S(k) = 4, (k) = 0.93417

(k) = 300, N(k) = 11, S(k) = 5, (k) = 0.92604

(k) = 400, N(k) = 34, S(k) = 25, (k) = 0.93861

(k) = 500, N(k) = 52, S(k) = 37, (k) = 0.70000

(k) = 600, N(k) = 52, S(k) = 37, (k) = 0.80000

(k) = 700, N(k) = 52, S(k) = 37, (k) = 0.90000

Fig. 11: Threshold update in the heuristic algorithm, for ηU = 0.95.

the SFCs from each other, and more extra virtual links areobserved. When the traffic rate grows larger than 500 packet/s,all SFCs are completely decoupled, with no resource sharingon NFV nodes, thus N(k) and S(k) are stabilized but η(k)increases. When the traffic rate is smaller than 500 packet/s,η(k) is close to ηU , while less migrations and extra virtuallinks are observed, showing a trade-off between load balancingand reconfiguration overhead.

VIII. CONCLUSION

In this paper, we study a delay-aware flow migrationproblem for embedded SFCs sharing a common physicalnetwork in SDN/NFV-enabled 5G communication systems. Amixed integer optimization problem is formulated based onan abstraction of virtual resource pool, addressing the trade-off between load balancing and reconfiguration overhead. Theproblem is non-convex and difficult to solve using optimizationsolvers. Hence, we reformulate a tractable MIQCP problembased on which the optimum of the original problem canbe obtained. Numerical results show that the proposed modelaccommodates more traffic from services, in comparison withan SFC configuration without flow migrations. Moreover, aflow migration strategy with similar priority in load balancingand migration reduction achieves medium load balancing, ascompared with flow migration strategies with a priority oneither goal. Nevertheless, it achieves approximately as goodperformance in terms of the reconfiguration overhead as a flowmigration strategy which aims at migration reduction. Thisresult indicates the benefit of joint consideration of the twogoals. A performance comparison between the MIQCP andheuristic solutions demonstrates the effectiveness and time effi-ciency of the heuristic solution. Under the assumption that flowmigrations do not take place frequently and a time interval issufficiently large, traffic arrival of SFCs is modeled as Poissonwith different rates across time intervals. Correspondingly, theM/M/1 queueing model is employed in both the MIQCP andheuristic solutions, for performance modeling and predictionwith predicted traffic variations. In reality, traffic can be highlydynamic especially in 5G new use cases. Motivated by thelimitation of Poisson traffic model, we are working on anextension of this work for adaptive flow migration, usingpromising model-free techniques such as machine learning.

13

APPENDIX: DERIVATION OF δi(k)

Let ηrhi (k) denote a ratio between resources occupied byVNF V

(r)h and resource capacity of NFV node i, given by

ηrhi (k) =P rhi σ(r)

Ci

(λ(r)(k) +

1

Drh(k)

)xrhi (k). (A1)

VNF set V is divided into two subsets, i.e., V = V1 ∪ V2,where V1 = {(r, h) ∈ V |xrhi (k) = f

(r)2 = 1} is a set of

VNFs belonging to SFC category III on NFV node i, and V2

is a set of all other VNFs. Vertical delay scaling is appliedto only VNFs in V1. Before delay scaling, resource usage atNFV node i is composed of three parts, given by

ηi(k)=∑

(r,h)∈V1∪V2

ηrhi (k) +∑

(r,h)∈V1∪V2

wi(k)xrhi (k)W

Ti. (A2)

The ratio of resources occupied by VNFs in V1 before verticaldelay scaling is given by∑(r,h)∈V1

ηrhi (k)=∑

(r,h)∈V1

P rhi σ(r)

Ci

(λ(r)(k) +

1

Drh(k)

). (A3)

For a vertical delay scaling by a positive coefficient δi(k) toincrease loading factor of NFV node i from ηi(k) to ηth, wehave the following relationship among parameters, given by

ηth −∑

(r,h)∈V2

ηrhi (k)−∑

(r,h)∈V1∪V2

wi(k)xrhi (k)W

Ti

=∑

(r,h)∈V1

P rhi σ(r)

Ci

(λ(r)(k) +

1

δi(k)Drh(k)

).

(A4)

Subtracting (A3) from (A4) and arranging items, we obtain

δi(k) =

∑(r,h)∈V1

P rhi σ(r)

Drh(k)

[ηth−ηi(k)]Ci+∑

(r,h)∈V1P rh

i σ(r)

Drh(k)

(A5)

which is equivalent to (25).

REFERENCES

[1] K. Qu, W. Zhuang, Q. Ye, X. Shen, X. Li, and J. Rao, “Delay-awareflow migration for embedded services in 5G core networks,” in Proc.IEEE ICC, May 2019, pp. 1–6.

[2] I. Vision, “Framework and overall objectives of the future developmentof IMT for 2020 and beyond,” International Telecommunication Union(ITU), Document, Radiocommunication Study Groups, 2015.

[3] J. Ordonez-Lucena, P. Ameigeiras, D. Lopez, J. J. Ramos-Munoz,J. Lorca, and J. Folgueira, “Network slicing for 5G with SDN/NFV:Concepts, architectures, and challenges,” IEEE Commun. Mag., vol. 55,no. 5, pp. 80–87, May 2017.

[4] X. Li, P. Djukic, and H. Zhang, “Zoning for hierarchical networkoptimization in software defined networks,” in Proc. IEEE NOMS, May2014, pp. 1–8.

[5] N. Zhang, S. Zhang, P. Yang, O. Alhussein, W. Zhuang, and X. Shen,“Software defined space-air-ground integrated vehicular networks: Chal-lenges and solutions,” IEEE Commun. Mag., vol. 55, no. 7, pp. 101–109,July 2017.

[6] J. G. Herrera and J. F. Botero, “Resource allocation in NFV: Acomprehensive survey,” IEEE Trans. Netw. Service Manag., vol. 13,no. 3, pp. 518–532, Sept. 2016.

[7] V.-G. Nguyen, A. Brunstrom, K.-J. Grinnemo, and J. Taheri,“SDN/NFV-based mobile packet core network architectures: A survey,”IEEE Commun. Surveys Tuts., vol. 19, no. 3, pp. 1567–1602, thirdquarter 2017.

[8] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson,J. Rexford, S. Shenker, and J. Turner, “Openflow: Enabling innovation incampus networks,” ACM SIGCOMM Computer Communication Review,vol. 38, no. 2, pp. 69–74, Apr. 2008.

[9] O. Alhussein, P. T. Do, J. Li, Q. Ye, W. Shi, W. Zhuang, X. Shen, X. Li,and J. Rao, “Joint VNF placement and multicast traffic routing in 5Gcore networks,” in Proc. IEEE GLOBECOM, Dec. 2018, pp. 1–6.

[10] X. Chen, W. Ni, I. B. Collings, X. Wang, and S. Xu, “Automated functionplacement and online optimization of network functions virtualization,”IEEE Trans. Commun., vol. 67, no. 2, pp. 1225–1237, Feb. 2019.

[11] Z. Xu, W. Liang, M. Huang, M. Jia, S. Guo, and A. Galis, “EfficientNFV-enabled multicasting in SDNs,” IEEE Trans. Commun., vol. 67,no. 3, pp. 2052–2070, Mar. 2019.

[12] Q. Ye, W. Zhuang, X. Li, and J. Rao, “End-to-end delay modeling forembedded VNF chains in 5G core networks,” IEEE Internet of ThingsJ., vol. 6, no. 1, pp. 692–704, Feb. 2019.

[13] Q. Ye, J. Li, K. Qu, W. Zhuang, X. Shen, and X. Li, “End-to-end qualityof service in 5G networks – Examining the effectiveness of a networkslicing framework,” IEEE Veh. Technol. Mag., vol. 13, no. 2, pp. 65–74,June 2018.

[14] “Gurobi Optimizer Reference Manual,” http://www.gurobi.com, 2018,[Online; accessed 15-January-2020].

[15] B. Fortz and M. Thorup, “Internet traffic engineering by optimizingOSPF weights,” in Proc. IEEE INFOCOM, June 2000, pp. 519–528.

[16] J. Rexford, “Route optimization in IP networks,” Handbook of Opti-mization in Telecommunications, pp. 679–700, 2006.

[17] H. Tang, D. Zhou, and D. Chen, “Dynamic network function instancescaling based on traffic forecasting and VNF placement in operator datacenters,” IEEE Trans. Parallel Distrib. Syst., vol. 30, no. 3, pp. 530–543,Mar. 2019.

[18] A. Gember-Jacobson, R. Viswanathan, C. Prakash, R. Grandl, J. Khalid,S. Das, and A. Akella, “OpenNF: Enabling innovation in networkfunction control,” in Proc. ACM SIGCOMM, Aug. 2014, pp. 163–174.

[19] M. Peuster and H. Karl, “E-state: Distributed state management in elasticnetwork function deployments,” in Proc. NetSoft, June 2016, pp. 6–10.

[20] L. Nobach, I. Rimac, V. Hilt, and D. Hausheer, “Statelet-based efficientand seamless NFV state transfer,” IEEE Trans. Netw. Service Manag.,vol. 14, no. 4, pp. 964–977, Dec. 2017.

[21] B. Zhang, P. Zhang, Y. Zhao, Y. Wang, X. Luo, and Y. Jin, “Co-scaler:Cooperative scaling of software-defined NFV service function chain,” inProc. IEEE Conf. Network Function Virtualization and Software DefinedNetworks, 2016, pp. 33–38.

[22] M. Ghaznavi, A. Khan, N. Shahriar, K. Alsubhi, R. Ahmed, andR. Boutaba, “Elastic virtual network function placement,” in Proc. IEEECloudNet, Oct. 2015, pp. 255–260.

[23] J. Liu, W. Lu, F. Zhou, P. Lu, and Z. Zhu, “On dynamic service functionchain deployment and readjustment,” IEEE Trans. Netw. Service Manag.,vol. 14, no. 3, pp. 543–553, Sept. 2017.

[24] W. Rankothge, F. Le, A. Russo, and J. Lobo, “Optimizing resourceallocation for virtualized network functions in a cloud center usinggenetic algorithms,” IEEE Trans. Netw. Service Manag., vol. 14, no. 2,pp. 343–356, June 2017.

[25] V. Eramo, E. Miucci, M. Ammar, and F. G. Lavacca, “An approachfor service function chain routing and virtual function network instancemigration in network function virtualization architectures,” IEEE/ACMTrans. Netw., vol. 25, no. 4, pp. 2008–2025, Aug. 2017.

[26] L. Guo, J. Pang, and A. Walid, “Dynamic service function chaining inSDN-enabled networks with middleboxes,” in Proc. IEEE ICNP, Nov.2016, pp. 1–10.

[27] J. Xia, D. Pang, Z. Cai, M. Xu, and G. Hu, “Reasonably migrating virtualmachine in NFV-featured networks,” in Proc. IEEE Conf. Computer andInformation Technology, Dec. 2016, pp. 361–366.

[28] F. Zhang, G. Liu, X. Fu, and R. Yahyapour, “A survey on virtual machinemigration: Challenges, techniques, and open issues,” IEEE Commun.Surveys Tuts., vol. 20, no. 2, pp. 1206–1243, second quarter 2018.

[29] M. Shin, S. Chong, and I. Rhee, “Dual-resource TCP/AQM forprocessing-constrained networks,” IEEE/ACM Trans. Netw., vol. 16,no. 2, pp. 435–449, Apr. 2008.

[30] L. Rizzo, M. Carbone, and G. Catalli, “Transparent acceleration ofsoftware packet forwarding using Netmap,” in Proc. IEEE INFOCOM,Mar. 2012, pp. 2471–2479.

[31] S. Garzarella, G. Lettieri, and L. Rizzo, “Virtual device passthroughfor high speed VM networking,” in Proc. 11th ACM/IEEE Symp.Architectures for Networking and Communications Systems, May 2015,pp. 99–110.

14

[32] P. Emmerich, D. Raumer, S. Gallenmuller, F. Wohlfart, and G. Carle,“Throughput and latency of virtual switching with Open vSwitch: Aquantitative analysis,” Journal of Network and Systems Management,vol. 26, no. 2, pp. 314–338, Apr. 2018.

[33] D. Bhamare, M. Samaka, A. Erbad, R. Jain, L. Gupta, and H. A. Chan,“Optimal virtual network function placement in multi-cloud servicefunction chaining architecture,” Computer Communications, vol. 102,pp. 1–16, Apr. 2017.

[34] F. Ben Jemaa, G. Pujolle, and M. Pariente, “Analytical models forQoS-driven VNF placement and provisioning in wireless carrier cloud,”in Proc. 19th ACM International Conf. on Modeling, Analysis andSimulation of Wireless and Mobile Systems, Nov. 2016, pp. 148–155.

[35] Z. Luo, C. Wu, Z. Li, and W. Zhou, “Scaling geo-distributed networkfunction chains: A prediction and learning framework,” IEEE J. Select.Areas Commun., vol. 37, no. 8, pp. 1838–1850, Aug. 2019.

[36] M. R. Garey and D. S. Johnson, Computers and Intractability: A Guideto the Theory of NP-Completeness. New York: wh freeman, 1978.

Kaige Qu (S’19) received her B.S. degree in com-munication engineering from Shandong University,Jinan, China, in 2013. She received her M.S. degreesin integrated circuits engineering and electrical en-gineering from Tsinghua University, Beijing, China,and KU Leuven, Leuven, Belgium, respectively, in2016. She is currently pursuing her Ph.D. degreewith the Department of Electrical and ComputerEngineering, University of Waterloo, Waterloo, ON,Canada. Her research interests include resource al-location in SDN/NFV-enabled networks, 5G and

beyond, and machine learning for future networking.

Weihua Zhuang (M’93−SM’01−F’08) has beenwith the Department of Electrical and ComputerEngineering, University of Waterloo, Canada, since1993, where she is a Professor and a Tier ICanada Research Chair in wireless communicationnetworks. She was a recipient of the 2017 TechnicalRecognition Award from the IEEE CommunicationsSociety Ad Hoc and Sensor Networks TechnicalCommittee, and several best paper awards fromIEEE conferences. She was the Technical ProgramChair/Co-Chair of the IEEE VTC Fall 2016 and Fall

2017, the Editor-in-Chief of IEEE Transactions on Vehicular Technology(2007−2013), and an IEEE Communications Society Distinguished Lecturer(2008−2011). Dr. Zhuang is a Fellow of the Royal Society of Canada, theCanadian Academy of Engineering, and the Engineering Institute of Canada.She is an Elected Member of the Board of Governors and VP Publicationsof the IEEE Vehicular Technology Society.

Qiang Ye (S’16−M’17) received his Ph.D. degreein electrical and computer engineering from the Uni-versity of Waterloo, Waterloo, ON, Canada, in 2016.He had been with the Department of Electrical andComputer Engineering, University of Waterloo, asa post-doctoral fellow and then a research associatefrom December 2016 to September 2019. He hasbeen an Assistant Professor with the Departmentof Electrical and Computer Engineering and Tech-nology, Minnesota State University, Mankato, MN,USA, since September 2019. His current research

interests include 5G networks, software-defined networking and networkfunction virtualization, network slicing, artificial intelligence and machinelearning for future networking, protocol design, and end-to-end performanceanalysis for the Internet of Things.

Xuemin (Sherman) Shen (M’97−SM’02−F’09)received his Ph.D. degree in electrical engineeringfrom Rutgers University, New Brunswick, NJ, USA,in 1990. He is currently a University Professorwith the Department of Electrical and ComputerEngineering, University of Waterloo, Canada. Hisresearch focuses on network resource management,wireless network security, social networks, 5G andbeyond, and vehicular ad hoc and sensor networks.He is a registered Professional Engineer of Ontario,Canada, an Engineering Institute of Canada Fellow, a

Canadian Academy of Engineering Fellow, a Royal Society of Canada Fellow,a Chinese Academy of Engineering Foreign Fellow, and a DistinguishedLecturer of the IEEE Vehicular Technology Society and CommunicationsSociety. Dr. Shen received the R.A. Fessenden Award in 2019 from IEEE,Canada, James Evans Avant Garde Award in 2018 from the IEEE VehicularTechnology Society, Joseph LoCicero Award in 2015 and Education Awardin 2017 from the IEEE Communications Society. He has also received theExcellent Graduate Supervision Award in 2006 and Outstanding PerformanceAward 5 times from the University of Waterloo and the Premiers ResearchExcellence Award (PREA) in 2003 from the Province of Ontario, Canada.He served as the Technical Program Committee Chair/Co-Chair for theIEEE Globecom’16, the IEEE Infocom’14, the IEEE VTC’10 Fall, the IEEEGlobecom’07, the Symposia Chair for the IEEE ICC’10, the Tutorial Chair forthe IEEE VTC’11 Spring, and the Chair for the IEEE Communications SocietyTechnical Committee on Wireless Communications. He was the Editor-in-Chief of the IEEE INTERNET OF THINGS JOURNAL from 2017 to 2019.He is the Vice President on Publications of the IEEE Communications Society.

Xu Li is a senior principal researcher at HuaweiTechnologies Canada. He received a PhD (2008)degree from Carleton University, an MSc (2005)degree from the University of Ottawa, and a BSc(1998) degree from Jilin University, China, allin computer science. Prior to joining Huawei, heworked as a research scientist (with tenure) at Inria,France. His current research interests are focusedin 5G and beyond. He contributed extensively tothe development of 3GPP 5G standards through 90+standard proposals. He has published 100+ refereed

scientific papers and is holding 40+ issued US patents.

Jaya Rao received his B.S. and M.S. degrees inElectrical Engineering from the University of Buf-falo, New York, in 2001 and 2004, respectively, andhis Ph.D. degree from the University of Calgary,Canada, in 2014. He is currently a Senior ResearchEngineer at Huawei Technologies Canada, Ottawa.Since joining Huawei in 2014, he has worked on re-search and design of CIoT, URLLC and V2X basedsolutions in 5G New Radio. He has contributedfor Huawei at 3GPP RAN WG2, RAN WG3, andSA2 meetings on topics related to URLLC, network

slicing, mobility management, and session management. From 2004 to 2010,he was a Research Engineer at Motorola Inc. He was a recipient of the BestPaper Award at IEEE WCNC 2014.