IEEE P

roof

1 Queueing Theoretic Approach for2 Performance-Aware Modeling of Sustainable3 SDN Control Planes4 Xinli Huang ,Member, IEEE, Fanshuo Li, Kun Cao , Peijin Cong ,

5 Tongquan Wei ,Member, IEEE, and Shiyan Hu , Senior Member, IEEE

6 Abstract—Software Defined Networking (SDN) provides flexibility and programmability for network management by using a layered

7 structure composed of data plane, control plane, and application plane. A key enabling technique for the sustainability of SDN-based

8 network infrastructure is the modeling of power consumed by SDN control planes. However, power modeling of control planes is not

9 extensively investigated yet, and no generic methods have been developed for performance and power comparison of sustainable

10 SDN control planes. In this paper, we propose analytical performance and power models for different network controllers by using

11 queuing theory, and design a generic framework for performance and power evaluation of different sustainable SDN control planes.

12 Extensive simulation results show that the proposed solution can precisely model the power and performance of the concerned SDN

13 control planes such that different control planes can be benchmarked under a general framework, which enables the identification of

14 suitable control planes for various SDN network applications.

15 Index Terms—Performance and power modeling, software defined networking (SDN), sustainable SDN control planes

Ç

16 1 INTRODUCTION

17 A traditional network device has a control plane that pro-18 vides information to build a forwarding table. It also19 includes a data plane with a forwarding table that is used by20 a network device to make decisions on where incoming21 frames or packets are to be sent. Both planes jointly support22 the fundamental operations of networking devices. Software23 Defined Networking (SDN) [1] is a novel approach that24 decouples the network control and forwarding functions. It25 enables the programmability of network control and abstrac-26 tion of underlying infrastructures for various network serv-27 ices and applications such as the defense of distributed28 denial-of-service (DDoS) attacks [2].29 The OpenFlow [3] protocol is a fundamental element for30 building SDN solutions. It is designed based on the concept31 of flow, and is the most popular protocol for the communi-32 cation between control plane and data plane in SDN. Each33 OpenFlow switch has a flow table that conducts packet34 lookup and forwarding. When a packet arrives at a switch,35 the packet is first matched against flow entries of flow table.

36If there is a match, the instruction set included in that flow37entry is executed. Otherwise, the packets may be dropped,38passed to another table or sent to a controller over the con-39trol channel via packet-in messages. If they are sent to a con-40troller, the controller will determine how to process these41packets and install forwarding rules in all switches on the42routing path of packets.43As the OpenFlow-based network scales up and the44number of switches increases, the OpenFlow-based net-45work becomes even more complex, and a single controller46cannot effectively manage the entire network. To tackle47this problem, various types of control planes are designed48in the literature [4], [5], [6], [7], [8], [9]. In this paper, as49shown in Table 1, we study control planes of 6 categories:50single-threaded single controllers (STSCs) [4], multithrea-51ding single controllers (MTSCs) [5], cluster controllers52(CLUCs) [6], two categories of flat structures (FSs) [7, [8]53and hierarchical structures (HSs) [9]. STSC is a single54controller with only one thread and processes only one55packet at a time. MTSC is also a single controller but runs56multiple threads and processes multiple packets at a time.57CLUC is logically a single controller but consists of multi-58ple physical controllers, and has a middle box for con-59necting the multiple controllers and switches. FS is a60distributed control plane with multiple controllers of the61same level. It can be further divided from aspects of local62and global view. In a local view strategy of flat structure63(LVFS), a controller has a local view of the network, while64in a global view strategy of flat structure (GVFS), each con-65troller has a global view of the network. HS has two levels66of controllers, that is, it has one root controller with multi-67ple leaf controllers.

� X. Huang, F. Li, K. Cao, P. Cong, and T.Wei are with the Computer ScienceDepartment, East China Normal University, Shanghai 200062, China.E-mail: {xlhuang, tqwei}@cs.ecnu.edu.cn, {51151201037, 52174506005,51164500019}@stu.ecnu.edu.cn.

� S. Hu is with the School of Computer Science and Electronic Engineering,University of Essex, Colchester CO4 3SQ, United Kingdom.E-mail: shiyan.hu@essex.ac.uk.

Manuscript received 19 Oct. 2017; revised 11 Nov. 2018; accepted 2 Dec.2018. Date of publication 0 . 0000; date of current version 0 . 0000.(Corresponding author: Tongquan Wei.)Recommended for acceptance by Y. Lin.For information on obtaining reprints of this article, please send e-mail to:reprints@ieee.org, and reference the Digital Object Identifier below.Digital Object Identifier no. 10.1109/TSUSC.2018.2889561

IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL. 4, NO. X, XXXXX 2018 1

2377-3782� 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See ht _tp://www.ieee.org/publications_standards/publications/rights/index.html for more information.

IEEE P

roof68 As an important component of SDN, the control plane of

69 an SDN network device processes a large number of packets.70 Therefore, the performance of the control plane, which71 specifically refers to the average time that the control plane72 takes to process a packet, needs to be evaluated. The perfor-73 mance of the control plane was evaluated via simulations or74 real-world experiments in the literature [10], [11]. Tootoon-75 chian et al. [10] presented a single-controller microbe-76 nchmark, which is an initial step towards the understanding77 of performance implications of SDN. Vishwasrao et al. [11]78 proposed a parallel algorithm for clustering and parallel79 processing of continuous position-based queries (CPQs).80 The algorithm can recognize solid clusters in the wireless81 search space area, and was validated by using simulation82 experiments. Although it is more reliable and realistic to83 study the control plane through experiments, experiment-84 based strategy of performance evaluation requires a prior85 construction of the concerned control plane, or a comprehen-86 sive simulation platform, which is prohibitive and time-87 consuming.88 Mathematical modeling of control planes is an alterna-89 tive approach for their performance evaluation. Analytical90 modeling of single-threaded single control planes has been91 investigated in [12], [13], [14]. Research efforts have also92 been made to the investigation of flat and hierarchical struc-93 tures in [15], [16], [17]. However, no analytical models have94 been established for complex control planes such as multi-95 threading single controllers. In addition, no generic evalu-96 ation methods have been designed for comparing different97 types of control planes in terms of performance.98 Power consumption is a central research and engineering99 topic for green SDNnetworks. It has been shown that control

100 plane [18] and data plane [19], [20] consume a large amount101 of power of SDN networks. The control plane serves as the102 operating system of the SDN network. It is a more crucial103 part of an SDN network compared to data plane. Thus, the104 evaluation of power consumption for control planes is a105 pressing research topic for SDN community [21], [22].106 To the best of our knowledge, no existing works [22], [23],107 [24], [25], [26], [27], [28] analytically model the power con-108 sumption of SDN control planes excluding STSCs. As a109 result, it is difficult to find a suitable control plane for a data110 plane from the point view of power consumption. In this111 paper, we propose a queueing theoretical framework to112 jointly model the performance and power consumption of113 SDN control planes. The proposed approach investigates the114 modeling of power and performance of various SDN control115 planes followed by the benchmarking of the concerned SDN

116control planes under an unified evaluation and comparison117framework. Themain contributions of this paper are summa-118rized as follows:

119� We propose analytical models for multithreading120controllers and cluster controllers, and improve121existing analytical models of flat structures and hier-122archical structures.123� We design a comparison framework for sustainable124SDN control planes based on the proposed analytical125models. The proposed framework is capable of ana-126lyzing and benchmarking both performance and127power of sustainable SDN control planes.128� We conduct extensive simulation experiments to ver-129ify the effectiveness of the proposed approach. Our130proposed queueing theoretic approach can precisely131model the power and performance of the concerned132SDN control planes.133The rest of this paper is organized as follows. Section 2134overviews preliminary knowledge of this work. Section 3135describes the proposed framework for control plane perfor-136mance and energy modeling. The effectiveness of the pro-137posed scheme is verified by simulation in Section 4, related138works are discussed in Section 5, and concluding remarks139are given in Section 6.

1402 PRELIMINARIES

1412.1 Flow Table Installation

142In OpenFlow-based SDN, each switch contains a flow table143designed for flow matching. As shown in Fig. 1, a packet-in144event occurs and a new flow is formedwhen a packet arrives145at a switch and no flow entry matches the packet. In this146case, the switch sends all packets of the new flow to the con-147troller, which in turn calculates the forwarding path of the148packets and installs the corresponding flow table entries in149all switches on the path. When a subsequent packet of the150same type arrives at the switch, instead of being sent to the151controller, it checks the flow table to find its forwarding path.

1522.2 Data Plane and Control Plane

153We consider using a data plane as a common benchmark to154compare the 6 types of control planes. Similar to the work in155[13], we use open Jackson network to approximate the data156plane in this paper, which leads to a number of subsequent

TABLE 1The Main Abbreviations Used in the Paper

Abbreviation Definition

STSC(stsc) single-threaded single controllerMTSC(mtsc) multithreading single controllerCLUC(cluc) cluster controllerLVFS(lvfs) local view strategy of flat structureGVFS(gvfs) global view strategy of flat structureRCHS(rchs) root controller of hierarchical structureLCHS(lchs) leaf controller of hierarchical structureHS(hs) hierarchical structure

Fig. 1. The controller installs the flow table entries for the switches.

2 IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL. 4, NO. X, XXXXX 2018

IEEE P

roof

157 assumptions. We assume that the arrival rate of external158 packets to any switches follows a Poisson distribution of the159 probability and all service times are exponentially distrib-160 uted. All queues of switches adopt the first-come-first-served161 (FCFS) policy.We also assume that the traffic intensities of all162 queues, that is, the ratios of the packet arrival rates to the ser-163 vice rates, are less than 1, and the probability of switches for-164 warding packets to a controller is greater than 0. An SDN165 network of these characteristics is an open Jackson network166 [29]. In this paper we aim to present analytical models for167 OpenFlow-based SDNs with more than one switch in the168 data plane.169 The assumed data plane contains C data plane domains,170 and each domain has Scðc ¼ 1; 2; . . . ; CÞ switches. The aver-171 age external packet arrival rate of a switch is subject to the172 Poisson distribution with parameter �c;sðc ¼ 1; 2; . . . ; C; s ¼173 1; 2; . . . ; ScÞ, where �c;s is the average number of external174 packet arrivals. Let PRfail

c;s denote the probability that175 external packets fail to match the flow table. The service rate176 of a switch or controller is assumed to be higher than its177 average packet arrival rate so that the system is maintained178 in a steady state. As a result, the rate of packets output from179 the assumed data plane to the control plane is given by

180

XC

c¼1

XSc

s¼1�c;s � PRfail

c;s .

181 The 6 types of control planes are assumed to have the182 same total service rate and total capacity. The total service183 rate is the sum of the average number of data packets that184 can be processed per unit time in a control plane, and the185 total capacity is the total number of packets that the control186 plane can accommodate. Specifically, assume that control187 plane LVFS and GVFS have the same number of controllers,188 which is denoted by C. Each controller in these control189 planes manages a domain indexed by c ðc ¼ 1; 2; . . . ; CÞ,190 and each domain contains Sc switches. The capacity and191 service rate of each controller in control plane LVFS and192 GVFS is assumed to be K and m, respectively. Given these193 parameters, the total service rate and capacity of control194 plane LVFS and GVFS are calculated as C � m and C �K,195 respectively. The number of controllers of control plane196 CLUC is assumed to be C. The capacity and service rate of197 each controller in CLUC is assumed to be 1 and m, res-198 pectively, and the capacity of the middle box, which is

199introduced to connect multiple controllers and switches in200CLUC, is C �K � C. Thus, the total capacity of controllers201and middle box is C �K. Since there is only one controller in202plane STSC and MTSC, the service rate of the controller is203given by C � m and the capacity is given by C �K. This con-

204troller manages C domains ofXC

c¼1Sc switches. We also

205assume that the control plane HS has one root controller206(RCHS) and C leaf controllers (LCHSs). As a result, the total207service rate and capacity of these ðC þ 1Þ controller is C � m208andC �K, respectively.209Fig. 2 illustrates the framework of an SDNnetwork. It con-210sists of a control plane of total service rateC � m and the capac-

211ity C �K, and a data plane ofXC

c¼1Sc switches. The control

212plane is composed of a set of controllers and communication213links between controllers. For the case of control plane214CLUC, it is logically a single controller but consists of multi-215ple physical controllers, and has a middle box for connecting216themultiple controllers and switches.

2172.3 Power Consumption Model

218The power consumption of a controller consisting of two219major components is given by [30]

P ¼ Pstat þ Pdyn; (1)221221

222where Pstat is the static power consumption and Pdyn is the223dynamic power consumption. The static power consumption224Pstat mainly depends on CMOS technology, and is assumed225to be a constant determined by hardware. The dynamic226power consumption, denoted byPdyn, can be expressed as

PdynðmxÞ ¼ Nxm1þ2

gx SðrxÞð0 < g � 1Þ; (2)

228228

229where mx is the service rate of the forwarding engine,Nx is a230constant determined by hardware and g is a parameter in231the range of ð0 < g � 1Þ. The SðrxÞ in the above equation is232defined as

SðrxÞ ¼ r1ax ¼ �x

mx

� �1a

;a > 1; (3)

234234

235where �x denotes the packet arrival rate and a is a constant.236The power consumption of a controller turns out to be

P ¼ Pstat þNm3x

�x

mx

� �1a

: (4)238238

239

240The power consumption given in Equation (4) is derived241by using queueing models, and can be utilized to model the242power of different types of control planes. However, the243Equation (4) only applies to a single-threaded controller,244thus, we convert the multithreading single controller and245cluster controller to single-threaded single controller in246Section 3.1. In this way, the power consumption of various247control planes can be derived by using this power model.

2483 PROPOSED FRAMEWORK FOR CONTROL PLANE

249PERFORMANCE AND ENERGY MODELING

250In this section, we first present the proposed queueing251theoretic modeling framework, then detail the modeling of252different types of control planes.

Fig. 2. An abstract representation of the control plane under a commoncapacity, service rate, and packets input rate.

HUANG ET AL.: QUEUEING THEORETIC APPROACH FOR PERFORMANCE-AWARE MODELING OF SUSTAINABLE SDN CONTROL PLANES 3

IEEE P

roof

253 Fig. 3 illustrates the flow for analysis and comparison of254 control planes. Abbreviations of various control planes255 are given in Table 1 and definitions of main notations256 are given in Table 2. As shown in Fig. 3, we first set up257 these control planes so that they have the same data plane,258 the total service rate and capacity. Then we propose ana-259 lytical models for single thread single controller (STSC),260 multithreading single controller (MTSC) and cluster con-261 troller (CLUC), after which, the control plane of flat and262 hierarchical structure is modeled. Finally, we compare263 these control planes in terms of average processing time264 and power consumption.265 As far as we know, no analytical modeling for multi-266 threading single controller and cluster controller has been267 conducted in the literature. In this paper, we analytically

268model the performance of these two types of control planes.269We also improve the performance model of control planes of270local view strategy and hierarchical structure. In particular,271this is the first attempt to analytically model the power con-272sumption of the 6 types of control planes in addition to their273performance.

2743.1 Modeling Single-Threaded Single275Controller (STSC)

276The single-threaded single controller (STSC) is the simplest277control plane structurewith only one controller. The capacity278of the controller isC �K, whereC is the number of data plane279domains andK is the capacity of a controller of flat structure.280Since the average packet arrival rate is subject to Poisson dis-281tribution and the service time follows exponential distribu-282tion, the controller can be modeled as a M=M=1=C �K283queue, where the first M indicates that the average packet284arrival rate is subject to Poisson distribution and the second285M shows that the service time of STSC follows exponential286distribution. The queue has one controller, the capacity of287which is given by C �K. The average packet arrival rate �stsc

288of STSC is given by

�stsc ¼XCc¼1

XScs¼1

�c;s � PRfailc;s ;

290290

291where �c;s is the average external packet arrival rate of switch292ðc; sÞ, PRfail

c;s is the probability of external packets that mis-293match the flow table entries and Sc is the number of switches294of domain c. The service rate is given by mstsc ¼ C � m, where295m is the service rate of a controller of flat structure. Therefore,296the traffic intensity rstsc of STSC is given by

Fig. 3. The proposed framework for performance and power modeling ofSDN control planes.

TABLE 2Definitions of Main Notations Used in the Paper

Notation Definition

x Abbreviation of a control plane or controllerC The number of domains in a data planec The index of a data plane domainSc The number of switches of the domain

cðc ¼ 1; 2; . . . ; CÞðc; sÞ or c; s The switch s in domain c

K Capacity of controllers of flat structureC �K Total capacity of control planesm Service rate of controllers of flat structureC � m Total service rate of control planes�c;s Average external packet arrival

rate of switch ðc; sÞPRfail

c;s Probability of external packets mismatchingflow table entries of switch ðc; sÞ

�x The packet arrival rate of the controller xmx The service rate of the controller xrx The traffic intensity of the controller xPRloss

x The packet loss rate of the controller xWs�x Average packet processing time of control

plane xLs�x The mean total number of packets of

control plane xNx Hardware dependent constant for

controller xPstat�x Static power consumption of the controller xPdyn�x Dynamic power consumption of the controller xPt�x Total power consumption of the control plane x

4 IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL. 4, NO. X, XXXXX 2018

IEEE P

roof

rstsc ¼XC

c¼1

XSc

s¼1�c;s � PRfail

c;s

C � m :

298298

299 According to queueing theory [31], the steady-state distri-300 bution of the controller is

pk ¼ 1� rstsc

1� rC�Kþ1stsc

rkstscðk ¼ 0; 1; 2; ::; C �KÞ: (5)

302302

303 Then we obtain the packet loss rate PRlossstsc of STSC

PRlossstsc ¼ pC�K ¼ 1� rstsc

1� rC�Kþ1stsc

rC�Kstsc : (6)

305305

306 Hence, the mean total number Lstsc of packets of STSC is

Ls�stsc ¼ rstsc1� rstsc

� ðC �K þ 1ÞrC�Kþ1stsc

1� rC�Kþ1stsc

: (7)

308308

309 Using Little’s Law, we get the average processing time310 Ws�stsc of a packet of STSC, which is given by

Ws�stsc ¼ Ls�stsc

�stscð1� PRlossstscÞ

: (8)

312312

313 The Little’s Law is the queueing approachwhere the average314 processing time equals the ratio of the average number of315 packets in the queue to the average packet arrival rate [31].316 According to Equation (4), the power consumption Pt�stsc

317 of STSC is expressed as

Pt�stsc ¼ Pstat�stsc þNstscðC � mÞ3�½ð1� PRloss

stscÞPC

c¼1

PScs¼1 �c;s � PRfail

c;s �C � m

!1a

;(9)

319319

320 where Nstsc denotes the constant determined by the hard-321 ware of the STSC controller, and Pstat�stsc incidates the static322 power consumption of the control plane.

323 3.2 Modeling Multithreading Single324 Controller (MTSC)

325 A multithreading single controller (MTSC) simultaneously326 runs multiple threads to calculate routing paths of packets.327 It can be modeled by using M=Mr=1=C �K queue, indicting328 that the service time of the MTSC controller is subject to329 exponential distribution and the MTSC controller processes330 rð1 � r � C �K � 1Þ packets per service. C is the number of331 data plane domains, K is the capacity of a controller of flat332 structure, and C �K is the capacity of the MTSC controller.333 Packets will not be processed until the number of packets in334 the MTSC controller reaches r.335 Before analyzing the M=Mr=1=C �K queue, we consider336 M=Mr=1=1. The average packet arrival rate �mtsc of the337 queue is given by

�mtsc ¼XCc¼1

XScs¼1

�c;s � PRfailc;s ;

339339

340 and its service rate is mmtsc ¼ C � m, where �c;s is the average341 external packet arrival rate of switch ðc; sÞ, PRfail

c;s is the342 probability of external packets that mismatch flow table343 entries, Sc is the number of switches of domain c and m is

344the service rate of a controller of flat structure. Given these,345the traffic intensity rmtsc of MTSC is given by

rmtsc ¼XC

c¼1

XSc

s¼1�c;s � PRfail

c;s

C � m :347347

348

349According to queueing theory [31], the steady-state dis-350tribution of the number of packets in the controller is

pk ¼ 1� 1

u0

� �1

uk0

ðk ¼ 0; 1; 2; . . .Þ; (10)

352352

353where u0 is the unique solution of Equation (11) that is354greater than 1. The equation is given by

1þ rmtscurþ1 � ð1þ rmtscÞur ¼ 0; (11)

356356

357and u0 is given by

u0 ¼ 1þ 1

rmtsc

� 1

rmtscur0

: (12)359359

360

361The value of u0 is determined by variable r. For362each value of r, u0 has a unique value corresponding363to it. Thus, u0ðrÞ is a sequence of r. Substituting u0 in364(11), Equation (11) becomes equivalent to Equation (12).365As r approaches þ1, u0ðrÞ approximates to 1þ 1

rmtsc.

366Since

u0ðrþ 1Þ � u0ðrÞ ¼ u0 � 1

rmtscurþ10

> 0;

368368

369u0ðrÞ is a monotonic increasing sequence with limit of3701þ 1

rmtsc. Considering the fact that u0ð1Þ ¼ 1

rmtsc, we can get

371the minimum and supremum of the sequence. We then372extend the definition domain of u0ðrÞ to ½1;þ1Þ, such that373u0ðrÞ becomes a continuous function. According to interme-374diate value theorem, each item of the sequence can be375expressed as

" � 1

rmtsc

þ ð1� "Þ � 1þ 1

rmtsc

� �ð0 < " < 1Þ:

377377

378For simplicity, we take " ¼ 12. Therefore, the value of u0 is

37912 þ 1

rmtscand the value of r0 is logu0

2rmtsc

. We take these values

380to represent the average number of packets processed by

each controller in a batch.381Next we analyze the M=Mr=1=C �K model. When the382upper bound on k is C �K, according to queueing theory383[31], the steady-state distribution of the number of packets384in the controller is

pk ¼ 1

1�P1i¼C�Kþ1ð1� 1

u0Þ 1ui0

1� 1

u0

� �1

uk0

¼ 1

1� 1uC�Kþ10

1� 1

u0

� �1

uk0

ðk ¼ 0; 1; 2; . . . ; C �KÞ:(13)

386386

387

PRlossmtsc ¼ pC�K ¼ 1

1� 1uC�Kþ10

1� 1

u0

� �1

uC�K0

: (14)

389389

HUANG ET AL.: QUEUEING THEORETIC APPROACH FOR PERFORMANCE-AWARE MODELING OF SUSTAINABLE SDN CONTROL PLANES 5

IEEE P

roof

390 Thus, the mean total number Ls�mtsc of packets of MTSC is

Ls�mtsc ¼XC�K

k¼0

kpk ¼XC�K

k¼0

k1

1� 1

uC�Kþ10

1� 1

u0

� �1

uk0

392392

393

¼ 1

uC�Kþ10 � 1

uC�Kþ10 � u0

u0 � 1� C �K

� �: (15)

395395

396

397 Using Little’s Law [31], the average processing time398 Ws�mtsc of a packet of MTSC is calculated as

Ws�mtsc ¼ Ls�mtsc

�mtscð1� PRlossmtscÞ

: (16)400400

401

402 MTSC controller has r0 synchronous processors and a403 queue, and a processor can be approximated by a single-404 threaded controller. Consequently, using Equation (4), the405 total power consumption (Pt�mtsc) of MTSC is calculated as

Pt�mtsc ¼ r0 � Pstat�mtsc þ r0 �Nmtsc�

ðC � mÞ3 ð1� PRlossmtscÞ

PCc¼1

PScs¼1 �c;s � PRfail

c;s

r0 � C � m

!1a

;(17)

407407

408 where Nmtsc denotes the hardware dependent constant and409 Pstat�mtsc indicates the static power consumption of each410 thread of the control plane MTSC.

411 3.3 Modeling Cluster Controller (CLUC)

412 The cluster controller is logically a centralized cluster but413 physically consists of multiple controllers. In this structure414 [6], incoming packets first converge and queue in the mid-415 dle box such as a load balancer, then are forwarded to idle416 controllers. A controller in the cluster has no queue. When417 one or more controllers in the cluster are idle, the first418 packet dequeues for one of idle controllers. Otherwise, no419 controllers are idle and all packets queue in the load bal-420 ancer. Thus we adopt a M=M=C=C �K queue to model the421 system that includes a load balancer and multiple control-422 lers, where C represents the number of controllers and423 C �K gives the capacity of the control plane.424 Given the number C of CLUC controllers in a cluster, the425 average packet arrival rate �cluc of the cluster is represented by

�cluc ¼XCc¼1

XScs¼1

�c;s � PRfailc;s ;

427427

428 where �c;s is the average external packet arrival rate of a429 switch ðc; sÞ, PRfail

c;s is the probability of external packets mis-430 matching flow table entries and Sc is the number of switches431 of domain c. The service rate of the control plane is432 mcluc ¼ C � m and the capacity of middle box is C �K, where433 m andK is the service rate and capacity of a controller of flat434 structure, respectively. The traffic intensity rcluc of CLUC is435 thus given by

rcluc ¼XC

c¼1

XSc

s¼1�c;s � PRfail

c;s

C � m :437437

438

439 According to queueing theory [31], the steady-state dis-440 tribution of the number of packets in the middle box is

pk ¼k ¼ 0 p0;

1 � k < C ðC�rclucÞkk! p0; ðk ¼ 0; 1; 2; . . . ; C �K:Þ

k � C ðC�rclucÞkC!Ck�C p0;

8>><>>:

(18) 442442

443SincePC�K

k¼0 pk = 1, p0 is given by

p0 ¼XC�1

k¼0

ðC � rclucÞkk!

þXC�K

k¼C

ðC � rclucÞkC!Ck�C

" #�1

: (19)

445445

446Then we obtain the packet loss rate PRlosscluc of CLUC

PRlosscluc ¼ pC�K ¼ ðC � rclucÞC�K

C!CC�K�Cp0; (20)

448448

449and the mean total number Ls�cluc of packets in the control450plane

Ls�cluc ¼XC�K

k¼0

kpk

¼ C � rclucð1� PRlossclucÞ þ

p0 � rclucðC � rclucÞCC!ð1� rclucÞ2

� ½1� ðC �K � C þ 1ÞrC�K�Ccluc

þ ðC �K � CÞrC�K�Cþ1cluc �:

(21)

452452

453Using Little’s Law [31], the average processing time Ws�cluc

454of a packet of CLUC control plane is given by

Ws�cluc ¼ 1

mcluc

þ p0 � rclucðC � rclucÞCC!ð1� rclucÞ2�clucð1� PRloss

clucÞ� ½1� ðC �K � C þ 1ÞrC�K�C

cluc

þ ðC �K � CÞrC�K�Cþ1cluc �:

(22)

456456

457

458CLUC control plane has multiple mutually independent459controllers that share a queue. We regard the CLUC control460plane as multiple STSCs, calculate the sum of their power461consumptions, and take it as the total power consumption462Pt�cluc of CLUC, that is

Pt�cluc ¼ C � Pstat�cluc þ C �Ncluc

� m3ð1� PRloss

clucÞðPC

c¼1

PScs¼1 �c;s � PRfail

c;s ÞC � m

!1a

;(23)

464464

465where Ncluc and Pstat�cluc is the hardware-dependent con-466stant and the static power consumption of a controller of467CLUC control plane, respectively.

4683.4 Modeling Flat Structure

469Flat structures have two strategies: local view strategy (LVFS)470and global view strategy (GVFS). At the LVFS control plane,471controllers communicate with each other and generate pack-472ets, which are different from those forwarded by data plane.473On the contrary, GVFS control plane does not generate474packets.475Local View Strategy of Flat Structure (LVFS). Under the476LVFS control strategy, each controller is modeled as a477M=M=1=K queue, where K represents the capacity of each478controller. Each controller has only local information and

6 IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL. 4, NO. X, XXXXX 2018

IEEE P

roof

479 can only process local flows. When a global flow is for-480 warded to a controller, the controller will split it into a local481 flow and a global flow, which is forwarded to one of its482 neighboring controllers [17]. When the global flow arrives483 at the neighboring controller, the controller also splits the484 global flow into new local and global flows. This process485 repeats until no global flows remain. If the average hops486 between controllers is d, the original global flow will even-487 tually produce dþ 1 local flows. Because the topology of488 controllers is unknown, for the sake of simplicity, we489 take d ¼ Cþ1

6 , which is half of the average number of hops in490 the linear topology. The linear topology is the topology with491 the longest average hop counts.492 Assume that the probabilities of sending packets between493 switches are equal, then we can get the probabilities of494 whether an external flow is global or not. Let PRglb

c be the495 probability that a new flow arriving at domain c is global and496 PRnglb

c be the probability that the flow is local. Then the two497 probabilities are calculated as

PRglbc ¼

A1Sc

� A1CecpðcÞ

A1Sc

�A1CecpðcÞ þA2

Sc

; (24)499499

500

PRnglbc ¼ A2

Sc

A1Sc

�A1CecpðcÞ þA2

Sc

; (25)

502502

503 where A denotes the number of permutation, C is the num-504 ber of controllers, Sc is the number of switches of domain c,

505 and CecpðcÞ ¼XC

m¼1;m 6¼cSm. When a new packet arrives at

506 a switch, if the packet belongs to a global flow, it will be for-

warded to one or several controllers. The probability that a

packet arriving at a switch belongs to a global flow is PRglbc ,

and the probability that a packet does not belongs to aglobal flow is PRnglb

c . Thus, when a new packet arrives at a

switch, the mean total number of packets all controllers will

generate is

nc ¼ PRnglbc þ dþ 1

C � 1�XC

t¼1;t 6¼c

PRglbc : (26)

507 Given these, the average packet arrival rate �lvfs�c of each508 controller in this structure is calculated as

�lvfs�c ¼ nc �XScs¼1

�c;s � PRfailc;s ;

510510

511 and the service rate is derived asmlvfs�c ¼ m, where �c;s is the512 average external packet arrival rate of each switch, PRfail

c;s is513 the probability of external packets mismatching flow table514 entries of switch ðc; sÞ and m is the service rate of a controller.515 Thus, the traffic intensity rlvfs�c of the controller c is

rlvfs�c ¼XSc

s¼1�c;s � PRfail

c;s

m:

517517

518 Using Little’s Law [31], we obtain the processing time519 Ws�lvfs of a packet and the mean total number Ls�lvfs of520 packets of LVFS control plane, that is

Ls�lvfs ¼XCc¼1

rlvfs�c

1� rlvfs�c

� ðK þ 1ÞrKþ1lvfs�c

1� rKþ1lvfs�c

" #; (27)522522

523Ws�lvfs ¼ Ls�lvfsPCc¼1 �lvfs�c � ð1� PRloss

lvfs�cÞ; (28)

525525

526PRloss

lvfs�c ¼1� rlvfs�c

1� rKþ1lvfs�c

rKlvfs�c; (29)

528528

529where C is the number of controllers, which is equal to the530number of data plane domains, and K is the capacity of531each controller in this structure. Thus, the power consump-532tion Pt�lvfs of the control plane is calculated as

Pt�lvfs ¼ C � Pstat�lvfs þXCc¼1

Nlvfs

� m3nc � ð1� PRloss

lvfs�cÞPSc

s¼1 �c;s � PRfailc;s

m

!1a

;

(30)

534534

535where Nlvfs and Plvfs denotes the hardware-dependent con-536stant and the static power consumption of a LVFS control-537ler, respectively.538Global View Strategy of Flat Structure. In GVFS strategy,539each controller has global information and does not need to540split any global flows. Similar to LVFS, a GVFS controller is541also modeled by using M=M=1=K queue, where K repre-542sents the capacity of the controller. Under global view strat-543egy, the average packet arrival rate �gvfs�c of the controller c is

�gvfs�c ¼XScs¼1

�c;s � PRfailc;s ;

545545

546and the service rate mgvfs�c is m, where �c;s is the average547external packet arrival rate of a switch ðc; sÞ, PRfail

c;s is548the probability of external packets mismatching flow table549entries of switch ðc; sÞ, Sc is the number of switches of550domain c and m is the service rate of a controller. Then we551derive the traffic intensity rgvfs�c, which is given by

rgvfs�c ¼PSc

s¼1 �c;s � PRfailc;s

m: 553553

554

555Using Little’s Law [31], we derive the average processing556time Ws�gvfs�c of a packet and the mean total number557Ls�gvfs�c of packets of GVFS as follows:

Ls�gvfs ¼XCc¼1

rlvfs�c

1� rgvfs�c

� ðK þ 1ÞrKþ1gvfs�c

1� rKþ1gvfs�c

" #; (31) 559559

560

Ws�gvfs ¼ Ls�gvfsPCc¼1 �gvfs�c � ð1� PRloss

gvfs�cÞ; (32) 562562

563

PRlossgvfs�c ¼

1� rgvfs�c

1� rKþ1gvfs�c

rKgvfs�c; (33)

565565

566where C is the number of controllers andK is the capacity of567each controller in this structure. Given these, the power con-568sumptionPt�gvfs of controllers of this structure is derived as

Pt�gvfs ¼ C � Pstat�gvfs þXCc¼1

Ngvfs

� m3

PScs¼1 �c;s � PRfail

c;s ð1� PRlossgvfs�cÞ

m

!1a

;

(34)

570570

HUANG ET AL.: QUEUEING THEORETIC APPROACH FOR PERFORMANCE-AWARE MODELING OF SUSTAINABLE SDN CONTROL PLANES 7

IEEE P

roof

571 where Ngvfs and Pgvfs denotes the hardware-dependent con-572 stant and the static power consumption of a controller of573 GVFS, respectively.

574 3.5 Modeling Hierarchical Structure

575 The hierarchical structure can be classified into root control-576 ler of hierarchical structure (RCHS) and leaf controller of577 hierarchical structure (LCHS). The RCHS is modeled as578 M=M=1= C�K

2 queue [31] and the LCHS is modeled as C

579 M=M=1= K2 queue.

580 LCHS controller cannot process global flows. When a581 global flow arrives, it is first forwarded to the RCHS control-582 ler, then is forwarded to relevant LCHS controllers for proc-583 essing. Under LVFS control strategy, the arrival rates of584 RCHS (�lchs�c) and LCHS (�rchs) are given by

�lchs�c ¼XScs¼1

�c;s � PRfailc;s ½1þ

dþ 1

C � 1�XC

t¼1;t 6¼c

PRglbc �;

�rchs ¼XCc¼1

PRnglbc

XScs¼1

�c;s � PRfailc;s ;

586586

587 where �c;s is the average external packet arrival rate of switch588 ðc; sÞ, d is average hops between controllers, PRfailc;s is the589 probability of external packets mismatching flow table590 entries of switch ðc; sÞ, Sc is the number of switches the leaf591 controller cmanages and PRglb

c indicates the probability that592 a newflow is a global flow. Thenwe derive service intensities593 of LCHS (rlchs) and RCHS controllers (rrchs), that is

rlchs ¼2XScs¼1

�c;s � PRfailc;s ½1þ

dþ 1

C � 1�XC

t¼1;t 6¼c

PRglbc �

m;

rrchs ¼2XCc¼1

PRnglbc

XScs¼1

�c;s � PRfailc;s

C � m :595595

596

597 Using Little’s Law [31], we obtain the mean total number598 Ls�lchs�c of packets and average processing time Ws�lchs�c

599 of a packet of a LCHS controller as follows:

Ls�lchs�c ¼ rlchs�c

1� rlchs�c

� ðK2 þ 1ÞrK2þ1

lchs�c

1� rK2þ1

lchs�c

; (35)601601

602

Ws�lchs�c ¼ Ls�lchs�c

�lchs�cð1� PRlosslchs�cÞ

; (36)604604

605

PRlosslchs�c ¼

1� rlchs�c

1� rK2þ1

lchs�c

rK2lchs�c; (37)

607607

608 where PRlosslchs�c is the packet loss rate of a controller of the

609 LCHS control plane and K is the capacity of a controller of610 flat structure.611 Furthermore, we obtain the mean total number Ls�rchs of612 packets and average processing time Ws�rchs of a packet of613 a LCHS controller as follows:

Ls�rchs ¼ rrchs1� rrchs

� ðC�K2 þ 1Þr

C�K2 þ1

rchs

1� rC�K2 þ1

rchs

; (38)615615

616Ws�rchs ¼ Ls�rchs

�rchsð1� PRlossrchsÞ

; (39)

618618

619PRloss

rchs ¼1� rrchs

1� rC�K2 þ1

rchs

rC�K2

rchs; (40)

621621

622where PRlossrchs is the packet loss rate of the rchs controller.

623Subsequently, we derive the mean total number Ls�hs of624packets and average processing time Ws�hs of a packet of625HS control plane as follows:

Ls�hs ¼XCc¼1

Ls�lchs�c þ Ls�rchs: (41)627627

628

Ws�hs ¼ 1

C

XCc¼1

½PRglbc ðWs�rchs þ 2Ws�lchs�cÞ

þ PRnglbc �Ws�lchs�c�:

(42)

630630

631

632The power consumption Pt�hs of the HS control plane is633thus calculated as

Pt�hs ¼ C � Pstat�lchs þ Pstat�rchs þXCc¼1

Nlchsðm2Þ3

� ð2XScs¼1

�c;s � PRfailc;s ½1þ

dþ 1

C � 1�XC

t¼1;t 6¼c

PRglbc �

mÞ1a

þNrchsðC � m2

Þ3ð2XCc¼1

PRnglbc

XScs¼1

�c;s � PRfailc;s

C � m Þ1a;

(43)

635635

636where Nlchs denotes the hardware-dependent constant of a637leaf controller, Nrchs denotes that of the root controller,638Pstat�lchs is the static power consumption of a leaf controller,639and Pstat�rchs is that of the root controller.

6404 SIMULATION-BASED EVALUATION

641In this section, we present simulation results for 6 control642planes including single-threaded single controllers (STSCs),643multithreading single controllers (MTSCs), cluster control-644lers (CLUCs), local strategy of flat structures (LVFSs), global645strategy of flat structures (GVFSs) and hierarchical struc-646tures (HSs). We compare the processing time and power647consumption of these control planes by assuming a com-648mon data plane, the same total capacity and service rate.649The total capacity is the total number of packets that a con-650trol plane can accommodate, and the total service rate is the651sum of the average number of packets that can be processed652by controllers per unit time. To conduct a fair comparison653of the 6 control planes, we use the same number for packets654each controller of LVFS, GVFS, CLUC, and HS structure655needs to process. In other words, the number of switches656per domain of the data plane is equal, the packet arrival rate657of each switch is equal, and the probability of external pack-658ets mismatching flow table entries of each switch is also659equal in the simulation. Specifically, we set S1 ¼ � � �Sc ¼660� � � ¼ SC ¼ S, �1;1 ¼ � � � ¼ �c;s ¼ � � � ¼ �C;SC ¼ � and

661PRfail1;1 ¼ � � � ¼ PRfail

c;s ¼ � � � ¼ PRfailC;Sc

¼ PR1.

662We use Simulink of MATLAB to simulate the process of663packets queueing and subsequent processing in controllers,

8 IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL. 4, NO. X, XXXXX 2018

IEEE P

roof

664 and obtain the simulation results of packet processing time.665 Simulink defines how each control plane processes packets.666 It also initializes the number of controllers, the packet arrival667 rate, and the capacity and service rate of controllers. Once668 these parameters are determined, simulation is conducted,669 and simulation results of packet processing time are com-670 pared with analytical results of the 6 types of control planes.671 We useNS-3 [32] to simulate the power consumption of the 6672 types of control planes. NS-3 provides controller and power673 consumption modules compatible with OpenFlow protocol.674 As a node in the network, the controller uses the power con-675 sumption module to simulate the power consumed by the676 controller under different traffic intensities. In NS-3, we first677 define the number of controllers, the average packet arrival678 rate, and the capacity and service rate of each controller.679 After these parameters are determined, the power consump-680 tion of each controller can be obtained by running NS-3. The681 power consumption of a control plane is the sum of power682 consumptions of all controllers in the control plane.683 The 6 control planes are also analytically compared684 with respect to processing time and power consumption.685 Two aspects are considered. We first study how processing686 time and power consumption of control planes change with687 the average packet arrival rate � of a switch, we then inve-688 stigate how processing time and power consumption vary689 with the number of data plane domains C. Note that C is690 the key characteristic that defines a data plane. We varies691 the value of C such that the 6 control planes can be bench-692 marked under different data planes.

693 4.1 Simulation Settings

694 Table 3 shows the parameters used in the experiment. The695 probability that external packets mismatching flow table696 entries is PR1 = 0.04 [16]. The typical maximal capacity of697 control plane STSC and MTSC is C �K = 512 [33]. Consider-698 ing the fact that themaximumnumber of data plane domains699 is C ¼ 20, the capacity of a controller of flat structure is thus700 calculated asK ¼ 26. The number of switches that a control-701 ler can support depends on the SDN use cases that are being702 supported. In our simulation settings, the number of703 switches of each data plane domain is set to 3, that is, S ¼ 3.704 This is in fact an empirical value obtained in experiments705 and is widely used in the literature [34]. Similarly, the typical

706value of service rate of a controller of control plane CLUC,707GVFS and LVFS is 2, that is, m ¼ 2 (packets/ms) [12]. Thus708the probabilities that a global flow arrives at all domains are709equal. As a result, the probability that a new flow arriving at710domain c is global can be derived according to Equation (24).711The probability is denoted by PRglb

c and is given by

PRglb1 ¼ � � � ¼ PRglb

c ¼ � � � ¼ PRglbC ¼ PRglb ¼ C � 1

C � 13

:713713

714

715In local view strategy (LVFS) and the hierarchical struc-716ture (HS), controllers communicate with each other and717generate additional packets. Since each queueing system718maintains a steady state, for LVFS and HS structures, the719traffic intensity of a controller in the LVFS is also less than 1720(i.e., rlvfs < 1), the traffic intensity of each leaf controller in721the HS is less than 1 (i.e., rlchs < 1). We then derive the722range of � in packets/ms

� <m

2ð1þ C�2C�4

3

� ðCþ16 þ 1ÞÞPR1 � S

:

724724

725Similarly, for control plane STSC, MTSC, CLUC, and GVFS,726the � (packets/ms) is less than m

PR1�S.727Similar to the work in [34], we use the exponential power728model to estimate the power consumption of control planes,729and set the exponent a ¼ 2, where a is a constant in Equa-730tion (4), which is re-written as below

P ¼ Pstat þNm3x

�x

mx

� �1a

: 732732

733

734We assume 90-65 nm technology for processors of the 6735different types of control planes. Since the static power con-736sumption accounts for about 42-50 percent of the total737power in 90-65 nm processors [30], the ratio of static power738consumption to the total power consumption is given by

h ¼ Pstat�x

Pstat�x þNxm3xð�xmx

Þ12; (44)

740740

741where Pstat�x denotes the static power consumption of the742controller x, Nx denotes the hardware-dependent constant743of the controller, �x denotes the average packet arrival rate744of the controller, mx denotes the service rate of the control-745ler, and h falls in the range of ½0:42; 0:50�. Considering the746fact that dynamic power is maximized when �x ¼ mx, that747is, the static power approximately accounts for 42 percent of748the total power in this case, the total power consumption is749then estimated and given by

Pt�x ¼ 21

29þ��x

mx

�12

" #�Nx � m3

x: (45)

751751

752Since control planes are assumed to adopt the same technol-753ogy, their hardware dependent constants are equal to each754other, that is, Nstsc ¼ Nmtsc ¼ Ncluc ¼ Nlvfs ¼ Ngvfs ¼ Nlchs ¼755Nrchs ¼ N0. For the sake of easy presentation, we normalize756the 6 constants with respect toN0, then the power consump-757tion of control plane Pt�x is calculated as

Pt�x ¼�21

29þ��x

mx

�12�� m3

x: (46) 759759

760

TABLE 3The Main Parameters Used in the Experiment

Parameter Description value

PR1 The probability that external packets mismatching

flow table entries

0.04

C The maximum number of data plane domains 2; . . . ; 20

K The capacity of a controller of flat structure 26

S The number of switches of each data plane domain 3

m Service rate of a controller of control plane CLUC,

GVFS and LVFS

2

PRglbc The probability that a new flow arriving at domain c

is global

C�1C�1

3

�c;s Average external packet arrival rate of switch ðc; sÞ ð0; mPR1 �SÞ

Nx Hardware dependent constant for controller x 1

a A constant used to calculate power consumption 2

h Ratio of static power consumption to total power

consumption

½0:42; 0:50�

HUANG ET AL.: QUEUEING THEORETIC APPROACH FOR PERFORMANCE-AWARE MODELING OF SUSTAINABLE SDN CONTROL PLANES 9

IEEE P

roof

761 4.2 Performance Comparison and Analysis

762 The Figs. 4, 5, 6 and 7 give the processing time of different con-763 trol planes. The solid lines represent the processing time764 derived using analytical equations, while the dashed lines (X-765 SIM in legends) indicate the processing time obtained using766 Simulink simulator. The proposed modeling is based on767 queuing theory and is verified by simulations. The 95 percent768 confidence interval is given for the simulation results.769 It can be seen from Fig. 4 that the average processing time770 of control planes increase as � increases. The average proc-771 essing time of the hierarchical structure (HS) is far greater772 than that of other 5 control planes. This is because the leaf773 controllers in the HS structure need to handle at least 2 times774 the packets the controllers of GVFS and LVFS structures775 need to process while it has only half the service rate of these776 controllers. Moreover, packets in the HS structure may be777 processed by leaf controllers multiple times. For example,778 when a global packet arrives at a leaf controller (i.e., local779 controller), the leaf controller cannot process the packet,780 hence needs to send the packet to the root controller, which781 in turn sends the packet to relevant leaf controllers. This pro-782 cess incurs significant amount of time overheads. In addition783 to this, due to higher packet arrival rate, the processing time784 of the local view strategy of flat structure (HS) is always lon-785 ger than that of control plane GVFS and LVFS. The process-786 ing time of control plane GVFS and LVFS are almost787 constant, since the range of their packet input rates is small.788 Fig. 5 demonstrates the difference in the averagte proc-789 essing time of control plane STSC, MTSC, CLUC and GVFS.

790The processing time of the global view strategy of flat struc-791ture (GVFS) is the longest in the 4 control planes. In addition,792when � increases, the processing time of CLUC controller is793close to the processing time of the STSC controller. This is794because the number of idle controllers in the CLUC structure795decreases. It can also be seen from the figure that MTSC con-796troller spends theminimal timewhen processing a packet.797As shown in Fig. 6, when C increases, the average process-798ing time of HS and LVFS structure increases. As compared to799processing time of other control planes, the processing time of800HS is longer due to the increased number of global packets801generated by the HS structure. As shown in Fig. 7, regardless802of what valueC takes, the average processing time of GVFS is803constant. This is because when C changes, the parameters of804each controller remain unchanged. However, the processing805time of control plane STSC, MTSC and CLUC decreases. The806average processing time of CLUC is longer than that of STSC,807which is in turn longer than that ofMTSC.

8084.3 Power Consumption Comparison and Analysis

809The Figs. 8, 9, 10 and 11 give the power consumptions of dif-810ferent control planes. The solid lines represent the power811consumption derived using analytical equations, while the812dashed lines (X-SIM in legends) indicate the power con-813sumptions obtained using NS-3 simulator. It can be seen814from these figures that the analytical results generated using815the proposed models are consistent with the simulation816results produced by NS-3. The 95 percent confidence inter-817vals are used in the comparison.

Fig. 4. The average processing time of the three control planes varieswith average packet arrival rate of each switch ð�Þ, where C is the num-ber of data plane domains.

Fig. 5. The average processing time of the four control planes varies withaverage packet arrival rate of each switch ð�Þ, where C is the number ofdata plane domains.

Fig. 6. The average processing time of the three control planes varieswith the number of data plane domains ðCÞ, and average packet arrivalrate of each switch takes different values.

Fig. 7. The average processing time of the four control planes varies withthe number of data plane domains ðCÞ, and average packet arrival rateof each switch takes different values.

10 IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL. 4, NO. X, XXXXX 2018

IEEE P

roof

818 We then compare the power consumptions of different819 control planes. Fig. 8 shows the impact of average packet820 arrival rate on power consumption of control planes. It can821 be seen from Fig. 8 that the power consumptions of the 6822 control planes increase with �, and the MTSC control planes823 consumes the most power as compared to other control824 planes. This is because MTSC has more than one processors.825 It can also be seen from Fig. 8 that the power consumption826 of the HS control plane increases faster than that of CLUC,827 GVFS, LVFS, and STSC. The reason is that inter-controller828 communication overhead increases with the increase of C.829 Fig. 9 shows that the power consumptions of control plane830 STSC, MTSC, CLUC and GVFS increase with �. Since these831 4 control planes have the same traffic intensity while MTSC832 has a higher service rate, MTSC consumes the most power.833 CLUC and GVFS consume approximately the same power834 since they have the same traffic intensity and service rate.835 Finally, STSC consumes slightly more power as compared836 to CLUC and GVFS as C increases.837 Fig. 10 illustrates changes of power consumptions as C838 increases. It can be seen that power consumptions of all con-839 trol planes increase with C. Since the inter-controller com-840 munication overhead increases with the increase of C, the841 power consumption of HS control planes increases faster842 than that of MTSC, STST, CLUC, GVFS and LVFS control843 plane. The power consumptions of the CLUC, GVFS and844 LVFS are close to each other when � ¼ 0:32=ms. This is845 because CLUC, GVFS and LVFS have the same service rate,

846and GVFS has near zero inter-controller communication847overheads. As shown in Fig. 11, the power consumption of848MTSC is higher than that of STSC, CLUC and GVFS, and the849differences between MTSC and STSC grow as � increases.850This is because MTSC has higher dynamic power consump-851tionwhen � increases.

8525 RELATED WORKS

853Extensive research efforts have been made to the investi-854gation of analytical modeling of control planes for their855performance evaluation. Queueing theory is a popular856approach to modeling different types of control planes. As857for STSC control plane, Goto et al. [12] developed a queue-858ing model of an OpenFlow-based SDN that takes into859account the processing of packets arriving at a switch. Miao860et al. [13] presented a new analytical model to investigate861the performance of SDN. A priority queue system has been862adopted to model the SDN data plane to capture the multi-863queue nature of forwarding devices. Yao et al. [14] modeled864the flow setup requests from switches to a controller as a865batch arrival process Mk=M=1. The controller performance866is analyzed and the average flow service time is derived by867using queuing theory. However, these works are not suited868to the performance modeling of multi-controller structures.869As the OpenFlow-based network scales up, a single870controller cannot effectively manage the entire network,871thus, multi-controller models also have been studied in the

Fig. 8. The power consumption of the six control planes varies with aver-age packet arrival rate of each switch ð�Þ, where C is the number of dataplane domains.

Fig. 9. The power consumption of the four control planes varies with theaverage packet arrival rate of each switch ð�Þ, where C is the number ofdata plane domains.

Fig. 10. The power consumption of the six control planes varies with thenumber of data plane domains ðCÞ, and average packet arrival rate ofeach switch takes different values.

Fig. 11. The power consumption of the four control planes varies with thenumber of data plane domains (C) and the average packet arrival rate ofeach switch.

HUANG ET AL.: QUEUEING THEORETIC APPROACH FOR PERFORMANCE-AWARE MODELING OF SUSTAINABLE SDN CONTROL PLANES 11

IEEE P

roof

872 literature [15], [16], [17] by using the queueing theory.873 Fu et al. [15] presented a flexible dormant multi-controller874 model. However, the model is only applicable to the parti-875 cular dormant multi-controller. Wang et al. [16] modeled876 the controller of hierarchical structure and analyzed the877 controller’s performance. Hu et al. [17] proposed a metric of878 scalability for SDN control planes, and studied performance879 of 4 types of SDN control planes including STSC, LVFS,880 GVFS and HS. However, the above two works ignore the881 probability that the arrival flow is directly processed by leaf882 controller of hierarchical structure. In addition, power con-883 sumption is not considered in these performance analytical884 models and performance benchmarking among different885 types of control planes is also not investigated.886 The problem of modeling the power consumption of SDN887 control planes deployed in data centers has been investigated888 from different perspectives in the literature [22], [23], [24],889 [25], [26], [27], [28]. Carlinet et al. [22] presented a mixed inte-890 ger linear programming formulation to compute the optimal891 requests assignment to data centers and demonstrated how892 SDN allows saving energy in a network of data centers. Kuap893 et al. [23] introduced a power measurement framework and894 developed a plugin for the Floodlight OpenFlow controller,895 and derived a power model that allows for an estimation of896 the power consumption. However, the model only captures897 the power consumption of network configuration and net-898 work traffic. Faraci et al. [24] defined an analytical model to899 evaluate performance of a green SDN and network function900 virtualization node. However, the model is only applicable to901 this particular node. Celenlioglu et al. [25] proposed an902 energy-aware routing and resource management model903 for Multi-Protocol Label Switching (MPLS) networks using904 SDN based approach, however, this model can only be used905 in pre-established multi-paths (PMPs) strategy. Kaidan906 et al. [26] developed a model to determine the cost of electric-907 ity between any nodes that will enable SDN controller to find908 the most power efficient route. The suggested approach can909 reduce the energy consumption of telecommunication net-910 works. Kaup et al. [27]modeled the power consumption of an911 OpenFlow-based hardware switch (NEC PF 5240) and an912 Open vSwitch, however, the power consumption of control913 planes is not considered in this work. The work in [28] intro-914 duced R-Sync, a powerful time synchronization scheme for915 Industrial Internet of Things (IIoT). Compared to the existing916 time synchronization algorithms, R-Sync synchronizes all917 nodes for better performance in terms of accuracy and power918 consumption. However, it does not model the power con-919 sumption of SDN control planes. All the above analytical920 models lack a comprehensive consideration of both perfor-921 mance and power consumption of control planes. In addition,922 no evaluation framework is established for a fair and fast923 benchmarking of different types of control planes in terms of924 performance and power consumption.

925 6 CONCLUSION

926 In this paper, we propose analytical performance and power927 models for sustainable SDN control planes including STSC,928 MTSC, CLUC, LVFS, GVFS, and HS, and design a generic929 framework for performance and power evaluation of these930 control planes. The proposed modeling framework is vali-931 dated with respect to the processing time and power con-932 sumption of SDN control planes. Extensive simulation results

933show that the proposed solution can precisely model the934power and performance of the concerned SDN control planes,935and the proposed framework can efficiently benchmark dif-936ferent SDN control planes. Of the concerned control planes,937the MTSC controller consumes the most power of up to 217

938units and STSC controller incurs the least processing time of9390.1 ms when the number of data domains is 8. This approach940is in particular useful to identify suitable control planes for941various sustainable SDN network applications. As a part of942future work, this work can be extended to model the power943consumption and precessing time of new control planes.

944ACKNOWLEDGMENTS

945This work was supported in part by the Key Project of946Shanghai Science and Technology Innovation Action Plan947under Grant 18511103302, in part by the Shanghai Munici-948pal NSF under Grant 16ZR1409000, and in part by the China949HGJ Project under Grant 2017ZX01038102-002.

950REFERENCES

951[1] A. Nunes, A. Bruno, M. Mendonca, X. Nguyen, K. Obraczka, and952T. Turletti, “A survey of software-defined networking: Past, present953and future of programmable networks,” IEEE Commun. Surveys954Tuts., vol. 16, no. 3, pp. 1617–1634, Jul.–Sep. 2014.955[2] Q. Yan, R. Yu, Q. Gong, and J. Li, “Software-defined networking956(SDN) and distributed denial of service (DDoS) attacks in cloud957computing environments: A survey, some research issues, and958challenges,” IEEE Commun. Surveys Tuts., vol. 18, no. 1, pp. 602–622,959Jan.–Mar. 2016.960[3] N. Mckeown, T. Anderson, H. Balakrishnan, G. Parulkar,961L. Peterson, J. Rexford, S. Shenker, and J. Turner, “OpenFlow:962Enabling innovation in campus networks,” ACM SIGCOMM963Comput. Commun. Rev., vol. 38, no. 2, pp. 69–74, 2008.964[4] N. Gude, T. Koponen, J. Pettit, B. Pfaff, M. Casado, N. McKeown,965and S. Shenker, “NOX: Towards an operating system for966networks,” ACM SIGCOMM Comput. Commun. Rev., vol. 38, no. 3,967pp. 105–110, 2008.968[5] Z. Cai, A. L. Cox, and T.S.E. Ng. Maestro: A system for scalable969OpenFlow control. Tech. Rep. TR10-08, Rice University, 2010.970[6] P. Lin, J. Bi, and H. Hu, “ASIC: An architecture for scalable971intra-domain control in OpenFlow,” in Proc. 7th Int. Conf. Future972Internet Technol., 2012, pp. 21–26.973[7] T. Koponen,M. Casado, N. Gude, J. Stribling, L. Poutievski, M. Zhu,974R. Ramanathan, Y. Iwata, H. Inoue, and T. Hama, “Onix: A975distributed control platform for large-scale production networks,”976in Proc. 9th USENIX Conf. Operating Syst. Des. Implementation, 2010,977pp. 1–6.978[8] A. Tootoonchian and Y. Ganjali, “HyperFlow: A distributed control979plane for OpenFlow,” in Proc. Internet Netw. Manage. Conf. Res.980Enterprise Netw., 2010, pp. 3–3.981[9] Y. Hassas and Y. Ganjali, “Kandoo: A framework for efficient and982scalable offloading of control applications,” in Proc. 1st Workshop983Hot Topics Softw. Defined Netw., 2012, pp. 19–24.984[10] A. Tootoonchian, S. Gorbunov, Y. Ganjali, M. Casado, and985R. Sherwood, “On controller performance in software-defined986networks,” in Proc. 2nd USENIX Conf. Hot Topics Manage. Internet987Cloud Enterprise Netw. Serv., 2012, pp. 10–10.988[11] M. Vishwasrao and A. Sangaiah, “ESCAPE: Effective scalable989clustering approach for parallel execution of continuous position-990based queries in position monitoring applications,” IEEE Trans.991Sustainable Comput., vol. 2, no. 2, pp. 49–61, Apr–Jun. 2017.992[12] Y. Goto, H. Masuyama, B. Ng, W. Seah, and Y. Takahashi,993“Queueing analysis of software defined network with realistic994OpenFlow-based switch model,” in Proc. IEEE Int. Symp. Model.995Anal. Simul. Comput. Telecommun. Syst., 2016, pp. 301–306.996[13] K. Mahmood, A. Chilwan, O. Østerbø, and M. Jarschel,997“Modelling of OpenFlow-based software-defined networks: The998multiple node case,” Institution Eng. Technol. Netw., vol. 4, no. 5,999pp. 278–284, 2015.1000[14] L. Yao, P.Hong, andW. Zhou, “Evaluating the controller capacity in1001software defined networking,” in Proc. Int. Conf. Comput. Commun.1002Netw., 2014, pp. 1–6.

12 IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL. 4, NO. X, XXXXX 2018

IEEE P

roof

1003 [15] Y. Fu, J. Bi, J. Wu, Z. Chen, K. Wang, and M. Luo, “A dormant1004 multi-controller model for software defined networking,” China1005 Commun., vol. 11, no. 3, pp. 45–55, 2014.1006 [16] G. Wang, J. Li, and X. Chang, “Modeling and performance analysis1007 of the multiple controllers’ approach in software defined1008 networking,” in Proc. Int. Symp. Quality Service, pp. 73–74, 2016.1009 [17] J. Hu, C. Lin, X. Li, and J. Huang, “Scalability of control planes for1010 software defined networks: Modeling and evaluation,” in Proc.1011 Int. Symp. Quality Service, 2014, pp. 147–152.1012 [18] M. Tuysuz, Z. Ankarali, and D. G€oz€upek, “A survey on energy1013 efficiency in software defined networks,” Comput. Netw., vol. 113,1014 pp. 188–204, 2017.1015 [19] C. Newport and W. Zhou, “The (surprising) computational power1016 of the SDN data plane,” in Proc. Int. Conf. Comput. Commun.,1017 pp. 496–504,2015.1018 [20] K. Xie, X. Huang, S. Hao, and M. Ma, “Distributed power saving1019 for large-scale software-defined data center networks,” IEEE1020 Access, vol. 6, pp. 5897–5909, 2018.1021 [21] A. Hammadi and L. Mhamdi, “A survey on architectures and1022 energy efficiency in data center networks,” Comput. Commun.,1023 vol. 40, pp. 1–21, 2014.1024 [22] Y. Carlinet and N. Perrot, “Energy-efficient load balancing in a1025 SDN-based data-center network,” in Proc. Int. Telecommun. Netw.1026 Strategy Planning Symp., 2016, pp. 138–143.1027 [23] F.Kaup, S.Melnikowitsch, andD.Hausheer, “AmodelbasedOpenFlow1028 powermonitoring Framework based on real-worldmeasurements,” PS1029 Technical ReportNo. PS-TR-2014-02, Tech.Rep., 2014.1030 [24] G. Faraci and G. Schembra, “An analytical model to design and1031 manage a green SDN/NFV CPE node,” IEEE Trans. Netw. Service1032 Manage., vol. 12, no. 3, pp. 435–450, Sep. 2015.1033 [25] R. Celenlioglu, B. Goger, and H. Mantar, “An SDN-based1034 energy-aware routing model for intra-domain networks,” in Proc.1035 Int. Conf. Softw. Telecommun. Comput. Netw., 2014, pp. 61–66.1036 [26] M. Kaidan, V. Andrushchak, and M. Pitsyk, “Calculation model of1037 energy efficiency in optical transport networks,” in Proc. Int. Sci.-1038 Practical Conf. Problems Infocommun. Sci. Technol., 2015, pp. 167–170.1039 [27] F. Kaup, S. Melnikowitsch, and D. Hausheer, “Measuring and1040 modeling the power consumption of OpenFlow switches,” in1041 Proc. Int. Conf. Netw. Service Manage. Workshop, 2014, pp. 181–186.1042 [28] T. Qiu, Y. Zhang, D. Qiao, X. Zhang, M. Wymore, and A. Sangaiah,1043 “A robust time synchronization scheme for industrial internet of1044 things,” IEEE Trans. Ind. Inform., vol. 14, no. 8, pp. 3570–3580,1045 Aug. 2018.1046 [29] J. Jackson, “Networks of waiting lines,” Operations Res., vol. 5,1047 pp. 518–521, 1957.1048 [30] R. Bolla, R. Bruschi, F. Davoli, and P. Lago, “Trading off power1049 consumption and delay in packet forwarding engines with1050 adjustable service rate,” in Proc. Int. Teletraffic Congr., 2014, pp. 1–6.1051 [31] K. Leonard, Queueing Systems. Hoboken, NJ, USA: Wiley, 1975.1052 [32] [Online]. Available: https://www.nsnam.org/doxygen/ 2017.1053 [33] M. Jarschel, S. Oechsner, D. Schlosser, R. Pries, G. Sebastian, and1054 P. Tran-Gia, “Modeling and performance evaluation of an Open-1055 Flow architecture,” in Proc. 23rd Int. Teletraffic Congr., 2011, pp. 1–7.1056 [34] A. Bessani, F. Ramos, and P. Ferreira, “On the design of practical1057 fault-tolerant SDN controllers,” in Proc. 3rd Eur. Workshop Softw.1058 Defined Netw., 2014, pp. 73–78.

1059 Xinli Huang received the PhD degree in com-1060 puter science from Shanghai Jiao Tong Univer-1061 sity, in 2007. He is currently an associate1062 professor with the Department of Computer Sci-1063 ence and Technology, East China Normal Uni-1064 versity. His research interests include the areas1065 of Internetworking, software-defined networking,1066 cloud computing, future networks, and network1067 security. He is a member of the IEEE.

1068 Fanshuo Li is currently working toward the mas-1069 ter’s degree in computer science at East China1070 Normal University. His current research interests1071 include the areas of software defined networking1072 (SDN) and queueing theory.

1073Kun Cao is currently working toward the PhD1074degree in computer science at East China Nor-1075mal University. His current research interests1076include the areas of high performance computing,1077multiprocessor systems-on-chip, and cyber phys-1078ical systems.

1079Peijin Cong received the BS degree from the1080Department of Computer Science and Technol-1081ogy, East China Normal University, Shanghai,1082China, in 2016. She is currently working toward1083the master’s degree in the Department of Com-1084puter Science and Technology, East China Normal1085University, Shanghai, China. Her current research1086interests include cloud computing and edge1087computing.

1088Tongquan Wei received the PhD degree in elec-1089trical engineering from Michigan Technological1090University, in 2009. He is currently an associate1091professor with the Department of Computer Sci-1092ence and Technology, East China Normal Uni-1093versity. His research interests include the areas1094of Internet of Things, cloud computing, edge com-1095puting, and design automation of intelligent and1096CPS systems. He has served as a regional editor1097of the Journal of Circuits, Systems, and Com-1098puters since 2012, and has served as a guest edi-1099tor of several IEEE/ACM journals such as the IEEE Transactions on1100Industrial Informatics and the ACM Transactions on Cyber-Physical Sys-1101tems. He is a member of the IEEE.

1102Shiyan Hu received the PhD degree in computer1103engineering from Texas A&M University, in 2008.1104He is the chair and professor of cyber-physical1105systems, University of Essex, United Kingdom. He1106wasan associate professor and director of theCen-1107ter for Cyber-Physical Systems,MichiganTech. He1108was also a visiting professor with IBM Research1109(Austin) in 2010, and a visiting associate professor1110with Stanford University from 2015 to 2016. His1111research interests include cyber-physical systems1112(CPS), CPS security, smart energy CPS, data ana-1113lytics, and computer-aided design of VLSI circuits, where he has published1114more than 100 refereed papers. He is an ACM distinguished speaker, an1115IEEE Systems Council distinguished lecturer, an IEEE Computer Society1116distinguished visitor, a recipient of the 2017 IEEEComputer Society TCSC1117Middle Career Researcher Award, the 2014 National Science Foundation1118(NSF) CAREER Award, and the 2009 ACM SIGDA Richard Newton DAC1119Scholarship. His publications have received a few distinctions, which1120includes the 2018 IEEESystems Journal Best Paper Award, the 2017Key-1121note Paper in the IEEE Transactions on Computer-Aided Design, and the1122Front Cover in the IEEE Transactions on Nanobioscience in March 2014.1123He is the chair for the IEEE Technical Committee on Cyber-Physical Sys-1124tems. He is the editor-in-chief of the IET Cyber-Physical Systems: Theory1125& Applications. He serves as an associate editor of the IEEE Transactions1126on Computer-Aided Design, the IEEE Transactions on Industrial Informat-1127ics, the IEEETransactions onCircuits and Systems, theACMTransactions1128on Design Automation for Electronic Systems, and the ACM Transactions1129on Cyber-Physical Systems. He has served as a guest editor for eight1130IEEE/ACM Journals such as the Proceedings of the IEEE and the IEEE1131Transactions on Computers. He has held chair positions in numerous1132IEEE/ACMconferences. He is a fellow of IET.

1133" For more information on this or any other computing topic,1134please visit our Digital Library at www.computer.org/publications/dlib.

HUANG ET AL.: QUEUEING THEORETIC APPROACH FOR PERFORMANCE-AWARE MODELING OF SUSTAINABLE SDN CONTROL PLANES 13