sdcon: integrated control platform for software-defined clouds · clouds [3]–[8], in which...

1045-9219 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPDS.2018.2855119, IEEETransactions on Parallel and Distributed Systems

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. X, NO. X, X 2017 1

SDCon: Integrated Control Platform forSoftware-Defined Clouds

Jungmin Son, Student Member, IEEE, and Rajkumar Buyya, Fellow, IEEE,

Abstract—Cloud computing has been empowered with the introduction of Software-Defined Networking (SDN) which enableddynamic controllability of cloud network infrastructure. Despite the increasing popularity of studies for joint resource optimization in thecloud environment with SDN technology, the realization is still limited for developing integrated management platform providing asimultaneous controllability of computing and networking infrastructures. In this paper, we propose SDCon, a practical platformdeveloped on OpenStack and OpenDaylight to provide integrated manageability for both resources in cloud infrastructure. The platformcan perform VM placement and migration, network flow scheduling and bandwidth allocation, real-time monitoring of computing andnetworking resources, and measuring power usage of the infrastructure with a single platform. We also propose a network topologyaware VM placement algorithm for heterogeneous resource configuration (TOPO-Het) that consolidates the connected VMs intoclosely connected compute nodes to reduce the overall network traffic. The proposed algorithm is evaluated on SDCon and comparedwith the results from the state-of-the-art baselines. Results of the empirical evaluation with Wikipedia application show that theproposed algorithm can effectively improve the response time while reducing the total network traffic. It also shows the effectiveness ofSDCon to manage both resources simultaneously.

Index Terms—Cloud computing, software-defined networking, resource management, software-defined clouds, IaaS, OpenStack,OpenDaylight.

F

1 INTRODUCTION

The emergence of software-defined networking (SDN) hasbrought many opportunities and challenges to both net-working and computing community. Decoupling of thenetwork control logic from data forwarding plane is a keyinnovation which contributes to the introduction of networkprogrammability. With the global view of the entire net-work, the centralized controller can manage the networkthrough the customized control logic programmed for anindividual use case. SDN can slice the network bandwidthinto multiple layers and provide different quality of ser-vice (QoS) on each slice for different applications. Networkfunction virtualization (NFV) becomes more feasible withthe introduction of SDN, where the virtualized networkfunction can move around the network dynamically withSDN’s network reconfiguration functionality.

SDN also triggers new innovations in cloud computing.The massive scale of a cloud data center where tens ofthousands of compute nodes connected through thousandsof network switches raises network manageability and per-formance issues. Cloud data centers also need to provisioncomputing and network resources dynamically to adapt tothe fluctuating requests from customers. Adoption of SDNin a cloud data center can address the networking issuesraised from the aforementioned characteristic of clouds.Recently, researchers have introduced a new terminology,software-defined clouds (SDC), that integrates SDN and

• J. Son and R. Buyya are with the Cloud Computing and DistributedSystems (CLOUDS) Laboratory, School of Computing and InformationSystems, The University of Melbourne, VIC 3010, Australia.E-mail: [email protected], [email protected].

Manuscript received December 12, 2017.

other SDx concepts in cloud computing where the pro-grammable controller provides dynamic and autonomicconfiguration and management of the underlying physicallayer, similar to the network controller managing the under-lying forwarding plane in SDN [1], [2].

Since both cloud computing and SDN have been widelyaccepted in the industry and open-source community, manyplatforms are introduced and developed in each area. Forcloud management, various full-stack softwares have beendeveloped by companies and open-source communities,such as OpenStack, VMWare, and Xen Cloud Platform. Forinstance, OpenStack is an open source cloud managementplatform that has been employed by many institutions foryears to run their private clouds. It provides manageabilityand provisioning for computing, networking, and storageresources based on the users and projects. OpenStack sup-ports network virtualization on compute nodes for virtualmachines (VM) so that cloud tenants can create their ownvirtual network to communicate between leased VMs.

SDN controllers are also actively developed by opensource communities including OpenDaylight, Floodlight,NOX, Open Network Operating System, and Ryu. Open-Daylight is an open source SDN controller based on amodular structure which has been actively developed underThe Linux Foundation. OpenDaylight provides compre-hensive functionalities through its modular plug-ins, suchas OpenFlow switch management, OpenVSwitch databasemanagement, network virtualization, group-based policy,and so on. Users can select features to install for theirpurpose. Other controllers provide similar functions withdifferent architecture and programming languages.

Although the platforms are matured in SDN and cloudcomputing individually, it is difficult to find a software plat-




form in practice for integration of SDN and clouds to jointlycontrol both networking and computing devices for clouds.For example, OpenStack adopts OpenVSwitch1, a softwarevirtual switch compatible with SDN, in its networking mod-ule to provide network virtualization of VMs. However,in OpenStack, OpenVSwitch is created and managed onlyfor compute nodes to establish virtual tunnels with othercompute nodes. OpenStack does not provide any feature tocontrol the network fabric of the cloud; network switches ina data center cannot be controlled by OpenStack’s internalnetwork controller (Neutron). Similarly, OpenDaylight canmanage the virtual network of OpenStack clouds using itsNetVirt feature, but it controls virtual switches on computenodes separately from network switches that connect com-pute nodes. OpenDaylight and other SDN controllers do notsupport integrated controllability of both compute nodesand network switches. Cisco developed Unified Comput-ing System2, on the other hand, integrates computing andnetworking resource management into a single platform tomanage Cisco’s servers and networking switches. While itprovides integrated controllability for cloud infrastructure,only Cisco’s products can be managed by the platformwhich is in lack of the openness and innovative featuresof SDN.

In this paper, we propose an integrated control plat-form, named SDCon (Software-Defined Clouds Controller)3,which can jointly manage both computing and network-ing resources in the real world in the perspective of acloud provider. SDCon is implemented on top of the pop-ular cloud and SDN controller software: OpenStack andOpenDaylight. It is designed to support various controllerswith the implementation of separate driver modules, butfor simplicity of development, we adopted OpenStack andOpenDaylight as underlying software. We integrate cloud’scomputing fabric and SDN’s network fabric controllabilityinto a combined SDC controller. In addition, we proposeTOPO-Het, a topology-aware resource allocation algorithmin heterogeneous configuration, in order to demonstrate theeffectiveness and capabilities of SDCon platform in empiri-cal systems.

The key contributions of the paper are:

• the design of a practical SDC controller that inte-grates the controllability of network switches withthe management of computing resources;

• the implementation of the integrated SDC controlsystem on a small-scale testbed upon existing soft-ware solutions with 8 compute nodes connectedthrough 2-pod modified fat-tree network topology;

• a topology aware joint resource provisioning al-gorithm for heterogeneous resource configuration(TOPO-Het);

• an evaluation of the joint resource provisioning algo-rithm on the testbed with the proposed SDC controlplatform.

1. http://openvswitch.org2. http://www.cisco.com/c/dam/en/us/solutions/collateral/

data-center-virtualization/unified-computing/at a glancec45-523181.pdf

3. Source code available at: http://github.com/cloudslab/sdcon

The rest of the paper is organized as follows: Sec-tion 2 provides the relevant literature studied and devel-oped for SDN integration platform for cloud computing.In Section 3, we depict the design concept of the proposedplatform and the control flows between modules and ex-ternal components, followed by the detailed explanationof the implementation method and the functionalities ofeach component of SDCon in Section 4. The heterogeneityand network topology aware VM placement algorithm isproposed in Section 5 in addition to the explanation ofbaseline algorithms. Section 6 provides the validation resultsof SDCon with bandwidth measurements, and Section 7shows the evaluation of SDCon and the proposed algorithmwith Wikipedia application by comparing with baselines.Finally, Section 8 concludes the paper along with directionsfor future work.

2 RELATED WORK

Increasing number of studies have investigated the jointprovisioning of networking and computing resources inclouds [3]–[8], in which experiments have been conductedeither in simulation environment [3], [4], or on their in-house customized empirical system [5]–[8]. In this section,we review the recent studies in the integration of SDN andclouds and the joint resource provisioning algorithms.

2.1 Integrated Platform of SDN and CloudsWe previously proposed SDN-enabled cloud computingsimulator, CloudSimSDN, to enable large-scale experimentof SDN functionality in cloud computing [9]. CloudSimSDNcan simulate various use-case scenarios in cloud data cen-ters with the support of SDN approaches, such as dynamicbandwidth allocation, dynamic path routing, and centralview and control of the network. Although simulation toolsare useful to evaluate the impact of new approaches ina large-scale data center, there are still significant gapsbetween the simulation and real implementation.

Mininet [10] has gained a great popularity for SDNemulation to experiment practical OpenFlow controllers.Any SDN controllers supporting OpenFlow protocol can betested with Mininet, where a customized network topologywith multiple hosts can be created and used for several eval-uations, such as measuring bandwidth utilization with theiperf 4 tool. Although Mininet opened up great opportunitiesin SDN studies, a lack of supporting multi-interface emula-tion limited its usage in cloud computing studies. Becauseresearchers need to experiment with virtual machines inhosts, it is impractical to use Mininet to test common cloudscenarios such as VM migration or consolidation in a datacenter.

To overcome the shortcoming of Mininet, OpenStack-Emu is proposed to integrate OpenStack with a large-scalenetwork emulator named CORE (Common Open ResearchEmulator) [11]. The proposed system combines OpenStackplatform with a network emulator to perform evalua-tions considering both cloud and networking. All networkswitches are emulated in one physical machine with thecapability of Linux software bridges and OpenVSwitch.

4. http://iperf.fr

http://openvswitch.org

http://www.cisco.com/c/dam/en/us/solutions/collateral/data-center-virtualization/unified-computing/at_a_glance_c45-523181.pdf



http://github.com/cloudslab/sdcon

http://iperf.fr




A real SDN controller manages the emulated switcheson which transfer network frames from physical computenodes. OpenStackEmu is a useful approach to build anSDN-integrated cloud testbed infrastructure with limitedbudget and resources, but the system does not provide anintegrated control platform for both OpenStack and SDN.In our approach, we propose a joint control platform whichcan even run on OpenStackEmu infrastructure to control itsnetworking and computing resources.

In their recent paper, Cziva et al. proposed S-SCORE,a VM management and orchestration system for live VMmigration to reduce the network cost in a data center [7].This platform is extended from Ryu SDN controller andincludes VM management function by controlling hypervi-sors (Libvirt) in compute nodes. The authors implementedthe system on the canonical tree topology testbed witheight hosts and showed the effectiveness of their migrationalgorithm to reduce the overall VM-to-VM network costby moving VMs into closer hosts. The proposed systemis suitable for studies focusing on networking resources,but is lack of optimizing computing resources such as CPUutilization of compute nodes.

Adami et al. also proposed an SDN orchestrator forcloud data centers based on POX as an SDN controller andXen Cloud Platform as a VM management platform [12]–[15]. The system provides a web portal for end users tosubmit VM requests which are handled by resource selec-tion and composition engine in the core of the system. Theengine utilizes virtual machine and OpenFlow rule handlersto configure VMs in hypervisors and flow tables in switchesrespectively. Similar to S-SCORE, this system focuses moreon networking aspects of clouds without a comprehensiveconsideration of computing resources such as VM and hostsutilization and virtual networks for VMs.

2.2 Joint Resource Provisioning Algorithms

Many studies considering resource heterogeneity in cloudshave focused on the level of brokers for different providersor geographically distributed data centers. For example,VM placement algorithm across different providers wasstudied to reduce the leasing cost of virtual machines whileproviding the same level of throughput from the hostedapplication [16]. Recently, a renewable-aware load balancingalgorithm across geologically distributed data centers wasproposed to increase the sustainability of data centers [17].This algorithm selects a data center operated by more por-tion of renewable power sources (e.g., data center sourcedby a solar power plant in a clear day) and places moreVMs on the data center to reduce the carbon footprint. Also,researchers studied the resource optimization within a datacenter considering heterogeneity to reduce the energy con-sumption of the data center [18]. The on-line deterministicalgorithm consolidates VMs dynamically into the smallernumber of compute nodes and switches off the idle nodesto save electricity.

VM management method considering network topologywas proposed by Cziva et al. [7] that migrates a VM witha high level of network usage onto the destination host toreduce the network cost for VMs and the traffic over thedata center. For a VM, the algorithm calculates a current

Cloud data center

Physical Resources

SDCon

Compute 1

Compute N

Switch 1

Switch M

Monitored data

VM creation / migration

Monitored data

Joint Resource Provisioner

TopologyDiscovery

Status Visualizer

Tenants

SystemAdministrator

Real-time Resource Monitor

Resourcerequest

…

…

SDN Controller

Cloud Management

Platform(VMs+networks)

Fig. 1. Design principle and control flows.

communication cost for each VM-to-VM flow and estimatesa new communication cost if the VM is migrated onto theother end of the flow. When it finds a new location for theVM which can reduce the total communication cost, theVM is migrated to the other host. However, the approachdoes not consider the capacity of compute nodes, thus itmay not migrate a large VM if the available computingresource of the selected host is not enough for the VM.Also, this algorithm limits migration target candidates byconsidering only the compute node that hosts the otherVM of the flow. If no connected VM is placed in a host,it is not considered as a migration target, which in practicecan be a proper destination if all the candidates in the firstplace cannot host the VM because of the resource limitation.Our algorithm deals with this limitation by considering thegroup of hosts, instead of an individual host, as a candidateof VM placement.

Although the heterogeneity in the cloud environmenthas been addressed in these studies, they are either consid-ering only high-level entities (e.g., different data centers orproviders) or only computing resources within a data centerwithout considering joint optimization for both computingand networking resources in heterogeneous infrastructurein a data center.

3 SDCON: SOFTWARE-DEFINED CLOUDS CON-TROLLER

SDCon is designed to provide integrated controllability forboth clouds and networks and implemented with popularsoftware widely used in practice. The conceptual designand the control flows between components are shown inFig. 1. As described in the previous section, cloud man-agement platforms and SDN controller software have beendeveloped and widely adopted for years. Therefore, inthis study, we designed our platform on top of thosemature software platforms. Cloud Management Platform,e.g., OpenStack, manages computing resources includingCPU, memory, and storage which are provided by multipleunderlying compute nodes. Monitored data, such as CPUutilization of VMs and the compute node, is sent back to




Cloud Management Platform from each compute node, sothat the information can be used for resource provisioningand optimization. Similarly, networking resources are con-trolled by SDN Controller, e.g., OpenDaylight. Switches areconnected and managed by the SDN Controller, and thenetwork utilization monitored in each switch is gatheredby the controller. Both Cloud Management Platform andSDN Controller are deployed and properly configured withexisting software solutions.

Based on the separate platform for cloud managementand SDN control, SDCon is designed to manage and mon-itor both of them jointly. When tenants request resourcesfrom a cloud data center, Joint Resource Provisioner inSDCon accepts the request and controls both Cloud Man-agement Platform and SDN Controller simultaneously toprovide the required amount of resources. In the tradi-tional cloud platform, tenants can specify only computingresources in detail, such as the number of CPU cores, theamount of memory and storage. With SDCon, we introduceadditional configuration in the resource request regardingnetworking requirements, such as requested bandwidth be-tween VMs, which can be specified along with the typeof VMs. Joint Resource Provisioner is in charge of notonly allocating the requested resources for tenants, but alsooptimizing both computing and networking resources basedon the optimization policy provided by the system adminis-trator of the cloud provider. For example, it can migrate low-utilized VMs and flows into the smaller number of hardwareand power off the unused resources to increase powerefficiency of the cloud data center. System administrators inthe data center can also set a customized default path policyfor the network, especially effective for multi-path networktopology.

Resource provisioning and optimization at Joint Re-source Provisioner is performed based on the networktopology and real-time monitored data acquired from CloudManagement Platform and SDN Controller. Topology Dis-covery receives the network topology and its capacity fromSDN Controller and the hardware specification of computenodes from Cloud Management Platform. Real-time Re-source Monitor also gathers the monitored data from bothcloud and SDN controller. Contrary to Topology Discovery,Resource Monitor keeps pulling measurements from com-pute nodes and switches periodically to update real-timeresource utilization, including CPU utilization of VMs andcompute nodes, and bandwidth utilization of network links.Please note that the pulling interval should be properlyconfigured by the administrator considering the size ofmonitored data and the number of monitoring computenodes and switches to prevent the system overload.

In addition to the resource provisioning, SDCon sup-ports a visualization of computing and networking re-sources through Status Visualizer components. Based onthe topology information and monitored measurements,the visualizer displays current utilization of network linksand compute nodes in real-time. System administrators cancheck the status of all resources on a graphical interface. Inthe following subsection, we explain the implementation ofthe proposed system in detail.

OpenFlow switchComputenode OpenFlow switchComputenode

OpenStack Controller

ComputeNode

Hypervisor(LibVirt)

VM

(OpenVSwitch)

OpenFlowSwitch

NeutronServer

NovaAPI/Conductor

NovaCompute

NeutronOVSAgent

OpenVSwitch

VMVM

FlowFlowFlow

OpenDayLight

OpenFlow/ OVSDB

OpenFlowPlugin

Oslomessages

Monitor(Ceilometer) L2Switch

SDCon: Software-Defined Cloud Controller

OpenDayLight APIOpenStackAPI

CloudMonitor NetworkMonitor

CloudManager NetworkManager

TopologyDiscovery

JointResourceProvisioner

sFlow Engine

Collector

sFlow

RestAPI

Visualization

Physicalresources

Individualcontrollers

SDCcontroller

QoSQueues

ForwardingTable

sFlowAgent

sFlow‐RTStatCollector(Gnocchi)

Fig. 2. System architecture of SDCon and underlying software compo-nents.

4 SYSTEM IMPLEMENTATION

We implemented SDCon by utilizing OpenStack cloud man-agement platform and OpenDaylight SDN controller tocontrol computing and networking resources respectively.Fig. 2 shows the implemented architecture of SDCon plat-form. The system is implemented with Python, on top ofOpenStack Python SDK5, Gnocchi Python API6, OpenDay-light OpenFlow plugin7 and OVSDB plugin8, and sFlow-RT REST API9. SDCon utilizes these existing open-sourcesoftware solutions to provide integrated controllability andoptimization to manage both computing and networkingresources. Please note that another software can be adoptedin SDCon by modifying the relevant API.

OpenStack platform is exploited to manage computenodes. Compute nodes are running OpenStack Nova-compute component to provide a bridge between the hy-pervisor and the OpenStack controller. VM creation, mi-gration, or deletion is managed by Nova-API and Nova-Conductor components in the controller through messagespassed to Nova-compute component in each compute node.Aside from Nova components, OpenStack Neutron is incharge of managing virtual networks for VMs. Neutron-OVS-Agent running on each compute node is creating anddeleting virtual tunnels between compute nodes by updat-ing OpenVSwitch rules, instructed by Neutron-Server in thecontroller. OpenStack Telemetry services are running on thecontroller to monitor compute nodes and VMs. OpenStackCeilometer gathers computing resource measurements, such

5. http://docs.openstack.org/openstacksdk/latest/6. http://gnocchi.xyz/gnocchiclient/api.html7. http://wiki.opendaylight.org/view/OpenDaylight OpenFlow

Plugin:End to End Flows8. http://docs.opendaylight.org/en/stable-carbon/user-guide/

ovsdb-user-guide.html9. http://www.sflow-rt.com/reference.php

http://docs.openstack.org/openstacksdk/latest/

http://gnocchi.xyz/gnocchiclient/api.html

http://wiki.opendaylight.org/view/OpenDaylight_OpenFlow_Plugin:End_to_End_Flows

http://wiki.opendaylight.org/view/OpenDaylight_OpenFlow_Plugin:End_to_End_Flows

http://docs.opendaylight.org/en/stable-carbon/user-guide/ovsdb-user-guide.html

http://docs.opendaylight.org/en/stable-carbon/user-guide/ovsdb-user-guide.html

http://www.sflow-rt.com/reference.php




as CPU utilization and RAM usage, of both VMs and com-pute nodes. The monitored measurements are stored withGnocchi component, which can be retrieved by its Rest API.

For networking resources, OpenDaylight controls Open-Flow switches that connect compute nodes. Each switch hasOpenFlow forwarding tables managed by OpenDaylight L2-Switch and OpenFlow-Plugin modules which can installor modify forwarding rules for specific flows. OpenDay-light also manages QoS queues in a switch to enable dy-namic bandwidth allocation. Due to the OpenFlow’s limitedsupport for QoS queues, queue management is providedthrough OpenDaylight’s OpenVSwitch plug-in which cancreate and delete Hierarchical Token Bucket (HTB)10 queuesdirectly in switches. For network monitoring, our platformemployed the sFlow protocol which can provide real-timemeasurements of the network by capturing sampled pack-ets. sFlow agents are set up in switches and send sampleddata to the sFlow engine running on the controller whichgathers the data and provide the measurements through theRest API.

On top of these services, SDCon is implemented tojointly manage both computing and networking resources.SDCon platform aims at integrating an SDN controller witha cloud manager to perform:

• allocating resources for VMs and flows with theinformation of applications and resources;

• monitoring both compute nodes and networkswitches for their utilization, availability, and capac-ity with the knowledge of network topology;

• dynamic provisioning of both computing and net-working resources based on the real-time monitoredmatrices;

• providing management toolkit for system adminis-trators to monitor power consumption and visualizethe operational status of a data center.

Below, we explain the detail of each component in SD-Con, data flows between components, and sequence dia-gram of VM allocation with SDCon.

4.1 Cloud ManagerCloud Manager is in charge of controlling OpenStack Con-troller through OpenStack SDK. It can create or migrate aVM on a specific compute node by specifying the targetnode. Current allocation status of compute nodes is alsoretrieved through OpenStack so that SDCon can identify thenumber of used and available cores and memory in eachnode. This information is passed to Topology Discovery inorder to keep the up-to-date resource allocation informationin the topology database. It also provides other functionsrelated to VMs and compute nodes, such as getting the IPaddress of a VM, VM types (Flavor), VM disk images, andvirtual networks for VMs. Note that the virtual networkconfigured in a compute node is managed by OpenStackNeutron through OpenVSwitch installed on compute nodes,whereas switches are controlled by OpenDaylight. Thus, theIP address of a VM is retrieved by Cloud Manager, not byNetwork Manager.

10. http://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.qdisc.classful.html

4.2 Cloud Monitor

Cloud Monitor is to retrieve real-time measurements ofcomputing resources monitored at compute nodes and VMs.We use OpenStack Ceilometer and Gnocchi components tomeasure and collect the monitored data in the implementa-tion. Ceilometer installed on each compute node measuresCPU and memory utilization of all VMs and the computenode itself, and sends to Gnocchi that collects and aggre-gates the data from every compute node. By default, Gnoc-chi polls and stores the resource utilization measurement ev-ery 60 seconds from all VMs and compute nodes, which canbe customized by changing Gnocchi configuration settings.Our Cloud Monitor uses Gnocchi Python API to retrievethe collected data, such as CPU utilization of both VMs andcompute nodes. It is also utilized by Power ConsumptionEstimator to calculate the estimated power consumption ofcompute nodes, in which the detail is explained later.

4.3 Network Manager

In SDCon, OpenDaylight SDN Controller is managed byNetwork Manager module to push OpenFlow forwardingtables, default path setting for multi-path load balancing,and QoS configurations. In SDN, we can add a specific flowrule based on source and destination IP/MAC addresses,TCP/UDP port numbers, protocol types, and other criteriasupported by OpenFlow standard. SDCon uses this featureto install or remove a flow rule onto switches throughOpenDaylight OpenFlow plugin. This feature can be usedfor dynamic flow control or flow consolidation to change thepath of a flow dynamically for performance improvementor power efficiency. We also implement a default path rout-ing module enabling the load-balancing routing in fat-treetopology as suggested in the original paper [19]. Because thedefault L2 switch module in OpenDaylight only supportsSTP for multi-path topology, we cannot use the benefit ofmultiple paths with the default module. Thus, we needed toimplement our own routing module to set the default pathbetween compute nodes based on the source address of thenode. Our default path module installs flow rules on eachswitch to forward the packet to different ports by looking atsource address if there are multiple paths. With the defaultpath module, packets are forwarded to the pre-configuredpath based on the source node without flooding them to theentire network or dumping all traffic to the SDN controller.

SDCon’s Network Manager is also capable of config-uring QoS queues on each switch to provide bandwidthallocation for a specific flow if a resource request includesspecific bandwidth requirements. For a priority flow whichneeds a higher priority over other traffics, a tenant canrequest a certain amount of bandwidth, then SDCon canallocate it by configuring minimum and maximum band-width in QoS queues on an output port in a switch. Forexample, OpenVSwitch supports traffic shaping function-ality, similar to the tc tool in Linux, which can controlthe network bandwidth for different QoS level using HTB.We implement this QoS management feature in NetworkManager. For VM-to-VM bandwidth allocation, NetworkManager installs QoS queues on each port in every switchalong the path passing the traffic. Network Manager atfirst aggregates all the requests for different source and

http://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.qdisc.classful.html

http://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.qdisc.classful.html




destination VMs, because the flows have to be mapped tooutput ports simultaneously if the link is shared by multipleflows. Once the number of flows to be configured at eachport in each switch is calculated, Network Manager sendsQoS and queue creation request to OpenDaylight OVSDBplugin which controls OpenVSwitch entries on each switch.Then, we push OpenFlow rules for the specific flow in needof the bandwidth allocation, so that packets for the specificflow can be enqueued at the created QoS queue. We showthe effectiveness of our bandwidth allocation method inSection 7.

4.4 Network Monitor

Similar to Cloud Monitor, Network Monitor pulls a real-time network status from switches and compute nodes. Weleverage sFlow protocol that collects sampled packets fromsFlow agents in switches and sends them to the collectorin the controller. By default, the sampling rate of sFlow is1 in 500 for 100Mbps link and 1 in 1000 for 1Gbps link11,which sends a packet for every 500 packets passing the100Mbps link. For data collection and aggregation, sFlow-RT is installed in the controller node aside from the SDNcontroller. SDCon Network Monitor utilizes REST APIsprovided by sFlow-RT to retrieve aggregated measurementsof bandwidth usage and traffic flows on each link. AlthoughOpenDaylight provides statistics of every port on OpenFlowswitches, we decide to use sFlow because it provides moredetailed information such as the source and destinationaddress of the flow and encapsulated packets for tunnelingprotocols. Please note that the collected monitoring datafrom both Cloud and Network Monitor is stored in themonitor modules which can be accessed by SDCon at anytime. Thus, the up-to-date information can be simultane-ously provided from both monitors and utilized by SDCon.

4.5 Topology Discovery

Network topology information is retrieved from OpenDay-light through Network Manager. With the support of hosttracker in OpenDaylight, SDCon can acknowledge computenodes and switch devices on the network along with portsand links. Topology Discovery module internally generatesa graph structure to store the topology information. In addi-tion, it also retrieves the hardware configuration of computenodes through Cloud Manager. With the retrieved informa-tion about computing resources, e.g., the number of CPUcores, size of RAM, etc., Topology Discovery can provide thedetailed information to other modules. Unlike Cloud andNetwork Monitor modules which needs a periodic updatefor real-time data, Topology Discovery is created when theplatform starts and updated only when there is an updateon the topology.

4.6 Joint Resource Provisioner

The main control logic of SDCon resides in Joint ResourceProvisioner module. When VM and flow requests are sub-mitted from cloud tenants, Joint Resource Provisioner mod-ule decides which compute nodes to place the VM and

11. http://blog.sflow.com/2009/06/sampling-rates.html

which network path to transfer the VM flows. With theretrieved information of the topology information throughTopology Discovery and real-time utilization via Cloud andNetwork Monitor, the decision is made with the provi-sioning algorithm provided by the system administrator.A customized VM allocation and migration policy can beimplemented for VM provisioning, as well as a networkflow scheduling algorithm for network provisioning.

Once the decision is made at the control logic, JointResource Provisioner exploits Cloud Manager and NetworkManager, to create a VM on the selected node and push newflow entries respectively. For bandwidth allocation, QoSrules and queues are created in switches by updating Open-VSwitch configuration through Network Manager. JointResource Provisioner can utilize the real-time monitoreddata at any time whenever necessary. For example, a VMmigration policy can be implemented upon Joint ResourceProvisioner which constantly monitors the current status ofVMs, compute nodes, and network bandwidth usage, thendynamically migrates a busy VM to less utilized resourcesin real time. If the migration policy is implemented, itcan schedule the migration process by sending migrationcommand to OpenStack through Cloud Manager.

4.7 Power Consumption EstimatorFor supporting energy-related experiments, we additionallyimplement Power Consumption Estimator that calculatesthe power consumption of the infrastructure based on theutilization of compute nodes and switch links using anenergy model. Although the best practice to measure thepower consumption is to use a commercialized monitoringdevice, we add this feature to provide a simple alternativeapproximation method at no additional cost. Accumulatedutilization data from both monitors is input to the energymodel, and the estimated power consumption is calculatedand output based on the energy model provided to SDCon.

For compute nodes, we use the linear power modelderived from Pelly et al. [20] for its simplicity and ease ofimplementation. Please note that the power model can bereplaced with other linear or quadratic models for moreaccurate estimates by changing the source code. In ourimplementation, the power consumption of the host hi withthe utilization ui is calculated by:

P (hi) = Pidle(hi) + (Ppeak(hi)− Pidle(hi)) · ui

where Pidle refers to the power consumption of the hostin idle state (0% utilization), and Ppeak refers to the peakpower consumption at 100% utilization. The power modelof switches is derived from Wang et al. [21] which can becalculated by:

P (si) = Pstatic(si) + Pport(si) · qiwhere qi refers to the number of active ports of the switch,Pstatic being the static power consumption for no networktraffic, and Pport being per port power consumption of theswitch.

4.8 Status VisualizerStatus Visualizer is implemented to intuitively visualize thecurrent state of the data center. This module retrieves the

http://blog.sflow.com/2009/06/sampling-rates.html




SDCon

Physical Resources

Cloud Management Platform APIs

ApplicationDeployment

VM / Flow requests

JointResource

Provisioner

VMcreation /migration

Topology Information

MonitoredMeasurements

Networktopology

Power Usage

Network MonitorCloud MonitorCloud Manager

Power Consumption

Estimator

StatusVisualizer

Network Manager

Forwardingtable /QoS

queues

Compute nodeinformation

NetworkBandwidth

usage

SystemAdministrator

Web GUI

CPUutilization

CalculatedPower

SDN Controller APIs

Control commandMonitoring dataTopology data

Fig. 3. Data flow diagram of SDCon components.

topology information to draw the physical infrastructurewith network links, and then draws the real-time monitoredflows with different colors. We use D3 JavaScript library12

to visualize the data onto web interface. By refreshing thepage periodically, the Visualizer continuously updates themonitored measurements in real time.

4.9 Data Flow and Sequence Diagram

Fig. 3 depicts the overall data flow between components inSDCon. Once application deployment request is submittedto Joint Resource Provisioner, it calls Cloud Manager andNetwork Manager to request VM creation and flow place-ment for computing and networking resources respectively.In order to decide where and how to provision the resources,it retrieves the topology information prepared by TopologyDiscovery, as well as the monitoring data gathered fromCloud Monitor and Network Monitor. In addition, PowerConsumption Estimator calculates the power usage basedon the monitored measurements and topology information.Measured information is visualized with Status Visualizerthrough a web interface.

To provide an in-depth understanding of the workingprocess of SDCon, we also depict sequence diagram ofVMs and flows deployment with SDcon (see Fig. 4). Thediagram shows the process to get VM and host informationfrom OpenStack API and network topology informationfrom OpenDaylight API. For VM deployment, it first runsthe VM placement algorithm to select the compute nodeand create a VM in the selected compute node throughCloudManager and OpenStack. For network deployment,we depict the procedure of updating a flow path based onthe real-time bandwidth usage of each link as an example.

12. http://d3js.org

IP address of source and destination VMs are retrieved fromCloud Manager and passed to Network Manager to startthe dynamic flow scheduling for the specified pair of IPaddresses. Network Manager periodically selects a series ofswitches along the path with the lowest bandwidth utiliza-tion by obtaining monitored data. Then, new forwardingrules with the updated path are pushed into the switches ofthe selected path.

5 JOINT RESOURCE PROVISIONING IN HETERO-GENEOUS CLOUDS

In this section, we propose a joint computing and net-working optimization algorithm for heterogeneous resourceconfiguration which can be used with SDCon in empiricalenvironment. Many approaches have been proposed forjoint optimization in clouds, but most of them consider ahomogeneous configuration where the hardware of physicalhosts and switches are identical across the data center [4],[7], [8]. This is an effective assumption in large-scale publiccloud considering the identical hardware within the samerack which is procured at the same time. However, as thedata center grows in need of upgrading or purchasing newhardware, the configuration of new machines can becomedifferent from old machines. Also, it is often impractical topurchase many machines at the same time in a small scaleprivate cloud, which can lead to having different hardwareconfiguration even in the same rack.

Resource optimization in heterogeneous configurationneeds a different approach from the one for homogeneousresources because of the varied per-unit performance andpower consumption. For example, power optimization ofa homogeneous cloud by consolidation considers only thenumber of hosts, whereas in heterogeneous it needs toconsider per-unit power consumption level of each computenode. When VMs can be consolidated into the smallernumber of physical hosts, unused hosts can be powered offto save energy of the data center. In a homogeneous model,reducing the number of turned-on hosts obviously leadsto the reduction of energy consumption of the entire datacenter, whereas in heterogeneous configuration, we haveto consider the different capacity and power consumptionlevel of each physical machine. If VMs are consolidated intoa less power-efficient host consuming more energy thantwo or more hosts, consolidation can actually increase thetotal power consumption of a data center. Thus, the powerconsumption level of different host types must be taken intoan account for power optimization in heterogeneous clouds.

5.1 Topology-aware VM Placement Algorithm for Het-erogeneous Cloud Infrastructure (TOPO-Het)

VM placement in heterogeneous resources can be consid-ered as a variable-sized bin packing problem, which isan NP-hard problem to find the optimal solution [22]. Asit is impractical to find an optimal solution in real-time,we present a heuristic algorithm in order to reduce theproblem complexity suitable for on-line VM placement.Our algorithm aims at network-aware VM allocation forheterogeneous cloud infrastructure for improved networkperformance of VMs.

http://d3js.org




JointResourceProvisioner CloudManager NetworkManagerTopologyDiscovery ResourceMonitorTenant

submitVmFlowRequest()getExistingVms()

getTopology()

ext:OpenDaylightAPIext:OpenStackAPI

getNetworkTopoInfo()

compute.find_server()

requests.get()

compute.find_hypervisor()

getVmPlacementMap()

getNodeCPUUtilization()getLinkBandwidth()

compute.find_image()compute.find_flavor()

network.find_network()compute.create_server()

requests.put()

findLowUtilizationPath()

getComputeNodeInfo()

getLinkBandwidth()

createVm()

startDynamicFlowScheduling()

compute.find_server()getVmIpAddress()

deployVm()

deployNetwork()

loop

Fig. 4. Sequence diagram to deploy VMs and flows with SDCon.

For compute nodes with different computing capacity(e.g., number of cores and size of memory), there is higherpossibility that the compute node with larger capacity runsmore numbers of VMs than the one with smaller capacity.Assuming that all compute nodes have the same networkcapacity, those VMs in the larger compute node will utilizeless bandwidth when the VMs use the network simultane-ously, because the same amount of network bandwidth isshared by more numbers of VMs in the larger node. On theother hand, VMs placed in the same node can communicateto each other via in-memory transfer rather than throughnetwork interface, which can provide far more inter-VMbandwidth within a node. Thus, we need to consider theconnectivity of VMs for placing VMs into the smaller num-ber of compute nodes. If the connected VMs are placedinto the same node, it not only provides more bandwidthfor VMs but also reduces the amount of traffic in networkinfrastructure. However, if unrelated VMs are placed in thesame node, it consumes more networking resources andreduces per-VM bandwidth because the limited bandwidthof a network interface should be shared by more number ofVMs.

Algorithm 1 is proposed to address the contradictionof VM placement in the same or closer compute nodes.The algorithm considers the connectivity between VMs andfinds the nearest compute node if the other VM is alreadyplaced, or the group of closely-connected compute nodeswhich can collectively provide the requested resources forVMs. In Line 5, VMs are grouped based on the networkconnectivity information gathered from the additional net-working requests of SDCon input data in JSON format. Ifthere is a connection request between two VMs, they areput into the same VM group. In Line 7, VMs in the samegroup are sorted in the order of required resources fromhigh to low to make sure a large VM is placed before smallerVMs. Please note that sort function in the Algorithm 1 is

a general sort function, such as quick sort, which has thecomplexity of O(n · log n). HG on Line 8 refers a list ofthe set of compute nodes (hosts) which collectively providesufficient resources to the group of VMs. The algorithmoverall constructs HG with the available host groups inthe order of closely connected host groups in the networktopology.

Line 9 finds if any VMs are already deployed in the in-frastructure. If there are VMs already placed, it tries to placenew VMs to nearby compute nodes close to the placed VMs,e.g., the same node (Line 11), the connected nodes under thesame edge switch (Line 12), or the nodes within the samepod (Line 13). In Line 16-22, the algorithm tries to find therest of host groups (a host, a host group under the sameedge network, or a host group in a same pod) regardless ofthe placed VMs, if they can provide the requested resourcesto construct a complete HG that assure the VMs are placedin any hosts regardless of the VM connectivity. These hostgroups are sorted by the available bandwidth calculatedfrom the number of VMs running on the entire host groupwith the information of network topology. If there are moreVMs running on the host group, it is more likely congestedeven if the total computing capacity of the group is higher.Note that the heterogeneity of resources is considered at thisstep when the VMs are assigned to the less congested hostgroup regarding the difference in the capacity of each hostgroup.

Once the algorithm found all the candidate host groupsfor the VM group, each VM in the group is tried to placein a host within the higher priority host group on Line 23-34. Note that the host list within a group is sorted by thenumber of free CPU cores before placing VM Line 25, inorder to consolidate VMs into a smaller number of hosts.If the host is not fit for the VM, the next candidate hostis tried until the available one is found. If all the candidatehost groups are failed, the algorithm places the VM onto any




Algorithm 1 TOPO-Het: Topology-aware collective VMplacement algorithm in heterogeneous cloud data center.

1: Data: VM : List of VMs to be placed.2: Data: F : List of network flows between VMs.3: Data: H : List of all hosts in data center.4: Data: topo: Network topology of data center.5: VMG← Group VM based on the connection in F .6: for each VM group vmg in VMG do7: sort(vmg, key=vm.size, order=desc)8: HG← empty list for available host groups;9: VMconn ← topo.getConnectedVMs(vmg);

10: if VMconn is not empty then11: Hhost ← topo.findHostRunningVMs(VMconn);12: Hedge ← topo.getHostGroupSameEdge(Hhost);13: Hpod ← topo.getHostGroupSamePod(Hhost);14: HG.append(Hhost, Hedge, Hpod);15: end if16: HGhost ← topo.findAvailableHost(vmg);17: sort(HGhost, key=hg.capacity, order=desc);18: HGedge ← topo.findAvailableHostGroupEdge(vmg);19: sort(HGedge, key=hg.capacity, order=desc);20: HGpod ← topo.findAvailableHostGroupPod(vmg);21: sort(HGpod, key=hg.capacity, order=desc);22: HG.appendAll(HGhost, HGedge, HGpod);23: for each vm in vmg do24: for each host group hg in HG do25: sort(hg, key=h.freeResource, order=asc);26: isPlaced← place(vm, hg);27: if isPlaced then28: break;29: end if30: end for31: if not isPlaced then32: place(vm, H);33: end if34: end for35: end for

available host in the data center regardless of the networktopology.

The time complexity of the proposed algorithm dependson the number of hosts |H|, VMs to be placed |VM |, andthe VM groups |VMG| which can be at most |VM | if eachgroup has only one VM. Sorting the VMs on Line 7 takesat most O(|VM | · log |VM |, and for each VM group sortingthe host group takes at most O(|H| log |H|) in the case of|HG| = |H|. As topo uses a tree data structure, all findingfunctions using topo structure takes log |H| time. Once thecandidate host groups are constructed, placing each VM inone of the candidate hosts takes O(|H| · log |H|) as it needsto sort the candidate hosts, which needs to be done for allVMs. Therefore, the total time complexity of TOPO-Het is:

O(|VM | log |VM |) +O(|VMG| · |H| log |H|)+O(|VM | · |H| log |H|)

= O(|VM | · |H| log |H|+ |VM | log |VM |)

which is feasible time for online VM placement.

5.2 Baseline Algorithms

Our heterogeneity-aware resource allocation algorithm iscompared with First-Fit Decreasing (FFD) algorithm in con-junction with Bandwidth Allocation (BWA) and DynamicFlow Scheduling (DF) network management schemes. FFDis a well-known heuristic algorithm for a bin-packing prob-lem which allocates the largest VM to the most full computenode capable of the VM adopted in many studies [23],[24]. VMs are sorted in descending order of the size ofthe required resource, and compute nodes are sorted inascending order of available resources. As it chooses themost full node first, the algorithm consolidates VMs into asmaller number of compute nodes which can benefit to theenergy efficiency and network traffic consolidation.

Further, we implement two network management meth-ods, Bandwidth Allocation (BWA) and Dynamic FlowScheduling (DF), in combination with FFD to empower FFDbaseline algorithm and show the effectiveness of SDCon. InBWA, the requested network bandwidth is allocated for traf-fic flows between VMs using the aforementioned method inSection 4.3. SDCon renders flow settings provided with VMrequest and creates QoS queues along switches forwardingthe VM traffic. DF (Dynamic Flow Scheduling), on the otherhand, updates the network path of the VM-to-VM flowperiodically by monitoring real-time traffic, which can beseen in many approaches [25]–[27]. It first finds the shortestpath between VMs and retrieves monitored traffic of thecandidate path. After collecting bandwidth usage statisticsof every link on each path, it selects the less congested pathand updates forwarding tables on switches along the se-lected path through SDN controller. In our implementation,DF checks and updates each network path every 60 secondsusing the monitored data, which can be adjusted in SDConfor delay sensitive applications.

We combine the network management methods withFFD VM allocation algorithm. In the following sections, werefer BWA and DF as a combination of FFD VM allocationmethod with the referred network management scheme.The key difference between the baselines and TOPO-Het isthat TOPO-Het considers network topology and distancebetween compute nodes in a data center for allocating VMsand network flows. TOPO-Het also considers the hetero-geneous configuration of compute nodes in the decisionprocess. It is also worth to note that both TOPO-Het andFFD utilize the topology information obtained from Topol-ogy Discovery module in SDCon, and DF utilizes the real-time monitored bandwidth utilization data from NetworkMonitor module. BWA installs flow rules in each OpenFlowswitch through SDCon’s Network Manager.

6 SYSTEM VALIDATION

In this section, we describe a testbed setup and the experi-ment for validating the system by testing bandwidth alloca-tion functionality of SDCon. Additional validations, suchas VM placement, dynamic flow scheduling, and powerusage estimation, are also undertaken in conjunction withthe performance evaluation of the topology-aware resourceprovisioning algorithm and network management schemesin Section 7.




Pod 2Pod 1

Compute1 Compute2 Compute3 Compute4

Core1

Compute5 Compute6 Compute8

Core2

ManagementNetwork

Controller

Compute7

Edge1 Edge2

Aggregate1 Aggregate2

Edge3

Aggregate3 Aggregate4

Edge4

Fig. 5. Testbed configuration.

6.1 Testbed Configuration

In order to evaluate the proposed system and algorithmon the empirical environment, we deployed SDCon on ourtestbed equipped with 8 compute nodes, a controller node,and 10 OpenFlow switches. Fig. 5 shows the architecturalconfiguration of our testbed. The network topology is builtas a modified 2-pod fat-tree architecture which has lessnumber of pods from the original proposal [19]. With thecost and physical limitation, we built two pods connectedwith two core switches instead of four. This modified topol-ogy still provides multiple paths between two hosts with thefull support of 1:1 over-subscription ratio within the samepod as proposed in the original fat-tree topology.

Core, aggregate, and edge switches are implementedwith 10 Raspberry-Pi hardware with external 100Mbps USB2.0 Ethernet connectors on which cables are connected to theother Raspberry-Pi or compute nodes. A similar approachwas proposed by researchers from Glasgow University [28]that Raspberry-Pi was used for compute nodes in clouds.In our testbed, we use Raspberry-Pi to build SDN switches,not compute nodes, as we have enough servers for computenodes whereas we cannot procure OpenFlow switches.

In each Raspberry-Pi switch, OpenVSwitch is runningas a forwarding plane on a Linux-based operating system.A virtual OpenVSwitch bridge is created in each RaspberryPi to include all Ethernet interfaces as ports and connectedto OpenDaylight SDN controller running on the controllernode. All links between Raspberry-Pi switches and computenodes are set to 100Mbps due to the limited bandwidthof the external USB Ethernet adapters of Raspberry-Pi.Please note that packet processing in OpenVSwitch uses runto completion model without buffering13. We configureda separate management network for controlling and APIcommunications between switches, compute nodes, and thecontroller.

Each compute nodes are running OpenStack Nova-Compute component to provide computing resources toVMs as well as OpenStack Neutron-OVS Agent for virtualnetwork configuration for the hosted VMs. OpenStack com-ponents in compute nodes are connected to the OpenStackserver running on the controller node through a separate

13. http://docs.openvswitch.org/en/latest/faq/design/

TABLE 1Hardware specification of controller and compute nodes.

Nodes CPU model Cores(VCPUs) RAM Quantity

Controller,Compute 1,2

Intel(R) E5-2620@ 2.00GHz

12 (24) 64GB 3

Compute 3-6 Intel(R) X3460 @2.80GHz

4 (8) 16GB 4

Compute 7,8 Intel(R) i7-2600@ 3.40GHz

4 (8) 8GB 2

management network. Due to our limited device avail-ability, three different types of computers are used for thecontroller and compute nodes. Table 1 shows the differenthardware configuration of each computer.

In OpenStack configuration, we create a flat network forVM communications instead of using tunneling methods,due to the limitation of flow matching rules in OpenFlow.Tunneled addresses are invisible at OpenFlow switches un-less decapsulating packets in each switch which can result insevere overheads at switches. Instead, IP addresses of VMsare assigned with the same range of the compute nodes inthe flat network, thus switches can differentiate flows basedon the pair of source and destination VM addresses withoutdecapsulation and re-encapsulation at every switch.

6.2 Bandwidth Allocation with QoS SettingsAs described in Section 4, we implement network band-width allocation by applying OpenVSwitch’s QoS andQueue configuration. This experiment is to see the effective-ness of the implemented bandwidth management methodin SDCon. We use iperf tool to measure the bandwidthof a flow between VMs sharing the same network path.In order to make the network bandwidth to be sharedby different flows, we run iperf on two or three differentVMs simultaneously. Also, various combinations of TCPand UDP protocols are used in experiments to compare theimpact of our bandwidth allocation mechanism on differentprotocols. Note that the maximum bandwidth of our testbedis measured at approximately 95 Mbps (bits/s), due to theEthernet port limitation of our Raspberry-Pi switches. Weset up QoS configuration with HTB policy and specify theminimum bandwidth of 70 Mbps for QoS flows and 10Mbps for other flows, and maximum of 95 Mbps for allflows.

Fig. 6 shows the measured bandwidth of different flowssharing the same network path. When iperf was used inTCP mode (Fig. 6a), the bandwidth is equally shared bytwo flows without QoS configuration. After applying QoSconfiguration for Flow 1, the bandwidth for Flow 1 isincreased to 73.6 Mbps, whereas Flow 2 can acquire only14.9 Mbps. Fig. 6b shows the measured bandwidth fortwo UDP flows. In UDP mode, we run iperf at a constantrate of 70 Mbps which causes a heavy congestion whentwo or more flows are shared the same path. Because ofthe heavy congestion, the shared bandwidth between twoflows, in this case, decreased to about 30 Mbps for each flowwithout QoS setting. However, similar to the TCP result,Flow 1’s bandwidth is increased to 56.1 Mbps after applyingQoS, which is slightly less than the minimum bandwidth

http://docs.openvswitch.org/en/latest/faq/design/




43.3

73.6

44.9

14.9

0

20

40

60

80

NoQoS QoS

NoQoS NoQoS

Ban

didt

h (M

bits

/sec

)

Flow1(TCP)Flow2(TCP)

(a) Measurerd bandwidthof two TCP flows.

28.4

56.1

30.517.2

0

20

40

60

80

NoQoS QoS

NoQoS NoQoS

Ban

didt

h (M

bits

/sec

)

Flow1(UDP)Flow2(UDP)

(b) Measurerd bandwidthof two UDP flows.

46.8

8.5 8.3

0

20

40

60

80

Ban

didt

h (M

bits

/sec

)

Flow 1(UDP)QoS

Flow 2(UDP)NoQoS

Flow 3(UDP)NoQoS

(c) Measurerd bandwidthof three UDP flows.

28.1

70.6

25.5

61.1

5.5

65.0

0

20

40

60

80

NoQoS QoS NoQoS

NoQoS NoQoS QoS

Ban

didt

h (M

bits

/sec

)

Flow1(TCP)Flow2(UDP)

(d) Measurerd bandwidth ofTCP and UDP flows.

Fig. 6. Bandwidth measurement of multiple flows sharing the same network resource.

specified in the QoS setting. This is due to the characteristicof UDP which does not provide any flow and congestioncontrol mechanism. For three UDP flows sharing the samepath (Fig. 6c), Flow 1 with QoS configuration acquires 46.8Mbps while the other two flows acquire less than 8.5 Mbps.

The last case is to measure the bandwidth with a mixedflow of TCP and UDP protocol (see Fig. 6d). When QoSwas not configured, the UDP flow could obtain 61.1 Mbpswhile the TCP flow acquired 28.1 Mbps, because TCP’sflow control mechanism adjusts the transmission rate toavoid network congestion and packet drop. However, afterapplying QoS to the TCP flow (Flow 1), its bandwidthis drastically increased to 70.6 Mbps, although Flow 2 isconstantly sending UDP packets at 70 Mbps. This showsthat the QoS queue could forward TCP packets at 70 Mbpsspeed as specified in the settings, whereas UDP packets in anon-QoS queue were mostly dropped resulting in 5.5 Mbpsbandwidth measurement at the receiver. A similar result isobserved when we applied QoS configuration for the UDPflow. The UDP flow (Flow 2) with QoS setting can obtain 65Mbps while the bandwidth for Flow 1 is measured at 25.5Mbps.

The results show that our QoS configuration mechanismfor bandwidth allocation is effective for both TCP and UDPflows in the situation of multiple flows simultaneouslysharing the same network path, although the acquired band-width is varied depending on the network status. Configur-ing QoS and queue settings on OpenVSwitch in addition tothe flow rules added for a specific flow can make the priorityflow exploit more bandwidth than non-priority flows in thecongested network.

7 PERFORMANCE EVALUATION

We evaluate the proposed system and the algorithm witha real-world application and workload: Wikipedia applica-tion. Wikipedia publishes its web application, named Medi-aWiki, and all database dump files online14 so that we canreplicate Wikipedia application on our own environmentwith various settings. Testing with Wikipedia applicationbecomes more straightforward with the introduction of Wik-iBench [29], a software to deploy Wikipedia database dumpsand measure web requests to MediaWiki servers basedon the historical traffic trace. Thanks to the open-sourced

14. http://dumps.wikimedia.org

MediaWiki application and WikiBench tool, Wikipedia ap-plication is widely adopted in cloud computing research forevaluation. Our experiments with Wikipedia are conductedon the same testbed explained in the previous section, com-paring our proposed algorithm with baseline algorithmsdiscussed in Section 5.

7.1 Application Settings and WorkloadsFor empirical evaluations, we deploy two Wikipedia ap-plications consisting of multiple VMs across our testbedusing SDCon to measure the response time. A generalweb application architecture is exploited to configure theWikipedia with three types of VMs: client (presentationtier), web server (application tier), and database (data tier)VMs. Client VM is acting as a client which sends requestsparsed from the trace to web servers, in which providesweb response with the local file or the data retrieved fromthe database server. Client VMs run WikiBench program toread trace files and measure the response time of the request.Web server VMs are configured with Apache2 server to hostMediaWiki application written in PHP language. DatabaseVMs run MySQL which retrieves Wikipedia database rebuiltfrom the dump file.

We set two applications to represent two different ten-ants using the data center with a different number of VMs tocheck the impact of the TOPO-Het algorithm on the scale ofthe application. Application 1 (App 1) is deployed with fourclient VMs, four web server VMs, and one database VM,

TABLE 2VM types for Wikipedia application deployment.

VM Type CPUcores RAM Disk Software

# of VMs

App 1 App 2

Client 1 2GB 20GB WikiBench 4 2

Web server 4 7GB 80GB MediaWiki 4 2

Database 8 15.5GB 160GB MySQL 1 1

TABLE 3Network flow settings for VM traffic.

Source VM Destination VM Requested Bandwidth

Database Web server 70 Mbits/s

Web server Client 30 Mbits/s

http://dumps.wikimedia.org




0

8

16

24

1 2 3 4 5 6 7 8

VC

PUs C

apac

ity

Compute Node #

RAM FullFFD (App1)FFD (App2)Initial

(a) Number of VCPUs afterplacing VMs with FFD algo-rithm.

0

8

16

24

1 2 3 4 5 6 7 8

VC

PU

s C

apac

ity

Compute Node #

RAM FullTOPO-Het (App1)TOPO-Het (App2)Initial

(b) Number of VCPUs afterplacing VMs with TOPO-Het algorithm.

Fig. 7. Used and available CPU cores of compute nodes after VMplacement in different algorithm.

whereas Application 2 (App 2) consists of two clients, twoweb servers, and one database VM. Although they are sep-arate applications without any network traffic in between,we deploy and run both applications at the same time to seethe impact of multiple applications run by different tenantssharing the network resources of a data center. Each VM isconfigured with a predefined VM Type shown in Table 2.Before running the experiment, we created several VMs insome compute nodes (12 cores in Compute node 2, 4 coresin Compute node 4, and 1 core in Compute node 5 and 9)to represent the other tenants occupying resources. We alsomanually placed the database VM of App 2 onto Compute5 node in order to evaluate the case when some VMs of theapplication are pre-deployed. VM images are prepared withproper software settings to boost the speed of applicationdeployment.

In addition to VM configurations, network flow settingsare defined for VM traffics (Table 3). We set 30 Mbps fortraffics from web server VM to client VM, and 70 Mbpsfrom the database to web server VM. Since the maximumbandwidth available for each physical link in our testbed is100 Mbps, a web server still can utilize the entire bandwidthwhen it needs to send data to both database and clientif a whole compute node is excursively occupied by aweb server VM. Please note that all the VMs in the sameapplication are grouped as a same VM group in TOPO-Het algorithm, because they have connections to each other.Thus, there are two VM groups (one group per application)in the scenario.

7.2 Analysis of VM Allocation Decision

We submit the VM configurations and flow definitions fortwo applications to SDCon and run the VM allocationalgorithms. Fig. 7 shows CPU capacity of compute nodesin the testbed before and after allocating VMs with differentalgorithms. Before deploying two Wikipedia applications,Compute node 1, 6, and 7 were empty, while the othernodes had running VMs representing other tenants, which isshown as Initial in the figure. FFD algorithm allocates VMsfor App 2 to Compute 3 and 4 nodes, and VMs for App 1across the data center filling up empty slots. Because FFDdoes not consider network topology, VMs of both App 1and App 2 are scattered across the data center. VMs for App1 are placed almost every compute node, whereas VMs forApp 2 are consolidated in Compute 3, 4, and 5 which are still

Core1 Core2

Aggr1 Aggr2 Aggr3 Aggr4

Edge1 Edge2 Edge3 Edge4Compute2

Compute3Compute4

Compute1 Compute5

Compute6

Compute7

Compute8

ControllerCompute nodes / VMs

Switches

(a) Visualized network traffic of VMs placed with FFD algorithm.

Core1 Core2

Aggr1 Aggr2 Aggr3 Aggr4

Edge1 Edge2 Edge3 Edge4

Compute2Compute3

Compute4

Compute1Compute5

Compute6

Compute7

Compute8

Controller Compute nodes / VMs

Switches

(b) Visualized network traffic of VMs placed with TOPO-Het.

Fig. 8. Screenshots of Status Visualizer showing network traffic betweencompute nodes and switches after Wikipedia application deploymentwith different algorithms.

scattered on different network pods. Note that Compute 7and 8 could not host more VMs because of RAM limitation,even though they have free CPUs.

In comparison, TOPO-Het considers network topology,and the VMs for the same application are consolidatedwithin the same pod or edge network (Fig. 7b). For App2, new VMs are placed in Compute 5 and 8, which areunder the same network pod with the already placed VM(in Compute 5). Similarly, all VMs of App 1 are placed eitherin Compute 1 or 2, which are connected to the same edgeswitch. Also, Compute 7 remains empty after VM allocationso that the data center can save more power consumption ifCompute 7 is turned into idle mode or powered off. In short,the proposed algorithm allocates VMs to nearby computenodes aware of network topology while still provides VMconsolidation to the minimum number of compute nodes.

The network traffic visualized by SDCon Status Visu-alizer shows the effectiveness of TOPO-Het in networkresources. Fig. 8 depicts the screenshot of Status Visualizerweb UI captured two minutes after starting each experi-ment. The network topology follows the testbed architecture(Fig. 5), with orange circles with dash line refer to network




switches (Core1-2, Aggr1-4, Edge1-4), and blue circles referto compute nodes (Compute1-8) and VMs. Please note thatboth VMs and compute nodes are represented as blue dotsbecause we use a flat network for VMs instead of VLAN orother tunneling methods. The arc lines between dots referto the network traffic between those nodes, separated bythe end-to-end VM flows. Thus, more numbers of lines withdifferent colors indicate that the link had more networktraffics for different source-destination pairs of VMs. WhenVMs are placed with FFD that is lack of the informationof network topology, network traffics for both applicationsflood across all the network links in the testbed. In contrast,VMs placed with TOPO-Het are consolidated under thesame pod or edge switches which results in significantreduction of network traffics.

7.3 Analysis of Response Time

The average response time of Wikipedia applications mea-sured with WikiBench program from App 1 and App2 areshown in Fig. 9 and Fig. 10 respectively. Web requests aresent for 30 minutes duration with 20% for App 1 (approx.140,000 requests per client) and 10% (approx. 75,000 requestsper client) for App 2 from the original trace from Wikipedia.Each client injects the workload individually from a localfile to avoid the network latency and traffic between clientVMs. For clear comparison, we further separate the averageresponse time based on the type of request. DB access isthe request that needs a database access to retrieve therequested contents, whereas static file is to transfer files fromthe web server without access to the database.

Fig. 9 shows the average and CDF of response timesmeasured from App 1. The overall response time is mea-sured at 45.4 ms with the TOPO-Het algorithm which isabout one-third of the response time from FFD algorithm(153.8 ms). Because TOPO-Het places the connected VMsin a short distance to each other, the network latency issignificantly lowered transferring within an edge or podnetwork. On the other hand, FFD distributes VMs acrossthe data center which leads to increasing the response time.The result also shows that applying network managementmechanism in addition to the FFD allocation can improvethe performance. The average response time is reduced to102.8 ms by applying BWA method, and further to 59.3 mswith DF algorithm. Even for the VMs spread out across thedata center by FFD algorithm, network traffic engineeringschemes supported by SDCon can improve the performanceof the application. We can see the response times for DBaccess requests are longer than the static files because theyobviously need to access database server involving morenetwork transmission between the web server and databaseVMs and file operation in the database. Nonetheless, theresponse time for DB access and static file requests fol-low the same tendency as the overall result: TOPO-Hetresulting in the best performance whereas FFD without anetwork management scheme being the worst. CDF graphof response time from all requests (Fig. 9b) also shows thatTOPO-Het outperforms compared to baselines.

For App 2 which has been running simultaneously withApp 1, a similar result can be observed in Fig. 10. Comparedto App 1, the overall average response time is increased

to between 155.5 ms and 236.6 ms mainly because of thepoor performance of database VM. In our heterogeneoustestbed, the disk I/O performance of Compute 5 nodehosting the database VM for App 2 is lower than Compute1 and 2 nodes, because of the hardware specification. Dueto the poor performance, longer response time is observedespecially for DB access requests in App 2. Another reason isthat VMs for App 2 are more scattered across the data centerinto smaller compute nodes as observed in Fig. 7 leading toincreasing the network latency.

Nonetheless, we can still see that TOPO-Het outperformsother baselines although the average response time is in-creased compared to App 1. However, network manage-ment schemes are less effective for App 2 not improvingthe response time significantly. In fact, DF method degradesthe overall performance slightly compared to the originalFFD, increasing the response time from 233.3 ms to 236.6ms. The reason is that the smaller scale with less numberof VMs and workloads did not generate as much networktraffic as App 1, which results in the network bandwidth nota significant bottleneck for the overall performance. Ratherthan bandwidth which could be improved by SDCon’snetwork management schemes, the poor performance ofApp 2 is mainly due to the network latency caused byswitches. As shown in Fig. 10a, the time difference forstatic file requests are not as high as the difference for DBaccess requests between FFD and TOPO-Het. This impliesthe network traffic between the database and web serveris improved by TOPO-Het which allocates web server VMsinto the compute node under the same edge as the databaseVM.

In summary, results show that SDCon can performVM placement with the intended algorithm, either FFD orTOPO-Het, as well as configure network settings dynam-ically through SDN controller based on the selected trafficengineering scheme. TOPO-Het can improve the applicationperformance for both application scenarios by allocatingVMs to nearby compute nodes aware of network topology,although the level of improvement is different based onthe application configuration, the workload and the currentstatus of the infrastructure.

7.4 Analysis of Power ConsumptionWe also measured the power consumption of the testbedusing SDCon. Fig. 11 depicts the aggregated power usage ofall infrastructure per time unit measured at every minute.Note that the measurement is estimated by calculating fromenergy models with the CPU utilization level for computenodes and the number of online ports for switches asexplained in Section 4.7. We set Pidle at 147W, 111W, 102W,and Pidle at 160W, 80W, 80W, for Compute 1-2, 3-6, and 7-8respectively. For switches, we set Pstatic at 66.7W and Pport

at 1W. Although TOPO-Het could leave one compute nodeempty, the estimated power consumption is not significantlydifferent between two algorithms. Because we did not im-plement the method to power off idle compute nodes andthe aggregated CPU utilization level remains similar in bothalgorithms, the power consumption was not significantlydifferentiated between algorithms. In summary, the powerconsumption of data center with TOPO-Het remains at leastas the same level as the baseline.




434.6

58.0

153.8

346.9

41.2102.8

225.7

29.159.3

156.4

14.545.4

0

100

200

300

400

500

DB access Static files Overall

Avg

. Res

pons

e Ti

me

(ms) FFD BWA DF TOPO-Het

(a) Average response time per request of App 1.

0.00

0.25

0.50

0.75

1.00

0 500 1000 1500Response Time (ms)

Cum

ulat

ive

Pro

babi

lity

TOPO−Het

DF

BWA

FFD

(b) CDF graph of response time of App 1.

Fig. 9. Average and CDF of response time measured with Wikipedia workload from App 1.

632.8

65.8

233.3

612.0

64.7

226.4

623.0

77.3

236.6

394.1

55.0

155.5

0

100

200

300

400

500

600

700

DB access Static files Overall

Avg

. Res

pons

e Ti

me

(ms) FFD BWA DF TOPO-Het

(a) Average response time per request of App 2.

0.00

0.25

0.50

0.75

1.00

0 500 1000 1500Response Time (ms)

Cum

ulat

ive

Pro

babi

lity

TOPO−Het

DF

BWA

FFD

(b) CDF graph of response time of App 2.

Fig. 10. Average and CDF of response time measured with Wikipedia workload from App 2.

0

500

1000

1500

2000

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35

Pow

er (W

att)

Time (min)

TOPO-Het FFD

Fig. 11. Overall power usage of the testbed estimated from CPU andnetwork utilization.

Nevertheless, the result of power consumption measure-ment shows the potential application of SDCon for energy-related research such as energy-efficient joint resource pro-visioning. With the full integrity controlling both computingand networking resources, measuring power usage, andreal-time resource monitoring, empirical studies on SDC arefeasible with SDCon platform.

8 CONCLUSIONS AND FUTURE WORK

The introduction of SDN brings up many opportunities andchallenges when it is applied in cloud computing environ-ment. The increasing popularity of applying SDN in clouddata centers demands a management platform which canjointly control both networking and cloud resources. In thispaper, we introduced SDCon, an integrated control platform

for SDN and clouds to enable orchestrated resource man-agement for both resources. SDCon is capable of placing VMrequests and configuring network forwarding rules and QoSqueues with the knowledge of networking and computinginformation of the data center. It also implements real-timemonitoring for computing and networking resources whichcan be further exploited for dynamic resource provisioning.The power monitoring module can calculate the estimatedpower consumption based on the resource utilization usinga power model, and the visualizer displays an overall statusof the data center.

We also proposed a VM placement algorithm for jointlyconsidering both computing and networking resources inheterogeneous infrastructure. The algorithm consolidatesVMs for the same application with network connectivityinto closely connected compute nodes in order to decreasethe network traffic as well as to improve the applicationperformance. The algorithm is evaluated on our testbedcontrolled by SDCon platform and compared with baselines.The results show that the proposed algorithm outperformsthe compared state-of-the-art baselines. The effectiveness ofSDCon’s controllability is also shown through the validationand evaluation experiments, including bandwidth alloca-tion with QoS configuration and dynamic flow scheduling.

The current version of SDCon has a flaw on the scala-bility for large-scale infrastructure such as those involvingmultiple data centers. When we designed and implementedSDCon, it aims at working on a small-scale testbed con-figured within the lab, due to the limited infrastructure




resources. Scalability can be achieved by creating a clusterwith multiple instances of SDCon with the introduction ofEast-West-bound APIs adopted from SDN controller de-sign [30]. In addition to SDCon, the underlying cloud andSDN controllers (OpenStack and OpenDaylight) have to beable to scale out in accordance with the size of cloud infras-tructure, e.g., the monitoring modules need more carefulattention in scalability to be able to collect and aggregatethe real-time information in a timely manner.

For the same reason, the evaluation of TOPO-Het algo-rithm was performed only within the small-scale testbed inour laboratory with a Wikipedia application. More extensiveevaluations can be performed to show the effectiveness ofTOPO-Het in a large-scale infrastructure with various appli-cation models. To support large-scale evaluations, we havedeveloped a simulator called CloudSimSDN [9] and evalu-ated several joint host-network provisioning algorithms [4],[31] in the simulation environment setting. In these earlierresearch work, we conducted simulation-based experimentsin large-scale and demonstrated the scalability of similarjoint VM and network resource provisioning approacheswith hundreds of compute nodes and network switches. Inthis paper, we followed a similar approach for designing theintegrated platform and the joint provisioning algorithm, sothat our approaches are essentially scalable as shown in theprevious work.

In the future, SDCon can be extended to support alter-native cloud management platforms and SDN controllersoftwares other than OpenStack and OpenDaylight. Thepower monitoring module can be improved for accuratemeasurements by installing a physical monitoring equip-ment for compute nodes and switches instead of model-based theoretical calculation. Also, the platform can be im-proved to support a variety of network topologies whereasthe initial version was developed in consideration of three-tier Fat-tree topology only. More functional features, such asmanaging virtual network functionalities, network middle-boxes, and security modules, can be added to the platformfor autonomic software-defined clouds. Such features willallow SDCon to be employed for developing innovative ap-proaches dealing with energy-aware resource provisioning,NFV scheduling, and security threat protection. TOPO-Hetalgorithm can be improved by importing various migrationpolicies for VMs and flows to adapt to the dynamic updateof the infrastructure. More evaluations with complex appli-cation models including Map-Reduce and batch processingcan be further performed with a large-scale testbed.

ACKNOWLEDGMENTS

This work is partially supported by ARC Discovery Project.We thank Adel Nadjaran Toosi and Minxian Xu for provid-ing valuable feedback to improve the quality of the paper.We also thank Editor-in-Chief (Manish Parashar), AssociateEditor, and anonymous reviewers for their valuable com-ments and suggestions that helped in improving the papersignificantly.

REFERENCES

[1] R. Buyya, R. N. Calheiros, J. Son, A. V. Dastjerdi, and Y. Yoon,“Software-defined cloud computing: Architectural elements and

open challenges,” in Proceedings of the 3rd International Conferenceon Advances in Computing, Communications and Informatics. IEEEPress, 2014.

[2] Y. Jararweh, M. Al-Ayyoub, A. Darabseh, E. Benkhelifa, M. Vouk,and A. Rindos, “Software defined cloud: Survey, system andevaluation,” Future Generation Computer Systems, vol. 58, pp. 56– 74, 2016.

[3] H. Jin, T. Cheocherngngarn, D. Levy, A. Smith, D. Pan, J. Liu,and N. Pissinou, “Joint host-network optimization for energy-efficient data center networking,” in Proceedings of the 2013 IEEE27th International Symposium on Parallel Distributed Processing, May2013, pp. 623–634.

[4] J. Son, A. V. Dastjerdi, R. N. Calheiros, and R. Buyya, “SLA-awareand energy-efficient dynamic overbooking in SDN-based clouddata centers,” IEEE Transactions on Sustainable Computing, vol. 2,no. 2, pp. 76–89, April 2017.

[5] J. W. Jiang, T. Lan, S. Ha, M. Chen, and M. Chiang, “Joint vmplacement and routing for data center traffic engineering,” inProceedings of the 2012 IEEE INFOCOM, March 2012, pp. 2876–2880.

[6] K. Zheng, X. Wang, L. Li, and X. Wang, “Joint power optimizationof data center network and servers with correlation analysis,” inProceedings of the 2014 IEEE INFOCOM, April 2014, pp. 2598–2606.

[7] R. Cziva, S. Jout, D. Stapleton, F. P. Tso, and D. P. Pezaros, “SDN-based virtual machine management for cloud data centers,” IEEETransactions on Network and Service Management, vol. 13, no. 2, pp.212–225, June 2016.

[8] K. Zheng, W. Zheng, L. Li, and X. Wang, “PowerNetS: Coordinat-ing data center network with servers and cooling for power op-timization,” IEEE Transactions on Network and Service Management,vol. 14, no. 3, pp. 661–675, Sept 2017.

[9] J. Son, A. V. Dastjerdi, R. N. Calheiros, X. Ji, Y. Yoon, andR. Buyya, “CloudSimSDN: Modeling and simulation of software-defined cloud data centers,” in Proceedings of the 15th IEEE/ACMInternational Symposium on Cluster, Cloud and Grid Computing, May2015, pp. 475–484.

[10] B. Lantz, B. Heller, and N. McKeown, “A network in a laptop:rapid prototyping for software-defined networks,” in Proceedingsof the 9th ACM SIGCOMM Workshop on Hot Topics in Networks.ACM, 2010, p. 19.

[11] C. H. Benet, R. Nasim, K. A. Noghani, and A. Kassler, “Open-StackEmu - a cloud testbed combining network emulation withOpenStack and SDN,” in Proceedings of the 2017 14th IEEE AnnualConsumer Communications Networking Conference, Jan 2017, pp. 566–568.

[12] B. Martini, D. Adami, A. Sgambelluri, M. Gharbaoui, L. Donatini,S. Giordano, and P. Castoldi, “An SDN orchestrator for resourceschaining in cloud data centers,” in Proceedings of the 2014 EuropeanConference on Networks and Communications, June 2014, pp. 1–5.

[13] B. Martini, D. Adami, M. Gharbaoui, P. Castoldi, L. Donatini, andS. Giordano, “Design and evaluation of SDN-based orchestrationsystem for cloud data centers,” in Proceedings of the 2016 IEEEInternational Conference on Communications, May 2016, pp. 1–6.

[14] M. Gharbaoui, B. Martini, D. Adami, S. Giordano, and P. Castoldi,“Cloud and network orchestration in SDN data centers: Designprinciples and performance evaluation,” Computer Networks, vol.108, no. Supplement C, pp. 279 – 295, 2016.

[15] D. Adami, B. Martini, A. Sgambelluri, L. Donatini, M. Gharbaoui,P. Castoldi, and S. Giordano, “An SDN orchestrator for cloud datacenter: System design and experimental evaluation,” Transactionson Emerging Telecommunications Technologies, vol. 28, no. 11, p.e3172, 2017.

[16] J. Tordsson, R. S. Montero, R. Moreno-Vozmediano, and I. M.Llorente, “Cloud brokering mechanisms for optimized placementof virtual machines across multiple providers,” Future GenerationComputer Systems, vol. 28, no. 2, pp. 358 – 367, 2012.

[17] A. N. Toosi, C. Qu, M. D. de Assuno, and R. Buyya, “Renewable-aware geographical load balancing of web applications for sus-tainable data centers,” Journal of Network and Computer Applications,vol. 83, no. Supplement C, pp. 155 – 168, 2017.

[18] A. Beloglazov and R. Buyya, “Optimal online deterministic al-gorithms and adaptive heuristics for energy and performanceefficient dynamic consolidation of virtual machines in cloud datacenters,” Concurrency and Computation: Practice and Experience,vol. 24, no. 13, pp. 1397–1420, 2012.

[19] M. Al-Fares, A. Loukissas, and A. Vahdat, “A scalable, commoditydata center network architecture,” in Proceedings of the ACM SIG-




COMM 2008 Conference on Data Communication. New York, NY,USA: ACM, 2008, pp. 63–74.

[20] S. Pelley, D. Meisner, T. F. Wenisch, and J. W. VanGilder, “Under-standing and abstracting total data center power,” in Workshop onEnergy-Efficient Design (WEED), 2009.

[21] X. Wang, Y. Yao, X. Wang, K. Lu, and Q. Cao, “CARPO:Correlation-aware power optimization in data center networks,”in Proceedings of the 2012 IEEE INFOCOM, March 2012, pp. 1125–1133.

[22] J. Kang and S. Park, “Algorithms for the variable sized bin packingproblem,” European Journal of Operational Research, vol. 147, no. 2,pp. 365 – 372, 2003.

[23] A. Beloglazov, J. Abawajy, and R. Buyya, “Energy-aware resourceallocation heuristics for efficient management of data centers forcloud computing,” Future Generation Computer Systems, vol. 28,no. 5, pp. 755–768, May 2012.

[24] R. Wang, R. Esteves, L. Shi, J. A. Wickboldt, B. Jennings, andL. Z. Granville, “Network-aware placement of virtual machineensembles using effective bandwidth estimation,” in Proceedings ofthe 10th International Conference on Network and Service Management(CNSM) and Workshop, Nov 2014, pp. 100–108.

[25] H. E. Egilmez, S. T. Dane, K. T. Bagci, and A. M. Tekalp, “Open-QoS: An OpenFlow controller design for multimedia delivery withend-to-end quality of service over software-defined networks,”in Proceedings of the 2012 Asia-Pacific Signal Information ProcessingAssociation Annual Summit and Conference, Dec 2012, pp. 1–8.

[26] R. Wang, S. Mangiante, A. Davy, L. Shi, and B. Jennings, “QoS-aware multipathing in datacenters using effective bandwidth esti-mation and SDN,” in Proceedings of the 12th International Conferenceon Network and Service Management (CNSM), Oct 2016, pp. 342–347.

[27] K. Zheng, X. Wang, and J. Liu, “DISCO: Distributed traffic flowconsolidation for power efficient data center network,” in Pro-ceedings of the 16th International IFIP TC6 Networking Conference,Networking. IFIP Open Digital Library, 2017.

[28] F. P. Tso, D. R. White, S. Jouet, J. Singer, and D. P. Pezaros, “TheGlasgow Raspberry Pi cloud: A scale model for cloud computinginfrastructures,” in Proceedings of the 2013 IEEE 33rd InternationalConference on Distributed Computing Systems Workshops, July 2013,pp. 108–112.

[29] G. Urdaneta, G. Pierre, and M. van Steen, “Wikipedia workloadanalysis for decentralized hosting,” Elsevier Computer Networks,vol. 53, no. 11, pp. 1830–1845, July 2009.

[30] S. H. Yeganeh, A. Tootoonchian, and Y. Ganjali, “On scalabilityof software-defined networking,” IEEE Communications Magazine,vol. 51, no. 2, pp. 136–141, February 2013.

[31] J. Son and R. Buyya, “Priority-aware VM allocation and networkbandwidth provisioning in software-defined networking (SDN)-enabled clouds,” IEEE Transactions on Sustainable Computing, 2018.[Online]. Available: http://doi.org/10.1109/TSUSC.2018.2842074

Jungmin Son is a PhD candidate with the CloudComputing and Distributed Systems (CLOUDS)Laboratory at the University of Melbourne, Aus-tralia. He received the B.Eng. degree in Informa-tion and Computer Engineering from Ajou Uni-versity, Korea and the M.Info.Tech. degree fromthe University of Melbourne in 2006 and 2013respectively. His research interests include SDN-enabled cloud computing, resource provisioning,scheduling, and energy efficient data centers.

Dr. Rajkumar Buyya is a Redmond Barry Dis-tinguished Professor and Director of the CloudComputing and Distributed Systems (CLOUDS)Laboratory at the University of Melbourne, Aus-tralia. Dr. Buyya is recognized as a ”Web ofScience Highly Cited Researcher” in 2016 and2017 by Thomson Reuters, a Fellow of IEEE,and Scopus Researcher of the Year 2017 withExcellence in Innovative Research Award by El-sevier for his outstanding contributions to Cloudcomputing. For further information on Dr.Buyya,

please visit his cyberhome: www.buyya.com

http://doi.org/10.1109/TSUSC.2018.2842074

www.buyya.com

sdcon: integrated control platform for software-defined clouds · clouds [3]–[8], in which...

Documents