Experiments on Multi-Layer Network Virtualizationtowards the Software Defined Transport Network

Akeo Masuda, Akinori Isogai, Daisaku Shimazaki, Yoshihiko Uematsu and Atsushi HiramatsuNTT Network Service Systems Laboratories, NTT Corporation

Musashino-shi, Tokyo, JapanEmail: masuda.akeo@lab.ntt.co.jp

Abstract—This paper proposes a novel architecture whichenables software defined networking not only at the routinglayer but also at the transport layer. Proposed architectureprovides multiple SDTNs with wide range of controllabilitylevel, in spite that the SDTNs coexist upon a shared multi-layered network infrastructure. We have conducted a nation-wide experiment where we have provided SDTNs to practicalusers such as broadcasting studios. Through the experiments, wehave successfully verified the resource management mechanismand network control functionalities.

I. INTRODUCTION

Recently the main players and the drivers of the devel-opment of networking technologies seem to be shifting tooperators and users of datacenter. Software developers of cloudoperators are eager to totally program the operation of notonly their computing equipments, but also the network. Insideand between the datacenters, there are numerous dataflowsbetween virtual machines (VMs) running upon numerous com-puters, and they keep being generated and changed dynami-cally. The concept of software defined network is expectedto release the network operation from time-consuming tasksof manual configuration of each network equipment alongwhich the flow traverses. This enables programmed controlof the dataflow routing in order to achieve optimization,scalability and resiliency of the network, similar to the way ofmanagement where the cloud operators program the usage ofcomputing resources. OpenFlow[1] is expected to be the mainenabler of SDN (Software Defined Network). This technologylets cloud operators to explicitly designate the route at flowlevel granularity, and slice the network capacity to multipleindependent tenants.However, at least in the past, SDN had been seen to be only

the enabler of control function at the flow routing level. Webelieve that controllability should be enhanced deeply to thetransport level for full utilization of network resources. Themain contribution of this paper is to address an architectureof SDTN (Software Defined Transport Network) that enablesvirtualization and programmability of the transport layer.Virtualization and programmability is the major require-

ments for future network operation. From the user’s pointof view, they can be able to achieve much flexibility, highperformance and resiliency if they can also program thetransport layer of the inter-cloud network. For example, theymay lack of bandwidth in case they newly generate a flow or

change the route of the existing flows. In this case, they willbe able to achieve better performance by acquiring additionalnetwork resources, or optimizing the topology of tunnel paths.For another example, they can achieve high availability ifthey can prepare a SRLG(Shared Risk Link Group)-awareprotection path at the transport layer, designed in accordanceto the design of server redundancy.

On the other hand, network carriers do not devote their net-work resources to a single service or single user. They logicallyslice their network resources to launch a new service includinginter-cloud connection, and provide the portion of the slice toeach user. Sharing the infrastructure by multiple usage of thenetwork is usually done in most of the network providers tokeep their competitiveness by the cost efficient operation oftheir infrastructure. This can be seen as a virtualization of thenetwork infrastructure. The difficulty of network virtualizationis to offer the programmability at the same time.

Speaking generally, network providers do not desire to allowusers to freely configure the network equipments. It maycause serious problem if a certain user of the network directlyconfigures the functionalities of the routers and switches.It will prevent the fair use of the network among multipleusers and services, and causes conflict between the controlsfrom multiple users. In order to offer programmability of thetransport layer, we need a new technology to overcome thisproblem.

Several works had addressed the architecture of total controlof the network including the SDN layer and transport layer [2],[3]. However, previous researches only focus on the integratedcontrol of both layers by unification of the control plane.Software defined control of the transport layer where multipleusers share the infrastructure is still an open issue at thismoment.

We propose the SDTN architecture that enables networkvirtualization in the transport layer, which provides secureshared use and programmability at the same time to multipleusers.

This paper is organized as follows. Next section explainsthe architecture of SDTN. In section III, we illustrate thedesign of the experimental network. Section IV discuses aboutwhat we confirmed through the experiments. Finally, sectionV concludes the paper.

2012 13th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed

Computing

978-0-7695-4761-9/12 $26.00 © 2012 IEEE

DOI 10.1109/SNPD.2012.134

661

II. THE SOFTWARE DEFINED TRANSPORT NETWORK

A. SDTN ArchitectureSDTN architecture is designed to enable network virtualiza-

tion in the transport layer, that provides secure shared use andprogrammability the same time to multiple users. Key conceptis that we provide multiple SDTNs upon a shared multi-layered network infrastructure for users. Note that “users”could be the cloud tenants, cloud service providers and otheroperators of network services.SDTN is made of set of network resources such as links,

wavelengths, unit of bandwidth and switching capabilities.Each unit of resources is assigned permission to users. Usersare allowed to setup optical and packet transport paths mak-ing use of the resources that are assigned permission tothemselves. This ensures the portion of the network to beindependently controlled without any contention.The key component of the architecture is the Physical Net-

work Manager (PN Manager), which is the unified controllerof the optical network. It provides functions for the usersto invoke network control such as resource allocation andpath setup in order to program their own software of networktopology designing (Fig. 1). PN Manager provides API [4] forSDTN operators to develop a software to control their SDTN.Network providers are able to optimize the operation of their

network infrastructure. For example, optimization of resourceallocation to each slice according to the traffic demands willprovide statistical multiplexing effect. Furthermore, sharingredundant resources prepared for forecasted future demand anddetour routes in case of failures will provide high efficiencyof capital expenditure. Since the SDTN is logically formedby set of circuits that can be provisioned automatically, it alsoenables fast launch of new services by making use of availablenetwork resources. It may provide survivability of services incase of disaster, by letting the slices to share a small portionof the remained part of the network.On the other hand, SDTN benefits the users in terms of the

programmability of the transport network. As mentioned inthe previous section, users can optimize the transport layer aswell as the flow routing layer. Cloud operators can be able toprogram the total system including the computing resources,flow routing, underlying circuits, and the amount of allocatednetwork resource to configure the circuits.

B. Recursive VNT construction upon multi-layer network in-frastructureFor the physical network infrastructure, we assume a multi-

layer network which is consisted of layer-1, 2, 3 nodes suchas optical cross-connects (OXCs), L2 switches and IP routers.This can be prepared with ordinary products that are alreadyavailable in the market.In each layer, resources are defined in order to

setup a path. Here we incorporate the notion ofV irtualNetworkTopology(V NT )[5]. When a pair ofnodes in a certain layer is connected with a path in the lowerlayer, it will form a link in the upper layer. For example,

L2 switch, Router

OXC

LinkRouter

�

VN#1Dedicated Private

Resource

SharedVN#2Dedicated

SDTN(AllocatedResources)

(4) Setup Paths

VN Topology

PN Manager

(1) Collect Resource Info(OSPF-TE/LLDP)

Virtual Network (VN) #1

(3) Allocate resources

Physical network(PN)

Virtual Network (VN) #2

SDTN Controller

(2) Configure permission to VNs

Optical Paths

Fig. 1. Construction of VNTs using the resources allocated from the PNManager.

layer-2 link will be provided by connecting a pair of layer-2switches by an optical path through layer-1 nodes using thelayer-1 resources (e.g. GMPLS TE-links). Then, those layer-2links can be seen as resources to setup a path in the layer-2,by which the layer-3 IP routers can be connected in orderto form an IP link. In this manner, VNT in a certain layercan be provided dynamically and recursively. SDTNs areprovided to the users as the VNTs at the desired layers.

Consequently, layer-2 and 3 SDTNs are provided by uti-lizing network resources of layer-1 and 2. Resources usedto setup layer-1 optical paths are routers, OXCs, fibers andwavelengths. They are handled in a unit called TE-link whichdefined in the GMPLS technology. We can describe andutilize the resource to setup optical paths because proper-ties of TE-link provides sufficient information of the linksuch as connected node address, link address, maximum andminimum reservable bandwidth, switching capability (fiber,lambda, TDM and packet), SRLG, and so on. Information ofthe existing resources are automatically collected by listeningto OSPF-TE[6] advertisement in GMPLS.

Resources used to setup layer-2 paths can be handled byL2SC TE-links that are also defined in GMPLS. However, asL2SC is not actually popular in the market, we can make use ofethernet related technologies such as LLDP (IEEE 802.1AB).

To be precise, there are no exact technologies to be namedas layer-2 path. What is needed here is actually a technologyto setup a packet transport path to slice the huge bandwidthprovided by the layer-1 path that is too much to offer to users.Here we can employ MPLS-TP LSPs, or S-VLANs defined inPBB (Provider Backbone Bridge) configured with rate limits.

662

PN Operator

VN Operator

PN Operator

VN Operator

PN Operator

VN Operator

Layer-1

Layer-2

Layer-3AllocationReturn

Path setupPath ReleaseEquivalent

Resource

Path

L1 path between L2 switches

L2 path between L3 Router

IP Link

L1 path between L2 switches

Fig. 2. Multi-layer network resource state machine.

C. Multi-layer resource state machineFor each unit of resource, the administrator of the physical

network will apply permission for SDTNs to obtain them.Using the obtained resources, SDTN operators are allowed tosetup paths in order to form their own VNT. Fig.2 shows thestate machine we have designed for the multi-layer resourcemanagement model. Users are permitted to obtain layer-1resources. Using the layer-1 resources, users can setup layer-1paths between layer-2 or 3 node pairs in order to form linksat layer-2 or 3. In addition, resources can also be assigned tothe administrator of the total physical network infrastructure,which we call the PN (Physical Network) operator. PN oper-ator can setup layer-1 paths to produce layer-2 resources, andthen assign permission to users. By this, users are also able tostart from obtaining layer-2 resources in order to form layer-3VNT by connecting IP routers by layer-2 paths.Resources are permitted as either dedicated or shared.

Shared resources can be noticed by multiple VNs, but it willbe allocated to only one of that VNs. Sharing the unallocatedresources enables capital cost reduction of the physical net-work infrastructure, by sharing the redundant resource thatshould have been prepared for each of the network service ifno virtualization is adopted. Fig.3 shows the resource accesscontrol model.Balance of the amount of resources allocated to each virtual

network can be modified flexibly by changing the permissionof each resource. This enables efficient utilization of theresources in accordance to the change in traffic demands.

D. Resource abstraction and variety of controllability levelHere we discuss on abstracting the network resources.

Previously we explained that users form SDTN for them bythemselves, utilizing the resources obtained at the granularityof links. However, we should be aware that not all of theusers of the network require controllability at that level. Someof them don’t need to, some of them don’t want to, andsome of them are not the network experts skilled enough

Initially permitted only to PN administrator

#ADedi-cated

#BDedi-cated

#A #BShared

#CDedi-cated

Permission

#A #B #CAllocated exclusively

to each SDTN

Assignment of Permission

Obtain/Release

Resource detection(OSPF-TE/LLDP)

Fig. 3. Resource access control model.

to design the SDTN at that level. Therefore, abstraction ofnetwork resources may provide much usefulness to the users.We assume following three types of abstraction: type-T , atopology which contains links and nodes, type-P , a set ofpoint-to-point paths, and type-S, a virtual switch.In type-T , users are provided with links and nodes in order

to setup transport paths by their own. Users are provided alarge range of freedom to control the network, such as de-signing multi-layer topology optimization or capacity planningaccording to the traffic demands, and provisioning protectionpaths. This type can be seen as an abstraction at the mostlower level.In type-P , users are provided with a set of point-to-point

paths. Users only request the paths that connect the desiredendpoint in order to connect the nodes owned by the users.Users do not have the level of controllability as much as type-T , but still it is their work to design the topology formed bythe provided paths and their nodes.In type-S, the provided SDTN is seen as a single switch.

Users are provided with connection points, as if they areprovided with several ports of a big switch. Users only need toconnect their equipments to those ports, and the packets willbe forwarded to any of the points they have connected. Thistype can be seen as an abstraction at the most higher level.

III. NATION-WIDE EXPERIMENTAL NETWORKAs shown in Fig.4, we have implemented a network in-

frastructure for experiments, upon a national R&E networkin Japan, called JGN-X[7]. Through June 2011 to February2012, we have connected four OXCs, ten Layer-2 switches,and six IP routers upon JGN-X. Scale of the network in-frastructure changed at each experiment event. At most thenumber of nodes was 14. Network spanned over the nation,from Hokkaido to Okinawa, which are the north and southend of Japan. Some of the links had 10 Gbps capacity, andothers had 1 Gbps.We have implemented an SDTN controller software with

GUI that invoke the PN Manager API in order to let SDTNusers to obtain resources and setup paths.For some of the users, layer-1 resources were directly

allocated. Those users formed IP links by connecting IP router

663

IP Router

Fukuoka

Okinawa

Sapporo

MusashinoOsaka

Otemachi

Koganei

Layer-2 Switch OXC

Fig. 4. Experimental network infrastructure implemented upon JGN-X.

pairs with GMPLS optical paths. There was another case thatthe PN operator connected layer-2 Ethernet switches withoptical paths in order to produce layer-2 resources. Theseresources were divided by setting up point-to-point S-VLANswith upper rate limit. SDTNs consisted of set of S-VLANswere provided to users. Users setup C-VLANs between thedesired access points in order to transmit their data flows.Although we haven’t completed the evaluation from the

performance point of view, we report that time needed tosetup a single optical path was about 15 seconds, and thatof a single point-to-point S-VLAN connectivity was around10-15 seconds. Note that these results may differ according toconditions. These are expected to be shortened by additionaltuning efforts.

IV. RESULTSThrough the network operation in the experiment which was

close to practical use, we successfully confirmed the feasibilityand the benefits provided by our control architecture.

A. Multiple SDTN operationTotally we had provided 11 SDTNs to users including

experiment project of new generation network technologies,demonstrations for international conference, and live videotransmission for commercial TV program broadcasting. Atmost, five SDTNs were operated simultaneously.1) Independent control of multiple SDTNs: All of the users

of 11 SDTNs were able to completely carry out their eventof such as experiment, demonstration, and broadcasting. Thismeans that, we confirmed that user traffic was successfullyisolated in terms that no user experienced any trouble causedby network control or data traffic of other users that share thenetwork infrastructure. Indeed, two SDTNs used by broad-casting studios had changed the topology of their SDTN inadvance of a planned construction work that was known toforce outage of the connection at a certain physical link.Even in this case, network control to change topology hadcarried out while broadcasting studios using other SDTNswere transmitting their commercial video stream.

2) Dynamic resource allocation: In the experiment event inFebruary 2012, we have provided layer-2 SDTNs to four TVbroadcasting studio groups. As bandwidth capacity of most ofthe links was 1 Gbps, we sliced the network to provide SDTNswith limited capacity of 150 Mbps each. As the topologies ofSDTNs were different according to the required access pointamong users, reserved and residual capacity at each physicallinks were different. Residual capacity was maintained as abandwidth pool that can be allocated dynamically accordingto user’s requests.Two of the broadcasting studios turned out to require larger

amount of bandwidth capacity for their video transmission. Inone case, they needed to simultaneously transmit video filefor remote TV program editorial and live streaming for newsprogram. Total bandwidth usage exceeded the default alloca-tion of 150 Mbps, so we additional capacity was allocated tothem to enhance the limit to be 200 Mbps. In another case,a broadcasting studio desired to try a new video encoder thatconsumes bandwidth of 150 Mbps. Also in this case, we addedallocation to let it enhance to 200 Mbps. These operations ofresource allocation was also done during the time when otherSDTN users were transmitting commercial video stream.3) Abstraction variety: Through the experiment, we were

able to test the usage of SDTNs with all three variations ofabstraction level which mentioned above in section II.SDTN for a research project that tested their proposal of

high-efficiency layer-4 protocol was provided in the mannerof type-T , a topology which contains links and nodes, Wealso provided measurement functions that the user were ableto check the precise performance in terms of data rate,jitter at multiple measurement point implemented inside thenetwork. By analysis of the performance degradation point,they were able to optimize the transmission path. As a result,this user was successful in achieving their highest record ofperformance. This experiment can be seen as a successful casethat the high level of controllability of SDTN had providedbenefit to the user.Another experiment that we provided a SDTN for users to

demonstrate their OpenFlow enabled equipments. As the userside nodes were capable of controlling the flow route withOpenFlow technology, they only needed a path to connect theirnodes. This experiment can be seen as a use case of type-P ,a set of point-to-point paths. In addition, we have successfullytested path switchover in the transport layer. As the transportpath was provided by Ether-over-MPLS circuit, the switchoverdid not cause any packet losses, and we confirmed the isolatedcontrol in independent layers.Finally, SDTNs provided to most of the broadcasting stu-

dios, except the ones that operated the topology changedescribed above, was a case of resource abstraction type-S,which the network can be seen as a virtual switch. Most ofthe broadcasting studios do not care for the inner topologyof the network. They only desire to connect the camera crewsites and editorial facilities, and broadcasting stations to theaccess points of the network. In this experiment, topology ofthe SDTN was designed and operated by physical network

664

operator. However, ideally the network should automaticallydesign and setup an SDTN with the optimal topology withoptimal bandwidth capacity according to the connectivityrequirements submitted from the users. This case had impliedmany future issues for us of the value-adding functions thatthe transport network can provide.

V. CONCLUSIONSDTN is a slice of a physical network that can be controlled

independently by the user of it. As mentioned in this paper,we believe there should be many variety of how the SDTNis provided to the users, in terms of abstraction level andcontrollability level. The way to provide the SDTN should bedifferent according to the user’s requirements. For example,advanced users will be able to totally program the network ateach layer of the network, by making use of SDTN functionsin addition to the SDN functions at the flow routing layer.On the other hand, for users that do not care about the innernetworking technologies, it may be beneficial for them if thenetwork can offer useful functions to users such as automaticcapacity designing and topology optimization. The experimentresults shown in this paper are valuable findings derived frompractical use cases that suggests us of the future researchtopics. Further discussions are expected to be focused ondefining the total architecture and the interfaces between usersystems and the SDTN controllers such as our PN Manager.

ACKNOWLEDGEMENTS

The authors would like to thank Dr. Kazumasa Kobayashi,Yoshihiko Kanaumi and all of the JGN-X related researchersand engineers in NICT for strongly supporting us on theexperiments.

REFERENCES[1] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson,

J. Rexford, S. Shenker, and J. Turner, “OpenFlow: enabling innovationin campus networks,” SIGCOMM Comput. Commun. Rev., vol. 38, no. 2,pp. 69–74, Mar. 2008.

[2] S. Das, G. Parulkar, N. McKeown, P. Singh, D. Getachew, and L. Ong,“Packet and circuit network convergence with OpenFlow,” in OpticalFiber Communication Conference. Optical Society of America, 2010,p. OTuG1.

[3] S. Azodolmolky, R. Nejabati, E. Escalona, R. Jayakumar, N. Efstathiou,and D. Simeonidou, “Integrated OpenFlow–GMPLS control plane: anoverlay model for software defined packet over optical networks,” Opt.Express, vol. 19, no. 26, pp. B421–B428, Dec 2011.

[4] A. Masuda, A. Isogai, T. Miyamura, K. Shiomoto, and A. Hiramatsu,“Application-defined control of virtual networks over IP-optical net-works,” in CNSM. IEEE, 2011, pp. 1–6.

[5] K. Shiomoto, D. Papadimitriou, J. L. Roux, M. Vigoureux, and D. Brun-gard, “Requirements for GMPLS-Based Multi-Region and Multi-LayerNetworks (MRN/MLN),” RFC 5212 (Informational), Internet EngineeringTask Force, Jul. 2008.

[6] K. Kompella and Y. Rekhter, “OSPF Extensions in Support of Gener-alized Multi-Protocol Label Switching (GMPLS),” RFC 4203 (ProposedStandard), Internet Engineering Task Force, Oct. 2005.

[7] “New generation network testbed JGN-X,” http://www.jgn.nict.go.jp/.

665