document properties d 1.3 pu - cordis.europa.eu · been raised with the study and implementation of...

Document: FP7-ICT-2011-8-318115-CROWD/D 1.3Date: March 3, 2015 Security: ConfidentialStatus: Submitted to EC Version: 1.1

Document Properties

Document Number: D 1.3

Document Title:

Final architecture design

Document Responsible: A.Morelli (INCS)

Document Editor: A.Morelli (INCS)

Authors:

Arianna Morelli (INCS) Claudio Bottai (INCS)Fabio Toninelli (INCS) Rohit Goupta (SIG)Vincenzo Mancuso (IMDEA) Arash Asadi (IMDEA)Vincenzo Sciancalepore (IMDEA) Christian Vitale (IMDEA)Sebastien Auroux (UPB) Martin Dräxler (UPB)Antonio de la Oliva (UC3M) Pablo Serrano (UC3M)

Target Dissemination Level: PU

Status of the Document: Submitted to EC

Version: 1.1

Production Properties:

Reviewers: Erick Bizouarn (ALBLF), Antonio de la Oliva (UC3M), andEngin Zeydan (AVEA)

Document History:

Revision Date Issued by Description

1.0 2014-12-27 INCS Submitted to EC

1.1 2015-03-03 INCS Updated section 3.3

Disclaimer:This document has been produced in the context of the CROWD Project. The research leading to these results hasreceived funding from the European Community’s Seventh Framework Programme (FP7) under grant agreementn◦ 318115.All information in this document is provided “as is” and no guarantee or warranty is given that the informationis fit for any particular purpose. The user thereof uses the information at its sole risk and liability.For the avoidance of all doubts, the European Commission has no liability in respect of this document, which ismerely representing the authors view.

CROWD Confidential i

Ref. Ares(2015)964381 - 04/03/2015


Abstract:This document presents the final CROWD architecture, which is based on the preliminary ar-chitecture released in deliverable D1.1 and that takes into account the requirements that havebeen raised with the study and implementation of CROWD controllers and functionalities forMAC enhancements, backhaul optimization, and connectivity management. The document onlyreports on the architectural changes with respect to the previous release of the CROWD archi-tecture. However, the document provides the reader with an overview of the technical reasonsthat have driven the refinement of the architecture. The APIs developed in the project are alsoreported, for completeness, in the appendix. Moreover, the document includes the disseminationactivities carried out since August 2013, as an update to previous dissemination reports.

Keywords:

Control architecture, Energy efficiency, Long Term Evolution, WiFi, small cells, dense networks, HetNets,SDN, SDR

ii Confidential CROWD


Contents

List of Figures iv

List of Project Partners vi

List of Acronyms viii

Executive summary 1

Key contributions 3

1 Introduction 5

2 Scenarios 72.1 Scenario 1: Self-optimising dense networks . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 Scenario 2: Optimised mobility in dense radio access networks . . . . . . . . . . . . . 8

3 Architecture 113.1 Consolidated architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 Interfaces between sub-systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.3 The Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.4 Applications and MAC-layer enhancements . . . . . . . . . . . . . . . . . . . . . . . 16

3.5 Backhaul and long-term radio configuration . . . . . . . . . . . . . . . . . . . . . . . 22

3.6 Connectivity management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.7 Feedback on real test-bed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4 Dissemination activities 294.1 Journals and Magazines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.2 Conferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.3 Tutorials & Panels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.4 Posters & Demo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.5 Standardization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.5.1 IEEE 802.1cf (OMNIRAN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.5.2 IEEE 802.21d . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.5.3 IEEE 802 EC Privacy Recommendation SG . . . . . . . . . . . . . . . . . . . 33

4.5.4 IETF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.6 Editorial Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.7 Media and Representation Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.8 Submitted/Accepted in the next period . . . . . . . . . . . . . . . . . . . . . . . . . 35

5 Conclusion 37

A CLC Northbound interfaces 41A.1 OpenDaylight CLC CROWD northbound REST Interfaces . . . . . . . . . . . . . . 41

CROWD Confidential iii


B CLC Southbound interfaces 61B.1 OpenDaylight CLC CROWD southbound Interfaces . . . . . . . . . . . . . . . . . . 61B.2 Interfaces and primitives for Long Term Evolution (LTE) . . . . . . . . . . . . . . . 66B.3 Interfaces and primitives for WiFi . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

C CRC Northbound interfaces 75

Bibliography 79

iv Confidential CROWD


List of Figures

2.1 Illustration of Scenario 1: network components involved in self-organization operation. 82.2 Illustration of Scenario 1 in case of low offered load. . . . . . . . . . . . . . . . . . . 92.3 Illustration of Scenario 2: network components involved in connectivity management

operation with mobility optimisation. . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1 CROWD network architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.2 CLC: OpenDayLight extention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.3 LTE modules in the Local Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.4 CRC: CROWD CLCManager modules . . . . . . . . . . . . . . . . . . . . . . . . . . 163.5 Dynamic behaviour of CROWD Almost Blank Subframe technique (CABS) in a

network with 7 base stations and Almost Blank Sub-Frame (ABSF) patterns of 70subframes. At the beginning there are only 10 users per base station in the network,whereas the number of users is doubled abruptly after 230 subframes. . . . . . . . . 17

3.6 Example of network architecture with cluster-based Device to Device (D2D) relay (Dual Radio Opportunistic Networking for Energy Efficiency (DRONEE)). . . . . . . 18

3.7 Average per-user throughput and energy efficiency with DRONEE in network with4 clusters consisting of 2, 4, 6, and 8 users, respectively. . . . . . . . . . . . . . . . . 18

3.8 Schematic representation of overlay inband, underlay inband, and outband D2D forcellular scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.9 Comparison of D2D mode selection schemes in terms of the impact of user populationon the system performance with fully backlogged queues (achievable performance):aggregate cell throughput. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.10 Comparison of D2D mode selection schemes in terms of the impact of user populationon the system performance with fully backlogged queues (achievable performance):aggregate cell power. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.11 Example of wired resource distribution for dense cellular networks with access andbackhaul elements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.12 User throughput distribution under different user association policies (with 100 basestations, 80 users, backhaul capacity fixed to 500 Mbps, and Signal to Noise Ratio(SNR)min = -7 dB). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.13 User throughput distribution under different user association policies (with 100 basestations, 80 users, backhaul capacity fixed to 500 Mbps, and SNRmin = 5 dB). . . . 22

3.14 Backhaul network reconfiguration and FCPF architecture . . . . . . . . . . . . . . . 243.15 Refined Connectivity Management Sub-module architecture . . . . . . . . . . . . . . 243.16 LTE MAC/PHY SDN Architecture for CLC controlling eNBs. . . . . . . . . . . . . 263.17 LTE eICIC/ICIC Prototyping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.1 Media Lunch Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

CROWD Confidential v


vi Confidential CROWD


List of Project Partners

Name Acronym Country

Intecs S.p.A. (coordinator) INCS Italy

Alcatel-Lucent Bell Labs France ALBLF France

Fundacion IMDEA Networks IMDEA Spain

National Instruments SIG Germany

Universidad Carlos III de Madrid UC3M Spain

Universitaet Paderborn UPB Germany

AVEA Iletisim Hizmetleri AS AVEA Turkey

CROWD Confidential vii


viii Confidential CROWD


List of Acronyms

ABSF Almost Blank Sub-Frame

AP Access Point

API Application Programming Interface

BS Base Station

BWA Broadband Wireless Access

CABS CROWD Almost Blank Subframe technique

CLC CROWD Local Controller

CL(WRR) Cluster-based Weighted Round Robin

CL(MR) Cluster-based Max Rate

CRC CROWD Regional Controller

CROWD Connectivity management for eneRgy Optimised Wireless Dense networks

D2D Device to Device

DMM Distributed Mobility Management

DMM-GW Distributed Mobility Management Gateway

DRONEE Dual Radio Opportunistic Networking for Energy Efficiency

eICIC enhanced Inter-Cell Interference Coordination

eNB evolved NodeB

HeNB Home evolved NodeB

IP Internet Protocol

KPI Key Performance Indicator

JSON JavaScript Object Notation

LTE Long Term Evolution

MAC Medium Access Control

MN Mobile Node

MNO Mobile Network Operator

NB North-Bound

CROWD Confidential ix


ODL OpenDaylight

OF OpenFlow

PF Proportional Fair

PoA Point of Attachment

QoE Quality of Experience

RAN Radio Access Network

RAT Radio Access Technology

REST REpresentational State Transfer

RR Round Robin

RSRP Reference Signal Received Power

SB South-Bound

SDN Software Defined Network

SINR Signal to Interference plus Noise Ratio

SNR Signal to Noise Ratio

UE User Equipment

WLAN Wireless Local Area Network

x Confidential CROWD


Executive summary

The purpose of this document is to present the consolidated Connectivity management for eneRgyOptimised Wireless Dense networks (CROWD) architecture, detailing the key functionalities andthe modules implementing them. The architecture presented is the evolution of the preliminarydesign described in [1] and it takes into account the modifications needed due to MAC enhancement,connectivity management and dynamic reconfiguration studies performed.

A flexible architecture to accommodate the requirements of the extremely dense scenario, whichwill conform future access networks, has been the goal of the overall project and each piece of workcarried out in the project has been steered accordingly.

The concept of Software Defined Network (SDN) has been confirmed and definitely incorporatedin the architecture to make available the development of several intelligent modules able to providethe functionalities required in dense deployments.

CROWD’s goals about network cooperation, dynamic network configuration and improved ca-pacity tuning have been confirmed and studied in distinct work packages, under the perspective ofthe common SDN-based framework. The main goal of this document is to describe the conclusion,in architectural terms, of this study.

Starting from the description in [1] we have consolidated the proposal to adopt controllers intwo layers, with different roles: the CROWD Local Controller (CLC) and the CROWD RegionalController (CRC). We have identified OpenDayLight as the framework to exploit and integrate todevelop the CROWD controllers. The CLC is proposed for efficient utilization of resources via mech-anisms operating at short time scale, whereas the CRC optimises the radio resources availability interm of energy efficiency and capacity improvement on longer time scales. CROWD leverages theSDN to design the above mentioned controllers. Specifically, on the one hand, CROWD controllersexploit Southbound interfaces towards WiFi, LTE and backhaul links to i) collect real-time mea-surements, and ii) issue commands to the network devices. On the other hand, CROWD controllersexpose Northbound interfaces towards applications acting as policy decision points. Moreover toguarantee the Quality of Experience (QoE) to the users and improve the connectivity manage-ment, the Distributed Mobility Management Gateway (DMM-GW) is studied and integrated inthe CROWD SDN-based architecture.

This document explores the significant modification proposed for the revised CROWD architec-ture with respect to the preliminary architecture described in [1], according to the outcome of thetechnical work packages, and considering the preliminary results on testbed integration, which isone of the most important achievements expected in CROWD.

Finally the document explores the dissemination activities performed since August 2013, as anupdate on previous dissemination reports, including several journal and conference papers publishedin top venues, a number of demos and contributions to standardization activities.

CROWD Confidential 1


2 Confidential CROWD


Key contributions

The main technical contributions of this deliverable are as follows:

• Description of key challenging scenarios for the control of resource utilization and connectivityin very dense and heterogeneous wireless networks;

• Description of CROWD controllers and their SDN-style APIs for controlling LTE, WiFi andHetNets in general;

• Definition of the consolidated architecture for CROWD;

• Overview of the most relevant technical achievements in the design of network functionalitiesfor MAC enhancements, backhaul optimization, an connectivity management;

• Overview of testbed integration activities in view of the final demonstration of CROWDarchitectural features;

• Analysis of the first results on the real integrated testbed;

• Report on dissemination activities.

The key contributions to the research and standardisation communities resulting from the workperformed within the aforementioned activities are the following:

• Nine journal papers, specifically, one article in IEEE Transactions on Vehicular Technology,one in IEEE Surveys and Tutorials on Communications, one in ACM Computer Communica-tion Review, one in IEEE Communications Magazine, one in IEEE Wireless CommunicationsMagazine, two in Elsevier ComCom, one in Elsevier ComNet, and one in Elsevier Ad HocNetworks;

• Twenty-one conference papers, among which one paper in IEEE INFOCOM, one in IEEEWoWMoM, one in ACM MSWiM, one in IEEE/ACM COMSNETS, and one in IFIP WirelessDays;

• Nine tutorials and panels;

• Six posters and demos;

• Seven contributions to the standardisation of IEEE 802.1cf;

• Eight contributions to the standardisation of IEEE 802.21d;

• Three contributions to IEEE 802 EC Privacy Recommendation SG;

• Seven IETF contributions (internet drafts).



1 Introduction

CROWD had released a preliminary architecture for the control of access and backhaul resources ofvery dense wireless networks with strong support for connectivity management in [1]. Since then,the project has studied practical solutions for an SDN-like implementation of the architecture, andhas refined the basic mechanisms proposed to enhance the efficiency of the system Key PerformanceIndicators (KPIs), such as spectrum utilization, energy efficiency, user fairness, handover delay, etc.The architecture has been then refined with the feedback received from such activities.

Indeed, CROWD controllers have been implemented by adapting and integrating SDN-orientedplatforms, and control applications have been coded in accordance with the architectural function-alities identified with the study of solutions for MAC enhancements, backhaul optimization, andconnectivity management.

CROWD’s goals about network cooperation, dynamic network configuration and improved ca-pacity tuning have been studied in distinct work packages, under the perspective of the commonSDN-based framework. However, while the technical details that have driven the refinement of thearchitecture will be reported in other specific deliverables, here we present an overview of scenariosand functionalities identified in other work packages of the project to support high performancein very dense and heterogeneous wireless networks. In fact, the main goal of this document is todescribe the conclusion, in architectural terms, of these studies. Moreover, we only report on thearchitectural changes with respect to the previous release of the CROWD architecture, whereasmany details on the architecture and the control Application Programming Interfaces (APIs) re-mained unchanged and can be found in [1]. However, the document includes an overview of thetechnical reasons that have driven the refinement of the architecture, and a listing of the APIsdeveloped in the project is also reported in the appendix.

Starting from the description in [1] we have consolidated the proposal to adopt controllers intwo layers, with different roles: the CLC and the CRC. We have identified OpenDayLight as theframework to exploit and integrate to develop the CROWD controllers. The CLC is proposed forefficient utilization of resources via mechanisms operating at short time scale, whereas the CRCoptimises the radio resources availability in term of energy efficiency and capacity improvement onlonger time scales. CROWD leverages the SDN to design the above mentioned controllers. Specifi-cally, on the one hand, CROWD controllers exploit Southbound interfaces towards WiFi, LTE andbackhaul links to i) collect real-time measurements, and ii) issue commands to the network devices.On the other hand, CROWD controllers expose Northbound interfaces towards applications actingas policy decision points. Moreover to guarantee the QoE to the users and improve the connectiv-ity management, i.e. mobility, the concept of Distributed Mobility Management (DMM) is studiedtogether with its integration in the CROWD SDN-based architecture. Preliminary testbeds havebeen deployed to test the various techniques proposed in the project, starting with the feasibilityof the two-tiered control architecture proposed in CROWD.

A flexible architecture to accommodate the requirements of the extremely dense scenario, whichwill conform future access networks, has been the goal of the overall project and each piece ofwork carried out in the project has been steered accordingly, including dissemination. Indeed, thedocument explores the dissemination activities performed since the beginning of the project, whichincludes several journal and conference papers published in topi venues, a number of demos andcontributions to standardization activities.

The rest of the document is organized as follows. Chapter 2 presents the basic scenarios for which



the CROWD architecture has been designed. Chapter 3 highlights the refinements introduced inthe CROWD architecture after its preliminary release and the technical motivations that drove thearchitectural design. Chapter 4 reports on the dissemination activities carried out in the project.Finally, Chapter 5 summarizes and concludes the document.



2 Scenarios

In this chapter we describe the two references scenarios distilled from the objectives of the projectand reviewed according to the design of the architecture. The controllers, regional and local, worka fundamental role in the project. The need of coordination and management of a large numberof point of access, also considering the heterogeneity of the technologies, promote the controllersas the key element to reach the objectives of exploiting, supervising and managing heterogeneousdense networks. The controllers collect information coming from the lower layer and abstract themfor the higher layer (bottom-up). Here the applications can take decisions and the controllers areable to enforce them in the network (top-down).

2.1 Scenario 1: Self-optimising dense networks

Scenario 1 depicts a new generation mobile Broadband Wireless Access (BWA) network includinghot spots with a very high density of heterogeneous access points (e.g., WiFi Access Points (APs)and LTE Home evolved NodeBs (HeNBs)). Very high density means that the number of basestations has the same order of magnitude as the number of user terminals. Users equipped with asingle-radio terminal may connect to any of the reachable access networks, whose number is evenincreased if the mobile device is equipped with multiple radios of different technologies. For allthe access points under the control of a single operator, dedicated controllers optimise the RadioAccess Network (RAN) and backhaul network, to increase network capacity, QoE and achieveenergy efficiency.

In this scenario we pursue the following objectives:

1. Increase network capacity.

2. Achieve energy consumption proportional to traffic load.

The scenario is illustrated by means of Fig. 2.1. As can be seen, coverage is provided by severalbase stations that are heterogeneous in type (LTE vs. WiFi) and range (macro vs. pico vs. small).This creates the need for optimisation and coordination, which may happen either at a local scopeor with a broader view. Here the controllers carry out their task: they can take decision basedon the collected information or receive decisions from the application layer with the objective, forexample, to minimise the number of handover rather than to perform offloading of the cellularnetwork (D2D) or, even, minimise energy consumption playing on the user association.

Two examples of dynamic reconfiguration are reported in the right part of Fig. 2.1:

• optimised association of LTE User Equipments (UEs): A is detached from eNB1, which isclosest and with highest Signal to Interference plus Noise Ratio (SINR), and attached toeNB2 instead, possibly because the latter is less occupied than the former, or it enjoys morelightly loaded backhaul or any other reason that cannot be realised by eNB1 and A by meansof a stand-alone optimisation.

• LTE offloading to WiFi: B is detached from its serving LTE eNB3 and it becomes associatedto WiFi AP1, which requires inter-Radio Access Technology (RAT) hand-over. While thisoperation can be implemented easily within the Mobile Node (MN), as it is today, in this



LTE pic

o-cell

LTE macro-cell

Internet / EPC

Small cell (LTE HeNB, WiFi AP) A

eNB2

B

eNB1

eNB3AP1

CLC CLC CLC

CRC

Figure 2.1: Illustration of Scenario 1: network components involved in self-organization operation.

scenario the operation is assisted by the Mobile Network Operator (MNO), since it can takeinformed decisions aiming at global objectives.

An additional benefit of the proposed scenario is energy consumption reduction:

• the backhaul network elements are reconfigured dynamically, they optimise the network re-source usage and, indirectly, the power consumption because less hardware is required toachieve the same level of performance;

• base stations can be automatically switched on and off, which has a direct effect on powerconsumption.

The latter is also illustrated by means of the example in Fig. 2.2, which shows the same network asFig. 2.1 but with a lower load / density of users, e.g., because the snapshot is taken during nighttime. Thanks to a broad view optimisation process, the overall power required by the MNO tooperate the network can be reduced significantly.

2.2 Scenario 2: Optimised mobility in dense radio access networks

In Scenario 2 we consider the same mobile BWA network with very high density of access pointsas in Scenario 1, but we focus on mobile nodes exploiting network-terminal cooperation to selectthe most appropriate technology and access point. We consider a urban area with high densityof Point of Attachments (PoAs) and heterogeneous technologies (LTE and WiFi). Mobile nodes



LTE pic

o-cell

LTE macro-cell

Small cell (LTE HeNB, WiFi AP)

Internet / EPC

CLC CLC CLC

CRC

Figure 2.2: Illustration of Scenario 1 in case of low offered load.

roam around and network-terminal cooperation is required in order to optimise the connection.Cooperation among mobile users and among multiple mobile network accesses is also considered.Inter-RAT and LTE to/from WiFi mobility may occur for multi-homed mobile nodes.

In this scenario we pursue the following objectives:

1. Optimise technology selection in terms of a telecom operator objective function, e.g., maximisecapacity or minimise energy efficiency, while keeping a satisfying user’s QoE.

2. Hand-over of user’s terminal traffic from the cellular to the WiFi access networks (offload)management.

3. Opportunistic network selection.

Again the controllers help in performing these objectives allowing the collaboration betweendifferent technologies, as from the controller perspective the technology is abstracted according tothe SDN paradigm.

The scenario is illustrated by means of Fig. 2.3. As the density of base stations increases, so doesthe frequency of hand-overs, until the deployment of new PoAs is not beneficial anymore, but ratherit causes inconvenience due to high signalling overhead. In CROWD we address this challenge by:i) defining new schemes for intra-RAT local mobility management, which are scalable by design andcan be seamlessly integrated within the existing LTE and Wireless Local Area Network (WLAN)networks; and, ii) developing new telecom operator assisted mechanisms for inter-RAT hand-over,i.e., offloading. Therefore, in this scenario the MN is transparently associated to the best PoA,



Internet

EPC

Information repositories

LTE eNB WiFi APMulti-radioterminal

eNB1

AP1

AP2

AP3

eNB2

CLCCLC

CRC

Figure 2.3: Illustration of Scenario 2: network components involved in connectivity managementoperation with mobility optimisation.

from the point of view of the MNO, which has the information and incentive to perform network-wide optimisation in terms of, e.g., capacity or energy consumption. Furthermore, the MN roamsthrough the different networks without losing its active connections, so that make before breakhand-over is realised also in the case of association to a network with a different wireless accesstechnology.



3 Architecture

In this chapter we describe the consolidate architecture designed in CROWD as conclusion of theTask 1.1 activities.

We provide an overview of the proposed architecture in Section 3.1, with a focus on the interfacesof the proposed sub-systems in Section 3.2. Section 3.3 describes the CROWD controllers, whereasa more detailed description of the architecture functionalities is then reported in Section 3.4 forMAC-layer enhancements (WP2), in Section 3.5 for the backhaul and long-term radio configuration(WP3), and in Section 3.6 for connectivity management (WP4). Finally the feedback on the realtest-bed about the real feasibility of the mechanisms developed is shown in Section 3.7

3.1 Consolidated architecture

As introduced in D1.1, LTE and WiFi are the technologies expected to have the highest penetrationin mass deployment in the future networks. In order to manage different technologies just to exploitdiversity as a resource rather than a limitation,intellingent elements that have a complete view ofthe network deployment is needed. Taking the decision to force handover from one cell to anotheror to perfom the off-loading of a cell, has to be managed to to optimise the resource allocation.We proposed a two-tiers architecture, with two-tiers SDN controllers, the CLC and the CRC . Thetwo controllers work in different timescales,taking the decision in short or medium-long timescale,respectively. The controller is one of the key elements of CROWD SDN-based architecture. Weidentified in OpenDaylight (ODL) the framework on which developing most of functionality of boththe CLC and CRC, while certain mobility related functionality is implemented using the Ryu SDNcontroller (for simplicity). A detailed description of the evolution of ODL to CROWD Controlleris provided in Section 3.3

As introduced in [1], to tackle the challenges of the scenarios described in Chapter 2, we identifiedthe district which consist in base station, LTE and WiFi AP, and interconnecting backhaul linksreconfigurable via OpenFlow (OF) where the CLC works and enforce the decision in a short-timescale, and the region as a logical area including several district and where the CRC operatesin medium-long timescale. The optimisation in the network is performed by application that run ontop of the controllers. They are connected with CLC or CRC by the North-Bound (NB) interfaces.Some of the NB interfaces are shared among the applications, as they are more general than others,some other are specifically built and tailored on the application. In the figure Fig. 3.1 the South-Bound (SB) are also shown as interconnection between the controllers and the new and old networkelements.

In the following section the interfaces between subsystems are illustrated.

3.2 Interfaces between sub-systems

In D2.2, D3.2 and D4.2 an exhaustive description of the NB and SB interfaces has been provided.For completeness the identical list is reported in Appendix. Moreover, according to the implemen-tation, some changes occurred. Hereby we provide the list of major modification in NB and SBinterfaces.



X2

UuIEEE 802.11

HeNB eNB WiFi AP

IP

New interface Existing interface

WiFi District

X2

LTE District

`

ANDSF IEEE 802.21 MIIS ALTO Server

Information repositories

Operator services(IMS, HSS, AAA)

Northbound APISouthbound API

IP


NB-CLC NB-DMM NB-WiFi

SB-WiFi SB-OF

Connection

IF 1 IF2

X2

NB-LTE NB-DMM NB-CLC

SB-OFSB-LTE

Internet

CROWD element


Region

Backhaul

Backhaul

LTE accessselection

CLC

DMM GW

CRC

DMM session store

DMM GW

S1-U

S1-U

S1-U

S1-U S1

-U

S1-US1

-U

NB-CRC

Dynamic Backhaul Reconfiguration

SB-OF SB-OPSB-CLC

Manager

\\

ABSF WLAN Optimisation

D2DOffloading

D2DOffloading

Power Cycling

Figure 3.1: CROWD network architecture.

NorthBound Interfaces

Regarding the NB interfaces syntax modification occur.

Firstly we removed from the set of NB interfaces all the interfaces already existing in ODL,related to the switches and flows information:

• removed /directory/backhaul. It is replaced by SwitchManager ODL standard interface

• removed /directory/traffic flows. It is replaced by FlowProgrammer ODL standard interface

• removed /network elements/flows. It is replaced by FlowProgrammer ODL standard interface

Some interfaces has been removed as the information they carried on, have been included in thefields of other new or existing interfaces:

• replaced /network element/UE/{id}/Handover (PUT)with /mobile device/handover/{host ip}/{src id}/{dest id}.

• replaced /network element/UE/{id}/Handover (GET)with the field signal in /mobile device/{host ip}/channel state.

Moreover, additional interfaces are developped to manage WiFi statistics information:



• /network element/WiFi/{id}/bw stats?date={value}&duration={value}

• /network element/WiFi/bw stats

• /network element/WiFi/{id}/ld stats?date={value}&duration={value}

• /network element/WiFi/ld stats

SouthBound Interfaces

The complete list of the SB interfaces of the CLC are reported in Appendix B.1. The majormodifications are related to additional interfaces that occurred as

• ABSF message

• MCS message

• State message

• Change State message

3.3 The Controller

During the lifetime of the project several controllers have been studied and a deep analysis hasbeen carried out to understand how we could use and improve the existing platforms to be fittedin CROWD. We looked at controllers like Pox and Nox, Beacon, Ryu, Floodlight and FlowVisorand at frameworks like Opendaylight and OpenNaas. We selected Opendaylight for three reasons:

• software architecture: the software components are close to the architectural functional blocksproposed in CROWD;

• market impact: from the market impact point of view, we believed that Opendaylight frame-work paves the way of SDN paradigm introduction in the main stakeholders business plan;

• custom implementation: SDN is not limited to any single protocol. This allows to improveand customize the framework.

The structure of OpenDayLight (ODL) platform fits the main elements of CLC very well. Firstof all it offers a layered architecture in which northbound interfaces enable the applications, whichrun on top of the controller, to interact with the network elements. Secondly, the Base NetworkService Functions fulfill the requirements for Discovery and Collector functionalities: all the networkdevices are discovered, collected and monitored by this part of the framework. Thirdly, the ServiceAbstraction Layer contains a sort of database where all the information that comes from thelower layer are stored. Finally, southbound interfaces, e.g. OpenFlow, allow to interface differentnetwork devices to the same framework. ODL is based on OpenFlow, although it is extensible tosupport other SDN open standards (I2RS, VxLAN, etc.). We believe that developing the CROWDcontroller based on ODL allows the project to be aligned with the mainstream design of futurenetworks. Indeed, the SDN approach has had great success and has attracted the attention ofservice providers and stakeholders interested to enable new services and capabilities and extendthe life of existing infrastructures.

We begun with the development of CROWD components by using the Hydrogen version of ODLreleased in February 2014, and have later stepped to the latest Helium release. Therefore, bothCLC and CRC are ODL-based. The details of the modification and the behaviour of the two typesof CROWD controllers are reported in this section.



Figure 3.2: CLC: OpenDayLight extention

CLC

The CLC aims to manage the network devices that populate the district where the controlleris located. The scenario we are referring to is a heterogeneous network, where LTE and WiFItechnologies coexist and share backhaul switches. Due to the multitude of technologies we plannedto adopt an existing OpenFlow plugin to interface Openflow switches with controllers, and developadditional plugins for LTE and WiFi points of access.

In Fig. 3.3 the modules developed in ODL for LTE technology are shown. The same structure isused for the WiFi too. Northbound and Southbound interfaces are described in Section 3.2. Herethe Core service layer modules are shown:

• LTE manager: it manages the information coming from the Topology, Inventory an UEtracker.

• LteManager Interface: its role is to translate the NB interface and use the module of the core.

• IDataPacket Service: to translate the information from/to the SB interfaces and send/receivethe command to/from the core.

• Inventory: this module works together with SAL of ODL and extract the information needed.

• Topology: this module works together with Topology Manager of ODL, in the Base networkService Function, and takes the information needed.



Lte SBPlugin

NB REST API

OtherPlugins

CoreService

LteServices

Lte REST API

LTENorthboundManager

LteManager

IPluginInDataPacketService

IDataPacketService

Inventory

Topology

UeTracker

NetService

NetProtocolImpl

LteManager Interface

Figure 3.3: LTE modules in the Local Controller

• UE tracker: this module works together with Host Tracker of ODL and takes the informationrelated to the UE connected to LTE.

CRC

Based on our two-tier architecture, the main tasks of the CRC are to manage the CLCs anddistricts inside its region and to give the applications, running on top of it, access on a regionalscope. Consistently with the operation of CLCs, also the CRC uses an existing OpenFlow pluginof ODL to control OpenFlow-enabled devices on a regional scope. Besides, to allow access to theAPIs of the CLCs inside the region and to control the CLC instances, we have developed a new setof modules for ODL called CLCManager.

The modules of the CLCManager are shown in Fig. 3.4 and the Northbound and Southboundinterfaces are described in Section 3.2:

• CLCManager Interface: translates between the NB interface and the core modules.

• CLCManager Service: handles commands and function calls to the core plugin.

• CLCManager Plugin: manages the CLC inventory

• Inventory: stores the actual CLC inventory.

• CLCConnector: handles the communication with the CLCs



CRCPlugins

NB REST API

OtherPlugins

CoreService

CRCServices

CRC REST API

CLCManagerService

Inventory

CLCManager Interface

CLCManagerPlugin

CLCConnector

Figure 3.4: CRC: CROWD CLCManager modules

3.4 Applications and MAC-layer enhancements

The CROWD architecture and its controllers can be leveraged to push network performance towardsthe 1000× improvement in capacity and density targeted for 5G networks. Moreover, the modularand flexible controllers proposed in CROWD can be used to build new services in short time andto quickly reconfigure the network in negligible time, as per 5G objectives.

In general, CROWD offers algorithms and software for enhanced wireless mechanisms, which canbe used not only to benchmark novel 5G proposals, but also to derive novel density-aware network-ing approaches. In particular, from an architectural point of view, CROWD proposes a few keynetwork control functions to enhance the efficiency of wireless MAC mechanisms: (i) the distributedcontrol of the ABSF mechanism for enhanced Inter-Cell Interference Coordination (eICIC), (ii) theoptimization of D2D communications for cellular offloading, opportunistic spectrum utilization, andscheduled relay of traffic for devices with poor channel quality, and (iii) the optimization of user(re-)association mechanisms in presence of multiple access choices, including when multiple wirelesstechnologies are available to connect to the network (i.e., with vertical handover).

As concerns eICIC, efficient and lightweight control of ABSF has been designed by using gametheory and optimization techniques. We started with the design of a centralized [2] approach,and then we have studied the possibility to exploit the cooperation between base stations to makeour approach more practical and really implementable in commercial networks. Specifically, wepropose to delegate each base station to decide which set of users have to be scheduled in eachsubframe. Based on those decisions, each base station builds its ABSF pattern which is announcedto the neighbor in a round robin fashion. However, we impose an upper limit to the number ofsubframes that each base station can use in a given time horizon. Such limit is dynamically adjustedby the central controller, i.e., CLC, using an additive increase multiplicative decrease mechanism,which has been largely proved to be optimal for such problems. The controller properly tracksdynamic changes occurred in the offered load experienced in the network. We call our scheme



0 100 200 300 4000

5

10

15

20

ABSF patterns [#]Agg

rega

te m

inim

um u

ser

rate

η [k

bits

/TT

I]

Optimum CABS No ICIC

Figure 3.5: Dynamic behaviour of CABS in a network with 7 base stations and ABSF patterns of70 subframes. At the beginning there are only 10 users per base station in the network,whereas the number of users is doubled abruptly after 230 subframes.

CABS. Fig. 3.5 shows an example of dynamic adaptation of ABSF patterns as the number ofusers per base station is increased from 10 to 20 in a network with 7 base stations, Interestingly,CABS quickly reacts to network changes and achieves much better results than legacy solutions(“No ICIC” in the figure). CABS offers a good trade-off between complexity and performance.In fact, the “Optimum” reported in the figure, which represents the best performance achievablewith ABSF, is not far from what achieved with CABS, although it imposes an exponentially highercomplexity and requires a much higher signaling overhead. Interestingly, CABS only requires toannounce ABSF patterns in form of short bitstreams, while the computation of the optimal solutionrequires a perfect knowledge of the channel quality reported by each user in the network, whichmust be transferred to a central controller. Note that in CABS, the role of the controller onlyconsists in dynamically regulating the maximum number of subframes that each base station mightschedule.

As concerns the optimization of D2D strategies, CROWD has developed architectural solutionsfor enhancing both WiFi and LTE-based systems by means of opportunistic and smart relay mech-anisms [3, 4].

Fig. 3.6 depicts a scenario in which the users are organized in clusters and the cellular traffic isrelayed by means of cluster heads using WiFi Direct within the cluster. For such scenario, Fig. 3.7shows an example of throughput and energy efficiency gain that can be achieved with D2D relayof traffic in LTE networks. The relay scheme developed in CROWD is called DRONEE, and itis based on bridging LTE and WiFi Direct on mobile user’s devices and using clusters of usersfor scheduling operations [5, 6]. The figure shows that users receive the lowest throughput underRound Robin (RR) because they are scheduled irrespective of their channel quality. The Propor-tional Fair (PF) scheduler has much better performance than RR, but is significantly outperformedby CROWD schemes Cluster-based Weighted Round Robin (CL(WRR)) and Cluster-based Max



Figure 3.6: Example of network architecture with cluster-based D2D relay (DRONEE).

1

1.5

2

2.5

3

3.5

4

4.5

RR PF CL(WRR)CL(MR) RR PF CL(WRR)CL(MR)

Thro

ughput [M

bps]

Effic

iency [M

b/J

]

Throughput Efficiency

EfficiencyThroughput

Figure 3.7: Average per-user throughput and energy efficiency with DRONEE in network with 4clusters consisting of 2, 4, 6, and 8 users, respectively.

Rate (CL(MR)), which user RR and Max Rate allocate resources to clusters rather than to users.CROWD D2D schemes are advantageous also in terms of energy efficiency, although they requirethe use of two wireless interfaces per mobile node (LTE and WiFi).

Similarly, opportunistic relay has been also proposed to solve the performance anomaly problemof 802.11 networks, that is exacerbated in dense deployments as soon as at least one user per accesspoints receive significant interference. In this case, WiFi Direct is used to offload WiFi accesspoints. In particular, CROWD solution for this problem leverages on the flexibility of wirelesstopologies and relay scheduling, i.e., on how the relays split time between downstream nodes theyrelay for and upstream flows to access points. Our results, illustrated in [4], show significant gainsin terms of network throughput performance (more than doubling it) and power consumption (upto 75% reduction)

Moreover, we have studied alternative technologies to implement opportunistic relay schemes,



Unlicensed!

band!

Licensed!

band!

Underlay Overlay Outband

Fre

quen

cy!

Cellular!D2D!Cellular!

D2D!Cellular!

D2D!

Figure 3.8: Schematic representation of overlay inband, underlay inband, and outband D2D forcellular scenarios.

and in particular we have explored how to optimize D2D mode selection [7]. Indeed, the significanceof the potential D2D gain had led to proposals in which a part of the cellular resources is dedicatedonly to D2D communications (i.e., inband overlay D2D mode), while the scarcity and the highprice of cellular spectrum motivated some researchers to explore D2D communications over theunlicensed band (i.e., outband D2D mode). These D2D modes are schematically illustrated inFig. 3.8. In CROWD, we propose an adaptive D2D mode selection by eliminating the existing biastowards specific D2D mode or operating band. In particular, based on the practical implicationsof each D2D mode based on the latest standard releases of LTE-A and WiFi-Direct, we findthat there is no superior D2D mode and the potential of each mode is highly scenario/use-casedependent. Moreover, based on our analytical study, we proposed an innovative multi-mode multi-band setup, which accounts for both achieved throughput and energy costs. We call such a novelapproach Floating Band D2D, because D2D transmissions can occur on either inband or outbandmodes. However, since the optimization problem in Floating Band D2D is NP-hard, we proposecomputationally feasible 3 heuristics (namely, Social, Greedy and Ranked). For the details onanalysis and optimization we refer the reader to [7]; here we just report an example of potentialgain due to Floating Band D2D.

Figs. 3.9 and 3.10 illustrate the impact of the user population size N on achievable systemperformance when 30% of radio resources can be dedicated to inband D2D in a single cell LTE-Ascenario. The results reported in the figure pertain to a reference cell, i.e., an average behaviorof a macro-cell surrounded by other macro-cells. Error bars in the results are the 95% confidenceintervals. Cellular users are scheduled according to the PF algorithm.

In addition to our heuristics, we evaluate three benchmark schemes, namely, Forced-LTE, Forced-WiFi,and Optimal. In Forced-LTE, D2D users are forced to use legacy cellular communications (i.e.,mode 0). In Forced-WiFi, D2D users are forced to communicate over WiFi (i.e., mode 3). Optimalresults are based on the exact solution to the mode selection problem. The benchmark schemesprovide the means to compare our proposals with the legacy cellular system, to measure the gaindue to extra WiFi bandwidth, and to see how far the heuristics are from the optimal decision.

We can observe the achievable throughput in Fig. 3.9. The aggregate throughput has a negligiblechange with N under Forced-LTE, because the density of users does not change the distributionof channel qualities in the cell, and therefore the average aggregated throughput. The throughputof the rest of schemes increases with N because there are probabilistically more D2D pairs in adenser cell, hence D2D throughput is higher. In Forced-WiFi, the throughput grows slowly dueto the contention-based nature of WiFi, in which the MAC overhead increases with the numberof contending users. Since some of the outband D2D pairs do not interfere with each other (i.e.,they are more distant than 150 m), the aggregate throughput of Forced-WiFi in our experimentsreaches up to 98 Mbps. More importantly, not only the simple proposed heuristics greatly out-perform Forced-LTE and Forced-WiFi, but they also perform very close to Optimal (due to the



0

50

100

150

200

250

300

350

5 10 20 30 40 50 60 70 80 90 100

Thro

ughput [M

bps]

Number of users per cell (N)

Forced-LTEForced-WiFi

SocialGreedy

RankedOptimal

Figure 3.9: Comparison of D2D mode selection schemes in terms of the impact of user popula-tion on the system performance with fully backlogged queues (achievable performance):aggregate cell throughput.

0

20

40

60

80

100

120

140

160

5 10 20 30 40 50 60 70 80 90 100

Pow

er

[W]

Number of users per cell (N)

Forced-LTEForced-WiFi

SocialGreedy

RankedOptimal

Figure 3.10: Comparison of D2D mode selection schemes in terms of the impact of user populationon the system performance with fully backlogged queues (achievable performance):aggregate cell power.

computational complexity of such an ideal scheme, we only have the results up to 80 users). Interms of energy cost, the aggregate cell power increases with N , as shown in Fig. 3.10, mainlydue to the baseline energy consumption of wireless interfaces. Forced-WiFi has higher energyconsumption because outband users have to maintain two active wireless interfaces instead of one.



Figure 3.11: Example of wired resource distribution for dense cellular networks with access andbackhaul elements.

Finally, we comment on the architectural advantages due to wireless access selection and verticalhandover in very dense cellular networks. Indeed, access selection and vertical handover techniquesare proposed in CROWD to target multiple objectives, among which high throughput, user fairness,energy efficiency and load balancing [8]. Moreover, since the network to optimize not only consists ofmobile devices and base stations, but it also includes switches and routers in the backhaul, we havestudied how to distribute the capacity throughout the network elements when user (re-)associationin the access network can take into consideration such a capacity distribution. An example networkfollowing our suggested architecture is illustrated in the reference diagram of Fig.3.11. For suchkind of networks, we designed (re-)association policies that aim at either maximizing capacity orminimizing network-wide energy consumption.

In Figs. 3.12 and 3.13, we show the distribution of the user throughput (range of throughput vs.percentage of users) when using a fixed 500 Mbps capacity for the backhaul. The performancesare shown for 80 users and 100 base stations uniformly distributed over a 200m×100m area. In-terestingly, our study shows that little throughput has to be paid for achieving minimal energyconsumption. Moreover, we also found that, with our optimization, it is enough to guarantee SNRvalues as low as -7 dB to save considerable energy while, at the same time, obtaining a throughputdistribution pretty close to the one we would achieve by means of a pure signal-strength-baseduser association. Therefore, we conclude that the per-user perception of the service received is justslightly affected by an association policy that minimizes network energy consumption.

In conclusion, we have designed and validated a few key architectural features that make possibleto use network access and backhaul resources efficiently under several specific metrics (e.g., energyefficiency, throughput, spectral efficiency, fairness). With these features, which require either cen-tralized or distributed control, CROWD proposes a MAC/resource allocation framework ready totackle the challenges of future super-dense wireless networks.



0

20

40

60

80

100

[ 0.00 - 9.33 ]

[ 9.33 - 18.66 ]

[ 18.66 - 28.00 ]

[ 28.00 - 37.33 ]

[ 37.33 - 46.66 ]

[ 46.66 - 55.99 ]

[ 55.99 - 65.32 ]

[ 65.32 - 74.66 ]

[ 74.66 - 83.99 ]

[ 83.99 - 93.32 ]

% o

f U

Es

Ranges Mb/s

Max-CapacityMin-Energy

Figure 3.12: User throughput distribution under different user association policies (with 100 basestations, 80 users, backhaul capacity fixed to 500 Mbps, and SNRmin = -7 dB).

0

20

40

60

80

100

[ 0.00 - 9.33 ]

[ 9.33 - 18.66 ]

[ 18.66 - 28.00 ]

[ 28.00 - 37.33 ]

[ 37.33 - 46.66 ]

[ 46.66 - 55.99 ]

[ 55.99 - 65.32 ]

[ 65.32 - 74.66 ]

[ 74.66 - 83.99 ]

[ 83.99 - 93.32 ]

% o

f U

Es

Ranges Mb/s

Max-CapacityMin-Energy

Figure 3.13: User throughput distribution under different user association policies (with 100 basestations, 80 users, backhaul capacity fixed to 500 Mbps, and SNRmin = 5 dB).

3.5 Backhaul and long-term radio configuration

The backhaul and long-term radio configuration sub-system extends the short-term mechanisms ofWP2 with the medium-long timescale applications and algorithms running on top of the CRC. Askey contributions with impact to the overall architecture, WP3 offers (i) the controller life-cyclemanagement for providing a feasible infrastructure for the long-term radio configuration, (ii) the



backhaul management for operating the backhaul network in a traffic proportional manner and(ii) the power cycling of network elements. All aspects have been studied extensively during thelifetime of the project and have hence massively involved since D1.1.

The mechanisms for short-term decisions and medium-long timescale optimization within theproject have to be run in an execution environment that provides the necessary decision inputand fast access to the network equipment to enforce these decisions. Setting up and maintainingthis execution environment is the goal of the controller life-cycle management. While CROWDinvolves many kinds of different decision and optimization mechanisms, we have found that allof them can be generalized as data flows with special needs in terms of latency, data rate andespecially data processing at a controller. Hence, we have focused our research in this task on flowprocessing-aware controller placement and reassignment and we have developed a fast heuristicflow processing-aware controller placement framework (FCPF) based on the problem statementdescribed in D3.2. In particular, FCPF places the CROWD controller architecture in a networkwith given data flows and meets all requirements on latency, data rate and processing capacity thatare needed for both control applications and data flow processing. Further, FCPF is capable ofperforming flexible reassignment in reaction to changes of network load and/or network topology.More details about the theory and the implementation of this functionality will be reported inD3.3.

The backhaul management has already been identified as a long-term mechanism in D1.1 andthus benefits from the two-tier architecture in CROWD. The traffic proportional operation of thebackhaul network requires information on a regional scope, hence the backhaul reconfiguration ap-plication has to run on the CRC. This application requires an API to dynamically assign resourcesin the backhaul network for each CROWD district and CLC. This is especially facilitated by thechoice of OpenDaylight (ODL) as the target platform for the CRC. As described in Section 3.2,ODL already provides a full featured API for programming flows in the backhaul network. Italso provides APIs to obtain the backhaul network topology and to monitor changes to it. Theonly missing API was an API for the placement and management of CLCs. This API has beenintegrated into the CRC as described in Section 3.3

As shown in Fig. 3.14, both the backhaul network reconfiguration and the FCPF applicationsare running on the CRC.

The power cycling of network elements requires the joint operation of the CRC and the CLCs.Efficient power cycling that does not disrupt the operation of the network is only feasible if it isbased on information from a regional scope (i.e. available on the CRC). But the actual executionof the power cycling is specific to the network elements in each district and thus has to be executedon the CLC of each district. Thus the actual API for power cycling is implemented as part of theCLC as described in Section 3.2. But a power cycling application is able to run on the CRC andaccess the power cycling API of each CLC in the region of the CRC via the CLCManager API(Section 3.3) of the CRC.

3.6 Connectivity management

The architecture of the Connectivity Management sub-system has evolved during the lifetime ofthe project due to the different results obtained during the research phase. These results haveyield to changes in the design of the sub-system. This section is devoted to the analysis of theevolution of the WP4 architecture. It takes as basis the architecture first reported in D1.1 andupdated in D4.1 and D4.2. Fig. 3.15 shows the revised view of the Connectivity Managementsub-module where the main interfaces connecting the different modules have been included in thefigure. The definition of the interfaces have not changed from D4.2, hence they are not included inthis description to improve the clarity of the explanation. The WP4 architecture has suffer three



Figure 3.14: Backhaul network reconfiguration and FCPF architecture

Backhaul Elements

Data Path Anchoring Function

CRC

IP Mobility Network Engine

Tec

hn

olo

gy

Sp

ecif

ic

Mo

bil

ity

En

gin

e

MN

Access Selection

Module

Connection

Manager

IP Mobility

Terminal Engine

CLC

DMM Control Plane

DMM Control Plane

DMM GW

Data Path Anchoring Function

Intra-district Mobility

Control Plane

Information Repositories

ANDSF ALTO ServerIEEE 802.21 MIIS

Information Aggregation

UE_XXX_InfCRC_UE

CLC_GW_DMM

CRC_Inf_rep

CL

C_C

RC

_M

M

CL

C_D

P

Figure 3.15: Refined Connectivity Management Sub-module architecture



main changes according to the initial architecture explained in D1.1. These changes are explainedin the following:

• The aggregation of information regarding the access network is performed at the CLC of eachdistrict, but the CLC does not longer interact directly with the Information Repositories,but it indirectly interact with them through the CRC. The reason for this change is that wedetected that there will be quite complex to provide concurrent interaction between severalCLCs and the Information Repositories.

• The Technology Specific Mobility Engine does not longer deal exclusively with the recon-figuration of the data path elements within a district, but also considers the sharing of theRAN and the configuration of more complex setups, such as the use of VLANs or OpenFlowconfigured tunnels.

• The functionality of the Connection Manager in the UE has suffered a change in focus. Wewere focusing on providing mechanisms for the detection of the characteristics of the flowsin the terminal, to provide the required level of mobility considering the flow nature. Wehave realized this is not a feasible approach to the problem, hence we are now focusing onapplying the flow analysis to the design of the network to understand the best deployment inreal networks so they benefit from DMM.

As explained above, the first change corresponds to the modification of the interface betweenthe CLC and the Information Repositories. Originally, the CLC was aggregating the informationcoming from the different measurement probes in the RAN, and then providing this information tobe stored in the repositories. This information could be used by the CRC for global optimizationalgorithms, the UE to optimize mobility decisions and by the CLC for local optimizations. Afterstudying different use cases and scenarios, we have concluded that the real usage of the InformationRepository is coming from the CRC to optimize the overall performance of the network, while theUE will use district local information which will be available at the CLC. Hence we have simplifiedthe overall architecture so the CRC is the one keeping the information at the databases correct andthe CLCs provide updates on each district directly to the CRC. The key functionality providedby the Information Repository to the UE was related to access selection. After studying themechanisms available to influence the terminal attachment, we have concluded that for WiFi weshould use ANQP while for celular districts we should use ANDSF/MIIS. In this way, the CLC willinfluence both databases so they indicate the UE the information desired by the network. This sidone by direct interaction with the CLC.

The second of the architectural changes corresponds to the Technology Specific Mobility Engine.This module is in charge of modifying the data path followed by a certain flow, so it arrives at thecurrent point of attachment of the terminal, updating it when the user handovers to a different pointof attachment. Originally, this module was thought to be implemented by direct modification of theswitching entries on all data path elements in the path between the terminal and the DMM GW(through the use of OpenFlow). After implementing and analysing this proposal, we have realizedthat it is better in terms of path setup delay, if the district is pre-configured with a set of VLANsor OpenFlow configured paths between points of attachment and DMM GWs. In this way, thesetup of a path is just to add a user to one of these paths, greatly reducing the time required toget the packets from a user flowing. Hence, this new requirement increments the complexity ofthe interface between the data path elements and the CLC, which now must implement this newlogic. In addition to this, we have also work in the design of a solution to force the user handoverbetween different points of attachment in a given district. This is needed to optimize the network,for example enabling a new point of attachment when the count of users passes a certain thresholdand moving some users to it. We have successfully implemented a prototype of this functionalitybased on IEEE 802.11v, which will be reported in D4.3.



Finally, the third of the changes to the architecture regards to the Connection Manager atthe terminal. Originally this module was designed in such a way that it will be able to measurecharacteristics of the flows or applications and provide this view to the network, so the best mobilityprotocol could be provided. WP4 partners, UC3M and AVEA, have been working actively towardsachieving this feature but we have concluded it is not feasible. Major operating systems and serviceproviders make use massively of CDNs such as Akamai, hence making impossible the detection ofthe flows just by IP or transport protocol ports. Since a deep analysis of the traffic characteristicsof the users has been performed, we are now focusing on applying the findings to understand if thedeployment of DMM solutions in current cellular network deployments make sense and how thesenetworks can be optimized (e.g., cells associated to an SGSN) to take benefit of DMM approaches.

3.7 Feedback on real test-bed

CROWD Project imposes wide variety of requirements on testbed and experimentation on all layersof the protocol Stack. The complexity of integration with SDN Controller, for example CLC alsorequires the testbed platform be based on open components with direct access to different layers,especially PHY/MAC layers to study the impact of SDN-based algorithms for cross layer design.We took into account different CROWD Applications, for example (ABSF, D2D, mobility reconfig-uration, etc) as a basis to design a testbed architecture that could demonstrate the performance ofsubset of CROWD applications within a lab setting in a dense environment. In Fig. 3.16, we illus-

Remote Host

MAC

RLC

PDCP

eNB

IP

UDP

GTP

IP

UDP

GTP

IP

SGW/PGW

LV PHY

LV Real-Time

FF L1-L2 API

MAC

RLC

PDCP

IP

TCP/UDP

APP

LV PHY

LV Real-Time

UE

FF L1-L2 API

Femto Forum (FF)/Small Cell Forum L1-L2 API

Interface to SDN Controller

SDN Controller (CLC)

Algorithms to manage interference

(for ex. ABSF)

Figure 3.16: LTE MAC/PHY SDN Architecture for CLC controlling eNBs.



trate in more detail how proposed SDN controllers (i.e., CLCs, in this specific case) interact withLTE MAC/PHY interface of an LTE eNB to collect statistics regarding link performance in termsof throughput, channel state information, etc., while at the same time proposing eNB behavioralchanges via a CROWD Local controller (CLC)-MAC Scheduler API. In general, a CLC controllerinteracts with the MAC layer of eNBs to influence parameters related to scheduling; however, thisinterface can be easily extended to higher layer of protocol stack. We have chosen to implementhigh throughput LTE Physical layer in LabVIEW based PXI platform to meet high throughput/lowlatency requirements. However, we chose to use NS-3 LTE LENA stack [9] for implementing higherlayers. NS-3 is widely used system simulator in research community to conduct simulations. Wechose to leverage NS-3 LTE stack to reduce time from simulations to prototyping. We believe suchan approach can produce significant results in much shorter time frame for prototyping ideas fornext generation deployments. During the initial phase of the project, we explored several differentcommercial platforms such as Mymo wireless [10], BEECube [11], Amarisoft [12] and Nutaq [13].However, we found these platforms to be either too expensive for a research project or in somecases there was a lack of open source code, which prohibited the experimentation requirements forCROWD project.

On prominent application that we plan to demonstrate within CROWD project is CROWDAlmost Blank Sub Frame (CABS), which is described above. CABS refers to different base-stationstaking turns in muting some of their subframe to reduce inter-cell interference. Fig. 3.17 showsthe architecture of CROWD testbed to prototype CABS algorithm in an indoor environment. Itshould be noted that USRP-RIO/PXI system runs PHY layer components of LTE on LabVIEWFPGA, whereas a linux PC runs NS-3 LTE LENA stack on Ubuntu interfacing with LabVIEWFPGA with a UDP interface. CLC is implemented in another Linux machine and interfaces withseveral eNBs on a local Area Network (LAN) connection.

USRP-RIO USRP-RIO

USRP-RIO

USRP-RIO USRP-RIO USRP-RIO USRP-RIO

CROWD Local Controller (CLC)

Prototype ICIC/eICIC Algorithms: Almost Blank Sub-Frame (ABSF) Multi-user/Multi-cell Scheduling Rate adaptation

All experiments conducted within lab

setting

USRP-RIO USRP-RIO

LTE eNB

LTE UE

LTE UE

LTE UE

LTE UE

LTE UE

LTE UE

LTE eNB

LTE eNB

Figure 3.17: LTE eICIC/ICIC Prototyping



4 Dissemination activities

Dissemination of the results related to this deliverable has been carried out via the followingactivities:

4.1 Journals and Magazines

• A solution for transparent mobility with route optimization in the IP multimedia subsystem, I.Vidal, J. Garcia-Reinoso, I. Soto, A. de la Oliva Accepted for publication in Elsevier ComputerCommunications, September 2013.

• DRONEE: Dual-Radio Opportunistic Networking for Energy Efficiency Computer Commu-nications, A. Asadi and V. Mancuso. Elsevier Computer Communications Journal, February2014.

• On the implementation, deployment and evaluation of a networking protocol for VANETs:the VARON case, M. I. Sanchez, M. Gramaglia, C.J. Bernardos, A. de la Oliva, M. Calderon.Elsevier Ad Hoc Networks Journal, February 2014.

• Performance Analysis and Algorithm Selection for Reliable Multicast in IEEE 802.11aa Wire-less LAN, A. Banchs, A. de la Oliva, L. Eznarriaga, D. R. Kowalski, P. Serrano. IEEETransactions on Vehicular Technology, January 2014.

• Multicast group membership management in media independent handover services, CarlosGuimaraes, Daniel Corujo, Antonio de la Oliva, Yoshihiro Ohba, Rui L. Aguiar. ElsevierComputer Networks, January 2014.

• A First Implementation and Evaluation of the IEEE 802.11aa Group Addressed TransmissionService, P. Salvador, L. Cominardi, F. Gringoli, P. Serrano. ACM Computer CommunicationReview, January 2014.

• On IEEE 802.11k/r/v Amendments: Do They Have a Real Impact?, M.I. Sanchez, A. Bouk-erche. Accepted for publication in IEEE Wireless Communications Magazine.

• A Survey on Device-to-Device Communication in Cellular Networks, A. Asadi, Q. Wang, V.Mancuso. IEEE Surveys and Tutorials on Communications, 2014.

• On the Efficient Utilization of Radio Resources in Extremely Dense Wireless Networks, A.Asadi, V. Sciancalepore, V. Mancuso. IEEE Communication Magazine, 2014.

4.2 Conferences

• An SDN-based Network Architecture for Extremely Dense Wireless Networks, H. Ali- Ahmad,C. Cicconetti, A. de la Oliva, V. Mancuso, M. R. Sama, P. Seite, S. Shanmugalingam. InProceedings of IEEE Software Defined Networks for Future Networks and Services (IEEESDN4FNS), November 2013.



• WiFi Direct and LTE D2D in Action, A. Asadi, V. Mancuso. In Proceedings of WirelessDays’13, Valencia, Spain, November 2013.

• On the Compound Impact of Opportunistic Scheduling and D2D Communications in CellularNetworks. A. Asadi, V. Mancuso. ACM/IEEE MSWiM’13, Barcelona, Spain, Nov, 2013.

• CROWD: An SDN Approach for DenseNets, H. Ali-Ahmad, C. Cicconetti, A. de la Oliva,M. Draexler, R. Gupta, V. Mancuso, L. Roullet, V. Sciancalepore. European Workshop onSoftware Defined Networking (EWSDN13), Berlin, Germany, October 10-11, 2013.

• Distributed OFDMA resource and power allocation using Gibbs sampling methods, V. Garcia,C. S. Chen, Y. Zhou, J. Shi. 11th International Conference on Optical Internet (COIN),Beijing, Oct 2013.

• Energy consumption savings with 3G offload, M. I. Sanchez, C. J. Bernardos, A. de la Oliva,P. Serrano. First International Workshop on Cloud Technologies and Energy Efficiency inMobile Communication Networks (CLEEN’13), Las Vegas, USA, Sep. 2013

• Markov Model for Opportunistic Routing in Wireless Networks, A. Darehshoorzadeh, MI.Sanchez, A. Boukerche. In IEEE International Symposium on a World of Wireless Mobileand Multimedia Networks 2014 (WoWMoM 2014).

• A Generic Framework for Dynamic eICIC Optimization in LTE Heterogeneous Networks, N.Trabelsi, R. Laurent, A. Feki. At IEEE Vehicular Technology Conference 2014.

• Feasibility of Base Station Coordination and Dynamic Backhaul Network Configuration inBackhaul Networks with Limited Capacity, M. Dräxler, H. Karl. In European Wireless 2014(EW2014).

• Flow processing-aware Controller Placement in Wireless DenseNets, S. Auroux, H. Karl. InProceedings of the 25th IEEE International Symposium on Personal, Indoor and MobileRadio Communications (PIMRC2014).

• SmarterPhones: Anticipatory Download Scheduling for Segmented Wireless Video Streaming,M. Dräxler, H. Karl. In 1st KuVS Workshop on Anticipatory Networks, 2014.

• Anticipatory Power Cycling of Mobile Network Equipment for High Demand Multimedia Traf-fic, M. Dräxler, P. Dreimann, H. Karl. In IEEE Online Conference on Green Communications(IEEE Online GreenComm’14).

• MaxiNet: Distributed Emulation of Software-Defined Networks, P. Wette, M. Dräxler, A.Schwabe, F. Wallaschek, M. Hassan Zahraee, H. Karl. In Proceedings of the 2014 IFIPNetworking Conference (Networking 2014). IEEE, pp. 1-9.

• Modeling Multi-Mode D2D Communications in LTE. A. Asadi, P. Jacko, V. Mancuso. Inproceedings of ACM SIGMETRICS MAMA 2014, Austin, TX, USA, June 2014.

• Interference Coordination Strategies for Content Update Dissemination in LTE-A,V. Scian-calepore, V. Mancuso, A. Banchs, S. Zaks, A. Capone. In proceedings of INFOCOM 2014,Toronto, Canada, April-May 2014.

• LabVIEW based Platform for prototyping dense LTE Networks in CROWD Project, R. Gupta,T. Vogel, N. Kundargi, A. Ekbal, A. Morelli, V. Mancuso, V. Sciancalepore, R. Ford, S.Rangan. In proceedings of EuCNC 2014, Bologna, Italy, June 2014.



• Energy-Efficient User Association In Extremely Dense Small Cell Networks, C. Bottai, C.Cicconetti, A. Morelli, M. Rosellini, C. Vitale. In proceedings of EuCNC 2014, Bologna,Italy, June 2014.

• A Mechanism for Fair Distribution of Resources with Application to Sponsored Search, E.Christoforou, A. Fernndez Anta, A. Santos. In proceedings of Web and Internet Economics(WINE) 2014, Beijing, China, December 14-17, 2014

4.3 Tutorials & Panels

• Keynote speech of C. Cicconetti on The road towards heterogeneous & dense wireless networks:the CROWD perspective at CROSSFIRE plenary meeting, on July 2, 2013 in Paris, France.

• NI RF Round Table on 3-4 Dec, 2013 in Lyon, France Rohit Gupta showed the WP5 activitiesbased on NI/PXI platform for prototyping LTE-like dense wireless networks. This event wasNI internal event to network different researchers throughout Europe involved in prototypingactivities using NI/PXI platform. IMDEA, INCS, UC3M are invited and participated to theevent.

• SDN Experimentation Facilities and Tools, K. Pentikousis, U. Toseef, P. Wette, M. Dräxler.MaxiNet Tutorial at MONAMI 2014 (http://mon-ami.org/2014/show/tutorials).

• Invited talk of V. Mancuso (IMDEA), describing CROWD and the project activities: V. Man-cuso, ”Advanced Technologies for Extremely Dense and Heterogeneous Wireless Networks” atIWPC workshop on Advanced Small Cell Deployments and Cloud Technologies on 12-14 Novin Turin, Italy. In addition, V. Mancuso took part to the Research Session Panel” of theworkshop on November 14.

• Participation of Antonio de la Oliva as speaker in the Wireless SDN tutorial to the IEEE802 plenary meeting, Dallas, Tx, November 2013. The tutorial addressed applications ofSDN in wireless access and backhaul networks, with a focus on issues relevant to IEEE 802technologies. Tutorial background on the basics of SDN and OpenFlow was provided as anintroduction to the subject. The slides used during the presentation can be found at url:https://mentor.ieee.org/802-ec/dcn/13/ec-13-0055-01-00EC.pdf

• NIWeek’14 on 4-7 Aug, 2014 in Austin, USA Rohit Gupta showed the WP5 activities basedon NI/PXI platform for prototyping LTE-like dense wireless networks. This event was NIinternal event to network different researchers throughout the world involved in prototypingactivities using NI/PXI platform.

• 3GPP Closed Door Meeting on 20 Aug, 2014 in Dresden, Germany Rohit Gupta showed theWP5 activities based on NI/PXI platform for prototyping LTE-like dense wireless networks.This event was NI internal event to showcase NI/PXI platform within CROWD project to3GPP attendees in Dresden.

• Globecom’14 on 8-12 Dec, 2014 in Austin, USA Rohit Gupta organized a SDR panel inIndustry Forum and Exhibition Program titled, ”‘SDR Prototyping challenges for dense de-ployments for next generation wireless networks (LTE, WiFi)”’


https://mentor.ieee.org/802-ec/dcn/13/ec-13-0055-01-00EC.pdf


4.4 Posters & Demo

• The Playground of Wireless Dense Networks of the Future, H. Ali-Ahmad, C. Cicconetti, M.Draexler, R. Gupta, V. Mancuso, A. Morelli, L. Roullet, M. I. Sanchez, V. Sciancalepore.Future Network & Mobile Summit 2013, Lisbon, Portugal, July 3-5, 2013.

• DEMO: LabVIEW based framework for prototyping dense LTE networks, R. Gupta, B. Bach-man, R. Ford, S. Rangan, A. Morelli, V. Mancuso, N. Kundargi, and A. Ekbal. In proceedingsof WiNTECH 2014 (Demo/Poster Session).

• A prototyping methodology for SDN-controlled LTE using LabVIEW SDR and NS3,C. Vi-tale, R. Gupta, V. Mancuso, A. Morelli. Demo at ETSI workshop on Reconfigurable RadioSystems, Sophia Antipolis, France, December 3-4, 2014.

• LabVIEW based Platform for prototyping dense LTE Networks, R. Gupta, A. Ekbal, A.Nahler, V. Mancuso, A. De La Oliva, A. Morelli, R. Ford, S. Rangan. In proceedings ofGLOBECOM 2014 (Demo in the Industry Program).

• The CROWD framework for tackling challenges of dense small cell deployments using Soft-ware Defined Networking (SDN), R. Gupta, M. Draexler, H. Karl, V. Mancuso, A. De LaOliva, A. Morelli, E. Bizouarn, E. Zeydan. In proceedings of GLOBECOM 2014 (Poster inthe Industry Program).

• A prototyping methodology for SDN-controlled LTE using SDR, V. Mancuso, C. Vitale, R.Gupta, K. Rathi, and A. Morelli. Demo at ETSI workshop on Reconfigurable Radio Systems,Sophia Antipolis, France, December 3-4, 2014.

4.5 Standardization

4.5.1 IEEE 802.1cf (OMNIRAN)

• PAR and 5C Text Comments, Antonio de la Oliva (UC3M), September 2013. https://mentor.ieee.org/omniran/dcn/13/omniran-13-0076-00-0000-par-and-5c-text-comments.do

cx

• Proposed OmniRAN SDN Use Case for External Communication, Roger Marks, Antonio de laOliva, Juan Carlos Zuniga (EthAirNet, UC3M, InterDigital), August 2013. https://mentor.ieee.org/omniran/dcn/13/omniran-13-0059-00-0000-proposed-omniran-sdn-use-cas

e-for-external-communication.pptx

• SDN Use Cases Summary, Antonio de la Oliva (UC3M), Juan Carlos Zuniga (InterDigital),Roger Marks (EthAirNet), July 2013. https://mentor.ieee.org/omniran/dcn/13/omniran-13-0044-02-0000-sdn-use-cases-summary.pptx

• SDN Use Case ToC, Juan Carlos Zuniga (InterDigital), Antonio de la Oliva (UC3M), Pre-sented in the January 2014 meeting. https://mentor.ieee.org/omniran/dcn/14/omniran-14-0007-00-0000-sdn-use-case-toc.pptx

• SDN Use Cases for BoF, Juan Carlos Zuniga (InterDigital), Antonio de la Oliva (UC3M),Paul Congdon (Tallac), Presented in the March 2014 meeting. https://mentor.ieee.org/omniran/dcn/14/omniran-14-0029-00-0000-sdn-based-use-cases-for-bof.pptx


https://mentor.ieee.org/omniran/dcn/13/omniran-13-0076-00-0000-par-and-5c-text-comments.docxhttps://mentor.ieee.org/omniran/dcn/13/omniran-13-0076-00-0000-par-and-5c-text-comments.docxhttps://mentor.ieee.org/omniran/dcn/13/omniran-13-0076-00-0000-par-and-5c-text-comments.docxhttps://mentor.ieee.org/omniran/dcn/13/omniran-13-0059-00-0000-proposed-omniran-sdn-use- case-for-external-communication.pptxhttps://mentor.ieee.org/omniran/dcn/13/omniran-13-0059-00-0000-proposed-omniran-sdn-use- case-for-external-communication.pptxhttps://mentor.ieee.org/omniran/dcn/13/omniran-13-0059-00-0000-proposed-omniran-sdn-use- case-for-external-communication.pptxhttps://mentor.ieee.org/omniran/dcn/13/omniran-13-0044-02-0000-sdn-use-cases- summary.pptxhttps://mentor.ieee.org/omniran/dcn/13/omniran-13-0044-02-0000-sdn-use-cases- summary.pptxhttps://mentor.ieee.org/omniran/dcn/14/omniran-14 -0007-00-0000-sdn-use-case-toc.pptxhttps://mentor.ieee.org/omniran/dcn/14/omniran-14 -0007-00-0000-sdn-use-case-toc.pptxhttps://mentor.ieee.org/omniran/dcn/14/omniran-14-0029-00-0000-sdn-based-use-cases-for-bof.pptxhttps://mentor.ieee.org/omniran/dcn/14/omniran-14-0029-00-0000-sdn-based-use-cases-for-bof.pptx


• SDN Use Cases and Requirements, Juan Carlos Zuniga (InterDigital), Antonio de la Oliva(UC3M), Paul Congdon (Tallac), May 2014. https://mentor.ieee.org/omniran/dcn/14/omniran-14-0041-00-CF00-sdn-use-cases-and-requirements.docx

• Generic IEEE 802 Network Reference Model, Antonio de la Oliva (UC3M), Juan CarlosZuniga (InterDigital), Luis Contreras (Telefonica), Roger Marks (EthAirNet Associates),Nov 2014. https://mentor.ieee.org/omniran/dcn/14/omniran-14-0068-03-CF00-generic-ieee-802-network-reference-model.docx

4.5.2 IEEE 802.21d

• Restructure Clause 9.4, Lily Chen (NIST), Antonio de la Oliva (UC3M) , November 2013. https://mentor.ieee.org/802.21/dcn/13/21-13-0227-02-MuGM-restructure-cluase-9-4.

docx

• Section 9.4.2 flow chart, Antonio de la Oliva (UC3M), September 2013. https://mentor.ieee.org/802.21/dcn/13/21-13-0178-01-MuGM-section-9-4-2-flow-char.pdf

• Remedy for the Proxy for Information Services, Antonio de la Oliva, Subir Das, July 2013. https://mentor.ieee.org/802.21/dcn/13/21-13-0114-00-srho-remedy-for-the-proxy-f

or-information-services.docx

• Antonio de la Oliva has served as Technical Editor of IEEE 802.21d during 2014. Currentversion of the draft document is D7.0. This document is currently undergoing the secondsponsor ballot.

• Remedy for comment #79, 81, 82, 100, Antonio de la Oliva (UC3M), Yoshikazu Hanatani(Toshiba), April 2014. https://mentor.ieee.org/802.21/dcn/14/21-14-0057-01-MuGM-remedy-for-comment-79-81-82-100.docx

• Remedy comment #83, fig38, Antonio de la Oliva (UC3M), April 2014. https://mentor.ieee.org/802.21/dcn/14/21-14-0071-00-MuGM-remedy-comment-83-fig38.pdf



4.5.3 IEEE 802 EC Privacy Recommendation SG

• MAC experiments and Wiki procedures, Fabio Giust (UC3M), C.J. Bernardos (UC3M) andAntonio de la Oliva (UC3M), Oct 2014. https://mentor.ieee.org/privecsg/dcn/14/privecsg-14-0015-00-0000-mac-experiments-and-wiki-procedures.ppt

• UC3M has participated organising the WiFi Privacy Network Experiment at IETF91.

• W

document properties d 1.3 pu - cordis.europa.eu · been raised with the study and implementation of...

Documents