deliverable d1.1 report on stakeholders characterization and traffic characteristics

Seventh Framework STREP No. 317846 D1.1 - Report on Stakeholder Relationships and Traffic Characteristics Public

Version 1.4 of 161 © Copyright 2013, the Members of the SmartenIT Consortium

Socially-aware Management of New Overlay Application Traffic with

Energy Efficiency in the Internet

European Seventh Framework Project FP7-2012-ICT- 317846-STREP

Deliverable D1.1

Report on Stakeholders Characterization

and Traffic Characteristics

The SmartenIT Consortium Universität Zürich, UZH, Switzerland Athens University of Economics and Business - Research Centre, AUEB, Greece Julius-Maximilians Universität Würzburg, UniWue, Germany Technische Universität Darmstadt, TUD, Germany Akademia Gorniczo-Hutnicza im. Stanislawa Staszica W Krakowie, AGH, Poland Intracom SA Telecom Solutions, ICOM, Greece Alcatel Lucent Bell Labs, ALBLF, France Instytut Chemii Bioorganiicznej Pan, PSNC, Poland Interoute S.P.A, IRT, Italy Telekom Deutschland GmbH, TDG, Germany

© Copyright 2013, the Members of the SmartenIT Consortium For more information on this document or the SmartenIT project, please contact: Prof. Dr. Burkhard Stiller Universität Zürich, CSG@IFI Binzmühlestrasse 14 CH—8050 Zürich Switzerland Phone: +41 44 635 4331 Fax: +41 44 635 6809 E-mail: [email protected]

D1.1 - Report on Stakeholder Relationships Seventh Framework STREP No. 317846 and Traffic Characteristics Public

of 161 Version 1.4 © Copyright 2013, the Members of the SmartenIT Consortium

Document Control

Title: Report on Stakeholders Characterization and Traffic Characteristics

Type: Public

Editor(s): Ioanna Papafili, George D. Stamoulis

E-mail: [email protected], [email protected]

Author(s): Matteo Biancani, Manos Dramitinos, Gerhard Hasslinger, David Hausheer, Tobias Hossfeld, Roman Lapacz, Frank Lehrieder, Guiherme Machado, Ioanna Papafili, George Petropoulos, Alessandro Predieri, Julius Rueckert, Grzegorz Rzym, Sergios Soursos, Spiros Spirou, George Stamoulis, Burkhard Stiller, Christos Tsiaras, Krzysztof Wajda, Matthias Wichtlhuber, Mateusz Wielgosz, Arkadiusz Zwierz

Doc ID: D1.1-v1.4.doc

AMENDMENT HISTORY

Version Date Author Description/Comments

V0.1 November 1, 2012 Burkhard Stiller First version

V0.2 November 13, 2012

Ioanna, ALL ToC proposal


Ioanna, ALL ToC update, work allocation


Ioanna, David ToC update

V0.5 December 04, 2012

Roman, George P., Sergios, Spiros, Christos, Krzysztof, Frank, Tobias, Ioanna, George D., Matteo, Julius, David

Input in bullet form


Ioanna, Christos, Guiherme New structure and updated input by UZH


Gerhard, Frank, Tobias, Julius, Matthias, David, Christos, Alessandro, Roman, Matteo, George P. Sergios, Spiros, Ioanna, Manos, George S.

Input in text form; new structure

V0.7.2 January 03, 2013 Ioanna Interim version; comments for expected input in each section; structure update.

V0.7.8 January 31, 2013 Julius, Matthias, David, Christos, Guiherme, Alessandro, Roman, Matteo, George P., Sergios, Ioanna, Manos, George S.

Input in text form from all partners integrated - MS1

V0.8 February 22, 2013 George S., Manos, Ioanna Comments to all involved partners

V0.8.5 March 08, 2013 Julius, Matthias, David, Christos, Guiherme, Alessandro, Roman, Matteo, George P., Sergios, Ioanna, Manos, George S.

Address comments; integrate new input

V0.9 March 15, 2013 Julius, Matthias, David, Christos, Guiherme, Alessandro, Roman, Matteo, George P., Sergios, Ioanna, Manos, George S., Gerhard

Address comments; integrate new input

V0.9.5 March 26, 2013 Matteo, Chris, Roman, Gerhard, Ioanna, Guiherme, Thomas, Manos, George P.

Address comments, provide new input; integrate new input

V0.9.6 March 28, 2013 Krzysztof, Grzegorz, Mateusz, Arkadiusz, Guiherme, Thomas, Julius,

Updated input.



Ioanna

V0.9.7 April 3, 2013 Matteo, Sabine (internal reviewers) Preliminary comments

V1.0.2 April 8, 2013 Ioanna, George S. Address preliminary comments by reviewers; send final comments to authors

V1.0.3 April 10, 2013 Ioanna, George S. Latest version sent to reviewers

V1.0.4 April 17, 2013 Gino, Sabine Final comments to editors

V1.0.5 April 19, 2013 Tobias (WP1 leader), George S. Comments upon reviewers' comments

V1.1 April 22, 2013 Ioanna, George S. Address part of reviewers' comments; send integrated version to authors to finalize contributions based on reviewers' comments

V1.1.2 April 22, 2013 Burkhard (coordinator) Overall comments and guidelines

V1.1.5 April 23, 2013 George P. (ICOM), Guiherme (UZH), Gerhard (TDG), Alessandro, Matteo

(IRT), Manos, Ioanna (AUEB),

Address comments by reviewers

V1.2 April 24, 2013 Ioanna, George S. Integrate partners’ input; address comments by Burkhard; final comments sent to authors

V1.2.1 April 27, 2013 Guiherme, Julius, Chris, Gehrard, Valentin, Alessandro

Address final comments

V1.3 April 29, 2013 Ioanna, George S. Pre-final version in BSCW

V1.4 April 30, 2013 Ioanna, George S. Final version in BSCW

Legal Notices The information in this document is subject to change without notice. The Members of the SmartenIT Consortium make no warranty of any kind with regard to this document, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. The Members of the SmartenIT Consortium shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material.



Table of Contents

Table of Contents 4

Table of Figures 8

Tables 10

1 Executive Summary 12

2 Introduction 16

2.1 Purpose of the Deliverable D1.1 16

2.2 Document Outline 17

3 Traffic & Energy Characteristics of Cloud Computing 18

3.1 Cloud Basics 18

3.1.1 Cloud and Data Centre 20

3.1.2 Cloud and Grid 20

3.1.3 Cloud and CDN 21

3.2 Cloud Models 21

3.2.1 Deployment Models 22

3.2.2 Service Models 23

3.3 SLAs of Cloud Services and Indicative Parameters 24

3.4 Financial / Economic Implications of the Cloud 25

3.5 Cloud Services and Real Cloud Deployments 26

3.5.1 Amazon 26

3.5.2 Google 27

3.5.3 IBM 28

3.5.4 Microsoft 28

3.5.5 Interoute 30

3.6 Traffic Characteristics of Clouds and Data Centres 31

3.6.1 Global IP Traffic Statistics and Trends 31

3.6.2 Data Centre Traffic Statistics and Tends 32

3.6.3 Cloud Traffic Statistics and Trends 34

3.6.4 Characteristics of Communication between Cloud Data Centres 36

3.7 Characteristics of Energy Consumption by Cloud Services 37

3.7.1 Data Centres 37

3.7.2 ISPs' Networks 41

3.7.3 Mobile Devices 41

3.8 Summary of Cloud Characteristics 44



4 Characteristics of Cloud Applications 46

4.1 Video Streaming 47

4.1.1 YouTube 47

4.1.2 Netflix 48

4.1.3 Hulu 49

4.1.4 Relevance for SmartenIT 49

4.2 Mobile P2P Video Streaming 50

4.2.1 PPLive 51

4.2.2 PPStream 52


4.3 Online Storage 53

4.3.1 Dropbox 54

4.3.2 Google Drive 55

4.3.3 PiCsMu 57


4.4 Search Engines 59

4.4.1 Google Search 59

4.4.2 Yahoo! Search 60

4.4.3 Bing 60


4.5 Online Social Networks 62

4.5.1 Facebook 62

4.5.2 Twitter 63

4.5.3 Google+ 64

4.5.4 LinkedIn 65


4.6 Online Gaming 65

4.6.1 Onlive 66

4.6.2 Gaikai 66

4.6.3 Zynga 67


4.7 Cloud Content Delivery Networks 68

4.7.1 MetaCDN 68


4.8 Mobile P2P File-Sharing 70

4.8.1 BitTorrent 71




4.9 Mobile Instant Messaging and Voice-over-IP 72

4.9.1 Viber 73

4.9.2 HeyTell 73

4.9.3 AbaCUS 74


4.10 Summary of Overlay Applications Characteristics 75

5 Stakeholders Characterization 77

5.1 Basic Definitions 77

5.2 Tools and Methodologies for Stakeholders Characterization 78

5.2.1 Value Network Configuration 78

5.2.2 Business Model Reference Framework 79

5.2.3 Tussle Analysis Framework 82

5.3 Inter Data Centre / Inter-Cloud communication 83

5.3.1 Scenario 83

5.3.2 Stakeholders, Roles, Interests 85


5.3.4 Business Modeling 88

5.3.5 Potential for Collaboration and Relevance for SmartenIT 90

5.4 Collaboration for Energy Efficiency 91

5.4.1 Scenarios 91





5.5 Global Service Mobility 100



5.5.3 Business Model Analysis 105


5.6 Social-Awareness for Content Delivery 107

5.6.1 Scenarios 108







6 Summary and Conclusions 117

6.1 Key Outcomes & Lessons Learnt 117

6.1.1 Traffic Characteristics 117

6.1.2 Cloud-based Applications Characteristics 119

6.1.3 Stakeholders Characterization 120

6.2 Next steps 121

7 SMART objectives addressed 123

8 References 126

9 Abbreviations 137

10 Acknowledgements 139

A. YouTube Detailed Traffic Characteristics 141

A.1 Steps in YouTube video download 141

A.2 YouTube redirection process 143

A.3 Application-layer traffic management 143

A.4 QoE of YouTube video streaming – key influence factors 145

A.5 Consequences for SmartenIT 146

B. Dropbox Detailed Traffic Characteristics 148

B.1 Monitoring setup 148

B.2 Methodology and data sets 149

B.3 Popularity of different cloud storage providers 150

B.4 Performance 151

B.5 Service workload 154

B.6 Consequences for SmartenIT 155

C. Facebook and YouTube Traffic Statistics 157

C.1 Packet and Flow Measurement for YouTube and Facebook 157

C.2 2nd

order statistics results for YouTube and Facebook traffic variability 158

C.2.1 Definition of 2nd

order statistics and autocorrelation function 158

C.2.2 2nd order statistics of self-similar processes 159

C.2.3 2nd order statistics for Gilbert-Elliott and 2-state (semi-)Markov models159

C.2.4 Measurement evaluations of the 2nd

order statistics 160

C.3 Consequences for SmartenIT 161



Table of Figures

Figure 3-1: Application of 80/20 rule to Company operating his own data centre [193]. .............. 26

Figure 3-2: Interoute’s VDC. ........................................................................................................ 30

Figure 3-3: Trends in IP traffic growth reported from different sources. ....................................... 32

Figure 3-4: Global data centre traffic growing rate [20]. ............................................................... 33

Figure 3-5: Global data centre traffic by destination [20]. ............................................................ 33

Figure 3-6: Cloud vs. traditional data centre traffic growth 2011-2016 [20]. ................................ 35

Figure 3-7: Predicted US electricity use for data centres from the EPA report to Congress [151] and the range estimated by Koomey [152]. .................................................................. 39

Figure 3-8: Breakdown of data centre energy overheads [139]. .................................................. 40

Figure 3-9: Interaction of measurement platform and developer [102]. ....................................... 42

Figure 3-10: Power breakdown of 20 users by hardware component, taken from [98]. ............... 43

Figure 3-11: DFA representing the WiFi energy consumption as modelled in [100]. ................... 43

Figure 3-12: Energy consumption of browser search; the green indicates the percentage of energy wasted in tail states [100]. ................................................................................. 44

Figure 4-1: Bing ISN structure [198]............................................................................................. 61

Figure 4-2: Google+ User Statistics (Dec. 2012) [33]. ................................................................. 64

Figure 4-3: MetaCDN architecture [47]. ....................................................................................... 69

Figure 5-1: VNC notation. ............................................................................................................ 79

Figure 5-2: Tussle analysis methodology - The SESERV project [107]. ...................................... 82

Figure 5-3: Stakeholders map - The SESERV project [115]. ....................................................... 83

Figure 5-4: A source data centre uses available bandwidth to perform bulk backup to sink data centre [116]. .................................................................................................................. 84

Figure 5-5: The scenario use cases and network placement of the data centres. ....................... 85

Figure 5-6: Basic VNC model. ..................................................................................................... 87

Figure 5-7: Modified generic VNC model. .................................................................................... 87

Figure 5-8: Generic VNC model. .................................................................................................. 88

Figure 5-9: An example of two data centre federations. .............................................................. 92

Figure 5-10: The NaDa architecture [192]. .................................................................................. 93

Figure 5-11: Generic VNC for separation of IaaS - PaaS Providers. ........................................... 95

Figure 5-12: Updated VNC for integration of IaaS - PaaS providers. .......................................... 95

Figure 5-13: Extended VNC to depict the insertion of the Energy Broker. ................................... 96

Figure 5-14: Extended VNC to depict the distributed approach (NaDa). ..................................... 96

Figure 5-15: Global Service Mobility high-level approaches for moving data resources. ........... 101

Figure 5-16: Application Provider without using Cloud to enable Global Service Mobility. ........ 103



Figure 5-17: Application Provider using Cloud Providers to enable Global Service Mobility. ..... 104

Figure 5-18: Application Provider using Cloud with NaDas to enable Global Service Mobility. . 105

Figure 5-19: Content dissemination of UGC over an OSN. ....................................................... 108

Figure 5-20: Structure of a socially aware P2P scheme for content delivery [149]. ................... 109

Figure 5-21: Basic VNC for Exploitation of Social Awareness for Content Delivery – Centralized; Collaborative. .............................................................................................................. 111

Figure 5-22: Updated VNC for Exploitation of Social Awareness for Content Delivery – Centralized; Non-Collaborative; CDN-driven. .............................................................. 112

Figure 5-23: Updated VNC for Exploitation of Social Awareness for Content Delivery – Centralized; Non-Collaborative; ISP-driven. ................................................................ 112

Figure 5-24: Extended VNC for Exploitation of Social Awareness for Content Delivery – Distributed; Non-Collaborative. ................................................................................... 113

Figure A-1: Organization of the YouTube Caches; dashed lines indicate possible redirections - Figure taken from [55]. ................................................................................................ 142

Figure A-2: Identification of key influence factors on YouTube QoE. ........................................ 146

Figure A-3: Initial delays have almost no influence on MOS for videos of duration 60 s and 30 s – compared to influence of stalling length [66]. ........................................................... 146

Figure B-1: An example of the Dropbox communication observed [29]. .................................... 148

Figure B-2: Number of unique clients accessing each cloud-based storage service in Home 1 [29]. ............................................................................................................................. 150

Figure B-3: Data volume per cloud-based storage service (in bytes) in Home 1 [29]. ............... 150

Figure B-4: Share of total traffic volume in Campus 2 [29]. ....................................................... 151

Figure B-5: Traffic breakdown between Dropbox servers [29]. .................................................. 152

Figure B-6: Number of contacted storage servers [29]. ............................................................. 152

Figure B-7: Distribution of TCP flow sizes of file storage for the Dropbox client [29]. ................ 153

Figure B-8: Throughput of storage flows in Campus 2 [29]. ....................................................... 153

Figure B-9: Distribution of the number of devices per household [29]. ...................................... 155

Figure B-10: Dropbox session duration [29]. .............................................................................. 155

Figure C-1: Statistics on packet and flow sizes [142]. ................................................................ 158

Figure C-2: Statistics on the mean traffic rates per flow [142]. .................................................. 158

Figure C-3: Burstiness over time scales for YouTube, Facebook traffic parts [142]. ................. 160



Tables

Table 3-1: Data centre traffic trends for 2011-2016 [20]. ............................................................. 33

Table 3-2: Cloud IP traffic trends 2011-2016 [20]. ....................................................................... 35

Table 3-3: Summary of initiatives for energy efficiency in data centres. ...................................... 38

Table 4-1: Top 10 applications in the Asia-Pacific region at peak time [68]................................. 51

Table 4-2: Typical energy reduction in company after migrating to Google Apps [202][128]. ...... 56

Table 4-3: Top 5 search engines [128]. ....................................................................................... 59

Table 4-4: MetaCDN performance and Content Providers’ benefits [47]. .................................... 69

Table 4-5: Summary of cloud-based applications' characteristics. .............................................. 76

Table 5-1: Business Model Reference Framework. ..................................................................... 80

Table 5-2: Stakeholder identification for inter data centre / inter-cloud communication. .............. 85

Table 5-3: Integrated stakeholder identification for collaboration for energy efficiency. .............. 93

Table 5-4: Stakeholder identification for global service mobility. ............................................... 102

Table 5-5: Stakeholder identification for social-awareness for content delivery. ....................... 110

Table 7-1: Overall SMART objectives addressed. ..................................................................... 123

Table 7-2: Specific SMART objectives addressed. .................................................................... 123

Table B-1: Datasets for Dropbox evaluation [29]. ...................................................................... 149

Table C-1: Packet and flow measurement for YouTube and Facebook [142]. .......................... 157



(This page is left blank intentionally.)



1 Executive Summary

This document is the deliverable D1.1 “Report on Stakeholders Characterization and Traffic Characteristics” of Work Package 1 “Traffic Measurements and Scenario Definitions” within the ICT SmartenIT Project 317846. The targets of Deliverable D1.1 are as follows:

Target 1 is to identify major characteristics of cloud-based overlay applications and services such as traffic volume generated, traffic patterns, energy consumption, QoE aspects, and end-user's devices implications. Additionally, the potential for intervention and optimization w.r.t. traffic, energy and QoE need to be addressed as the ultimate target of this investigation is to provide input to Deliverable D1.2 and the selection process of the overlay applications to be addressed by the SmartenIT project.

Target 2 is to describe basic scenarios of cloud-service provisioning in which SmartenIT can add value with innovative effective traffic management mechanisms to be developed, identify the involved stakeholders, i.e., network provider, cloud provider, end-users, etc., in these scenarios, as well as their interests in terms of cost, traffic, energy and QoE optimization. Finally, Target 2 was to analyze the identified stakeholders' relationships, so as to gain insight on the technical and business dependencies among them, and identify potential for collaboration among them to achieve a mutually beneficial situation.

Initially, addressing Target 1, we define crucial terms related to SmartenIT and provide our understanding of: what cloud computing stands for practically, the differences between cloud computing and grid computing, cloud and CDN, or cloud and data centre to narrow down the field of applications and deployments of our interest leaving legacy CDN deployments out of the scope of this deliverable and the project in general. Then, we describe deployment and service models, economic and QoS aspects of cloud computing to develop a common background on the impact of cloud computing on the various Internet stakeholders.

Moreover, we overview a variety of cloud services, inter alia Amazon's EC2 and S3, Windows Azure, IBM's SmartCloud, Google's App Engine, which are offered by well-known major cloud operators, the so-called Internet giants, such as Amazon, Google, IBM. Based on the aforementioned services, most of the cloud-based applications of SmartenIT's interest are built on, e.g., Dropbox employs both Amazon's EC2 and S3 to provide its personal online storage service. This description allows us to understand the architecture and operation of the cloud-based overlay applications, to define realistic scenarios of cloud service provision, and last to identify possible stakeholders corresponding to real operators.

Furthermore, we overview various sources performing studies of traffic generated by data centres and cloud, both as a share of global IP traffic, and in terms of absolute traffic volumes, popularity and future trends. Major outcomes of this investigation are as follows:

Global data centre IP traffic will nearly quadruple, while it will grow by 31% CAGR over the next 5 years.

On the other hand, global cloud IP traffic will increase 6 times and grow by 44% CAGR over the next 5 years, while global cloud IP traffic will account for nearly two-thirds of total data centre traffic by 2016.



The rapidly growing number of mobile devices (e.g., smartphones and tablets) used to access new overlay applications constitutes a key driver of the cloud traffic increase.

Concerning the increasing number of mobile devices, it has an important influence on how application ecosystems are implemented nowadays, namely, applications 'run' in the cloud (i.e., computation, data storage), while only a 'shell' providing an interface to access the application remains at the mobile device.

Then, addressing the energy consumption of data centres, ISPs' networks and mobile devices as an important characteristic of the energy efficiency in the cloud, we resort to the following conclusions:

Although the IP traffic was expected to grow by 46% per year in the last 5 years, the energy consumption for data centres did not increase as predicted. In particular, it has been estimated to have increased by up to 36% in the US, due to the use of virtualization that leads to more efficient hardware utilization.

Currently, only, 30% of total energy is delivered to IT equipment for critical operations such as hard disk drive operations, networking, memory, CPU, motherboards, fans, etc., implying PUE equal to 3.3.

On the other hand, the remaining 70% of total data centre energy consumption splits to 45% for cooling infrastructure and 25% for power delivery units.

Although virtualization is seen to be a key driver for energy efficiency in the cloud, it does not provide for energy efficiency in the network domain. SmartenIT will design mechanisms, which will manage cloud traffic by live VMs migration from data centre to data centre, migration that will be strictly tied to data centre resources orchestration and energy efficiency achievements.

Concerning ISPs' networks, the energy consumption has increased with traffic volume and popularity of Internet applications and has become a critical issue for further traffic and bandwidth growth. Moreover, energy saving by switching off parts of the network equipment in phases of lower load and reactivating them on demand is often not feasible currently due to long set up periods of networking equipment. On the other hand, considering users' mobile devices, nearly 50% of all smartphone interactions have been recorded as being related to communication, i.e., email, SMS, instant messaging and voice calls, while proposed approaches that aim to optimize the consumed energy utilize the trade-off of transmission range and energy consumption.

Based on our investigation of the energy implications of cloud computing, we observe that although cloud improved energy consumption w.r.t. traffic growth; yet significant inefficiencies still exist either in data centres and ISPs' networks, or in end users' mobile devices. Such inefficiencies provide an opportunity for SmartenIT for intervention and optimization.

Next, we perform an extensive overview of a wide selection of cloud based overlay applications. We have provided a description of the characteristics of such applications addressing their traffic, energy and QoE implications, potential for intervention and potential for optimization. The outcomes of this investigation are as follows:

Video streaming, P2P video streaming and file sharing, online storage, cloud CDNs and online gaming generate high or very high traffic volumes, while online social



networks, search engines and mobile instant messaging and VoIP generate medium to low volumes,

All categories of applications, except the mobile ones, cause a significant increase of energy consumption in data centres, whereas mobile applications increase the energy consumption in mobile devices, which is higher for video streaming,

Considering QoE aspects, all of these applications require low latency, which may not be always achieved, as their performance is highly dependent on network conditions such as congestion.

Regardless of the type of application, the potential for intervention by a SmartenIT traffic management mechanism is expected to be higher for non-proprietary solutions. Moreover, potential for optimization is considered to be directly dependent both on generated traffic volumes, energy implications and QoE requirements. Thus, potential for optimization is considered to be higher for video-related applications, e.g., video streaming and online gaming, where there is higher potential for a more “visible” impact by SmartenIT.

Addressing Target 2, we describe four basic scenarios of cloud-service provisioning in which SmartenIT can add value. Then, employing appropriate tools and methodologies we identify the involved stakeholders in the four scenarios of interest and characterize their relationships, revenues and money flows, etc. Major outcomes of this analysis is for each of the four scenarios as follows:

The inter data centre communication scenario depicts the need to efficiently and timely carry traffic among data centres that are located in different parts of the network so as to comply with certain market-driven requirements and deadlines. ISPs, Data Centre Providers, or even Application Providers, either existing or emerging, may try to perform bundling of the inter data centre communication service as provided to either other (possibly smaller) Data Centre, or Application Providers and End-Users with existing services, e.g., storage, computation, so as to dominate the market and strengthen their position in the markets where they are active. Clearly the additional offering of such a sophisticated service would be of high value to the data centre customers, whereas this in turn could create mixed incentives for collaboration and competition with the other data centres and also have adverse implications on the agreements with the ISPs.

Collaboration between cloud operators to achieve energy efficiency can be performed either in a distributed or a centralized manner. In the first case, a SmartenIT mechanism could be seen as a set of architecture/functional elements embodied within the aforementioned stakeholders, while in the second one, the SmartenIT mechanism could be developed or placed as an extra entity among them, so as to coordinate and enforce the collaboration for the efficient energy consumption behaviour among all players. Such an entity would be responsible for orchestrating the collaboration between cloud providers, and can be controlled by either a member of the federation, or by a trusted third-party. However, in order for the Energy Broker to be controlled by one member of the federation, then it would be necessary to achieve a win situation not only for the entire federation, but also for each member of it (due to the attained energy efficiency), at least in the long term; otherwise, negatively affected members of the federation will either leave from it, or even decide not to join at all. Nevertheless, depending on the considered setup, various inter-connection agreements need to be established between the various stakeholders, whereas different money flows will be generated then.



Global Service Mobility involves the capability to move resources between data centres or clouds, close to where the end-users are located, without disrupting the service delivered. In order to achieve Global Service Mobility, the user's location, i.e., AS, needs to be detected; then, service-related data need to be moved to the new user location. For the latter, two major approaches were identified, i.e., one time data move, and gradual data move. The two approaches serve different purposes and applications, while they can also be combined. The stakeholders’ characterization revealed that when the Application Provider is using a Cloud Provider in order to move users’ data/resources closer to the user, then the former stakeholder may focus on the application layer and on the data-moving policies, which are subsequently implemented by the Cloud Provider based on the latter's optimization criteria and constraints, e.g., one-time data move, or gradual data move. On the other hand, Global Service Mobility could be achieved by a distributed approach involving also ISPs and end-users; e.g., the NaDas solution could be extended to address service mobility apart from content mobility.

Finally, the exploitation of social awareness to perform efficient content delivery can be employed either in a centralized manner or in a more decentralized one, i.e., forming a P2P overlay where end-users participate by offering their storage and computation resources. A major question is who will apply the mechanism, e.g., the CDN or the ISP; in any case, though, the social awareness scenario within SmartenIT is considered to be a provider-driven solution. Moreover, a SmartenIT mechanism could be seen either as a set of functional elements embodied within the involved stakeholders, e.g., CDN provider, ISP, end-users' clients, or alternatively as a module that could be employed by a new entity, e.g., an OTT Provider, placed among the existing stakeholders, so as to collect social information and provide it to the CDN, or the ISP. Alternatively, in the peer-assisted setup, the SmartenIT point-of-intervention could be: either a functional module within the ISP, or a new OTT entity introduced, or the enhancement of the overlay tracker orchestrating the P2P overlay.

A general comment based on the stakeholders characterization outcome is that collaborative approaches addressing the various stakeholders’ incentives to collaborate are highly relevant to SmartenIT’s scope and will drive the design and specification of the traffic management mechanisms within WP2. The analysis in this deliverable has revealed that there is indeed considerable potential for collaboration in a variety of cases.

Concluding the contribution of Deliverable D1.1 both to the next phases of SmartenIT but also in general, we can state that D1.1 can serve as a complete handbook summarizing the variety and characteristics of a multitude of cloud services and applications provided over the cloud. D1.1 provides also a categorization of cloud-based applications based on the type of the application, i.e., the service offered to the end-user, in order to organize applications and address main characteristics of these categories, e.g., traffic volumes generated, or QoS requirements. Nevertheless, a complete classification of the cloud-based applications is to be conducted in D1.2, where also Task T1.1 "Cloud Services Classification" is to be concluded. Moreover, D1.1 describes the four major scenarios of high interest to SmartenIT and characterizes stakeholders so as to gain insight in their interests, relationships under different setups and potential for collaboration among them. Again, the complete and final definition of the SmartenIT scenarios will be provided in the upcoming Deliverable D1.2, where also the work of Task T1.4 “Scenarios Definitions” will be concluded.



2 Introduction

The Internet has seen a strong move to support overlay applications, which demand a coherent and integrated control in the underlying heterogeneous networks in a scalable, resilient, and energy-efficient manner. Cloud-based applications, i.e. applications which run completely or partly on the cloud or use cloud services, have set a strong need for network efficiency at all layers; therefore, cooperative cross-layer traffic optimization is a major goal to follow and can be achieved with tighter integration of network management and overlay service functionality.

Therefore, SmartenIT targets an incentive-compatible cross-layer network management scheme for providers of cloud-based applications, network providers, cloud providers, and end-users to ensure a Quality of Experience (QoE)-awareness, by addressing accordingly load and traffic patterns or special application requirements, and exploiting at the same time social awareness (in terms of user relations and interests). Additionally, the energy efficiency with respect to both end-user devices and underlying networking and application provisioning infrastructure is tackled to ensure an operationally efficient management. Moreover, incentive-compatibility of network management mechanisms for improving metrics in all layers and among all players will serve as the major mechanism to deal with real-life scenarios.

2.1 Purpose of the Deliverable D1.1

This document is the deliverable D1.1 “Report on Stakeholders Characterization and Traffic Characteristics” of Work Package 1 “Traffic Measurements and Scenario Definitions” within the ICT SmartenIT Project 317846.

The targets of this deliverable are as follows:

Target 1 was to identify major characteristics of cloud-based overlay applications and services such as traffic volume generated, traffic patterns, energy consumption, QoE aspects, and end-user's devices implications. Additionally, the potential for intervention and optimization w.r.t. traffic, energy and QoE need to be addressed as the ultimate target of this investigation is to provide input to Deliverable D1.2 and the selection process of the overlay applications to be addressed by the SmartenIT project.

Target 2 was to describe basic scenarios of cloud-service provisioning in which SmartenIT can add value with innovative effective traffic management mechanisms to be developed, identify the involved stakeholders, i.e., network provider, cloud provider, end-users, etc., in these scenarios, as well as their interests in terms of cost, traffic, energy and QoE optimization. Finally, Target 2 was to analyze the identified stakeholders' relationships, so as to gain insight on the technical and business dependencies among them, and identify potential for collaboration among them to achieve a mutually beneficial situation.

The two targets are inter-dependent in that the characterization of traffic and energy efficiency is performed towards the identification of potential for SmartenIT optimization for the various stakeholders, while, on the other hand, the characterization of stakeholders relationship is based on the description of the functioning of some representative cloud-based applications, with their traffic and energy requirements.



2.2 Document Outline

This document is organized as follows: Chapter 3 focuses on cloud computing and services provided over it. Initially, in Section 3.1, the definition of basic terms related to cloud computing is provided, while service and deployment models of cloud computing are subsequently described. Moreover, we provide definitions of terms which are crucial for our studies, i.e., what cloud computing stands for practically, and clarify the differences, to our understanding, between cloud computing and grid computing, cloud and Content Distribution Network (CDN), or cloud and data centre. Next, in Sections 3.2, 3.3, and 3.4, we briefly address deployment models, economic and QoS aspects of cloud computing to develop a common background on cloud computing impact on the various internet stakeholders.

Then, in Section 3.5, we provide descriptions of a variety of cloud services offered by well-known cloud operators. This description is highly useful to understand the architecture and operation of the cloud-based overlay applications in Chapter 4, as well as to define realistic scenarios of cloud service provision and identify possible stakeholders corresponding to real operators in Chapter 5. Additionally, in Section 3.6, we provide an overview of various sources on the implications of traffic generated by data centres and cloud, both as a share of global IP traffic, and in terms of absolute traffic volumes, popularity and future trends, whereas in Section 3.7, we overview several studies that deal with characteristics of energy consumption by data centres and cloud, ISPs' networks, and users' mobile devices. Sections 3.6 and 3.7 contribute to the qualitative assessment of the considered cloud-based applications performed at the end of the next chapter.

Chapter 4 focuses on the applications’ side. In particular, the first part of the chapter describes categories of applications provided over the cloud based on the type of service offered. Two specific well-known, representative example applications for two significant categories, i.e., YouTube for video streaming, and Dropbox for online personal storage, and their traffic implications are further addressed in the Appendix. Moreover, apart from well-established applications, we also address some less known emerging ones. The reason is that they can constitute nice examples for evaluating traffic management mechanisms where intervention potential is higher than other well established proprietary applications where intervention may not be applicable. Finally, in Section 4.10, we perform a qualitative analysis of the considered overlay applications given the description of their operation, traffic and energy characteristics, potential for intervention and potential for optimization provided in Chapter 4, and taking into account the traffic and energy considerations of Sections 3.6 and 3.7.

Chapter 5 focuses on stakeholders and their relationships in cloud computing environments. We overview selected tools and methodologies in literature for stakeholders’ identification and for analysis of their interests, relationships and potential conflicts; then, we apply some of these tools to identify the major stakeholders in four scenarios of cloud service provision of high interest to SmartenIT. The outcome of this analysis will serve as input to WP2 to identify incentives of stakeholders that need to be addressed by the mechanisms to be designed.

Chapter 6 summarizes the entire deliverable and draws our major conclusions firstly on the characterization of traffic generated and energy consumed by cloud-based services and applications, and secondly on the analysis of stakeholders’ relationships in the aforementioned setups. Finally, Chapter 7 briefly addresses the SMART objectives, as described in SmartenIT’s Description of Work (DoW) [204].



3 Traffic & Energy Characteristics of Cloud Computing

This chapter provides definitions of significant terms that will be employed in the rest of the document and in the project in general. Additionally, main service and deployment models of cloud are described, and cloud services provided currently by well-known cloud operators are briefly overviewed. Furthermore, traffic characteristics of inter data centre communications and generally traffic generated by cloud services and applications is reported, while also energy implications related to data centres, ISPs' networks and end-users' mobile devices are also addressed, based on information acquired from real cloud service providers and the literature.

3.1 Cloud Basics

The SmartenIT project addresses traffic generated by new overlay applications provided over the cloud, as well as inter-cloud / inter data centre communications facilitating such overlay applications. SmartenIT's target is to design and develop mechanisms addressing the incentives of all involved stakeholders so as to enable their collaboration in order to efficiently manage the cloud traffic towards a mutually beneficial situation.

However, in order to develop a common understanding of the meaning of significant terms such as overlay, cloud, data centre, etc., basic terms which will be used in this document and throughout the entire project are defined in this section. Additionally, we clarify what the major differences between cloud, data centre and CDN are, and identify the extra capability provided by the cloud that distinguishes it from the grid.

Generally, an application is a set of software components operated to perform a detailed

task within a specific environment. Overlay applications are all kinds of applications that are provided to the end-users as the outcome of a service provided by an overlay network, or simply overlay; examples cover peer-to-peer applications, e.g., peer-to-peer applications, cloud applications, or social networks.

The term overlay is ambiguous; it refers both to the overlay network and the overlay application, while it implies that that network's or application's operation is performed on top of, i.e., "lays over", another network. Specifically, any overlay network is a logical communications network that operates on top of another network called the underlay, e.g., the physical network, or the Internet, and it is typically a user or a customer of that underlay. In our analysis below, we tend to use the term underlay and physical network interchangeably. While one overlay node corresponds to one physical node, logical links may correspond to one or more physical links, i.e., a physical path; while a logical path definitely corresponds to multiple physical links. An example of an overlay network, s a

social network; a social network represents an Internet platform where typically a large number of users form a community inter-connecting themselves in friendship relations. Typical functions of social networks are the following. Every participant has a profile where personal information is located including hobbies and personal interests. Furthermore, users create groups according to their interests and share information relevant to the groups. This can for example be links to web pages or video on the respective topic. In the

SmartenIT context, social awareness links social information such as social network structure, users’ preferences and behaviours, etc. to network management mechanisms. This means that such mechanisms exploit the information contained in such networks in order to perform efficient network management and traffic optimization.



The SmartenIT project focuses on a major category of overlay applications, the cloud

applications; these include applications that are provided 'over the cloud'; an application is considered to be provided over the cloud, when a part of it or the entire application is built upon some cloud service, e.g., storage, computation, which are provided with remote resources physically distributed on several variable places. Cloud resources can be accessed with different levels of insight, which have been formalized by the National Institute of Standards and Technology (NIST) usage models. Users may simply run an e-mail application running in a Web server, build up an application using the cloud programming and networking environment and storage resources, manage such an environment for its applications. Moreover, the cloud concept has roots in fixed distributed infrastructure such as Grid Computing, accessed remotely by a restricted population such as scientists, bank and other business support employees. This legacy has been formalized and broadened to serve all types of population and cover all types of usage model with various degrees of application resources flexibility and security.

According to the NIST definition [1], the

'Cloud Computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and devices) that can be rapidly provisioned and released with minimal management effort or service provider interaction'.

Cloud Computing is a specialized distributed computing paradigm, as it differs from traditional distributed computing approaches in that 1) it is massively scalable, 2) can be encapsulated as an abstract entity that delivers different levels of services to customers outside the cloud, 3) it is driven by economies of scale, 4) it offers flexibility; i.e., the services can be dynamically configured (via virtualization or other approaches) and delivered on demand, and 5) it provides the capability to transfer workload to other engines/data centres, i.e., outsourcing. For instance, a well-known example of cloud computing is e.g., Amazon's Elastic Cloud Compute (EC2) platform which provides resizable compute capacity in the cloud (see Section 3.5.1).

Constituent elements of the cloud are data centres. A data centre is a facility used to house large computer, telecommunications and storage systems and their components (e.g., servers) that besides backup power and communications also provides environmental facilities, e.g., air conditioning. Usually, data centres consume enormous energy amounts for operation, and cooling.

Furthermore, some other significant categories of overlay applications include:

A. Content Delivery Networks: A CDN distributes and disseminates content to the edge of the access networks across Internet Regions. A CDN employs a large, distributed system of strategically deployed data centres, i.e., servers, over the Internet, aiming to store replicas of the content “close” to end-users and redirect users’ requests to one of the closest copies, according to a combination of selection criteria. Hence, end-users can access the requested content, with less delay and timeouts, while traffic is reduced as content is not delivered from the origin content server. Well-known examples of CDNs are Akamai [2] and Limelight [3]. The cache cluster for support of Google YouTube can be seen as an internal or content owner provided CDN as described in detail in Section 4.1.1. Moreover, large ISP often set up CDNs within their domain.

B. Peer-to-Peer networks (P2P): P2P networks are computer networks where tasks and workload is partitioned among the network nodes called peers without the



need for central coordination by servers or stable hosts [4]. P2Ps have become increasingly popular especially to support file sharing applications, e.g., BitTorrent [5], KaZaA [6], and Voice-over-IP (VoIP) applications e.g., Skype [7].

C. Virtual Private Networks (VPNs): A VPN extends a private network, e.g., the network of an enterprise, and the resources contained in the network across the Internet [8]. It enables a host computer to send and receive data across the Internet as if it were within a private network with all the functionality, security and management policies of the private network. This is accomplished by establishing a virtual point-to-point connection through the use of dedicated connections, encryption, or a combination of the two. VPNs include mainly proprietary solutions; additionally, a popular type of VPN is the IP MPLS VPN employing the Multi-Protocol Label Switching (MPLS) to prioritize, transport and route the VPN traffic over links in the public network.

As mentioned in the DoW [204], SmartenIT addresses cloud traffic, i.e., traffic generated by applications and services provided over the cloud. Moreover, based on the cloud services classification performed within Task T1.1, and the investigation of the various cloud application categories for the purposes of Tasks T1.2, T1.3 and T1.4, legacy CDN architectures, i.e., non-cloud ones, fall out of the scope of the project. Therefore, due to the fact that services provided by CDNs and special characteristics of the latter practically constitute a subset of the services and characteristics of the cloud computing itself, we try to explicitly clarify below the differences between the cloud and the CDN model, taking into account various criteria related to the physical infrastructure, its ownership and operation, the content and servers location, the distribution level, and others. Moreover, we address differences of the cloud to a data centre and cloud's predecessor, the Grid.

3.1.1 Cloud and Data Centre

Data centres provide their customers, i.e., Service Providers, CDNs, Application Providers, with specific servers, network, and storage. On the other hand, clouds provide their customers with abstracted versions of these components that are not tied to a specific data centre: virtual servers, virtual storage, and virtual networking. Being an abstraction of data centres, clouds are a representation of the very notion of location independence and

virtualization; this is a technique that allows the sharing of one physical server among multiple Virtual Machines (VMs), where each VM can serve different applications, while the CPU and memory resources can be dynamically provisioned for a VM according to the current performance requirements. Clouds provide resources reasonably reliable and redundant, while where or how these are hosted is variable and dynamic. Reliability and redundancy comes from cloud providers using multiple data centres, so clouds almost certainly span one or more data centres, but themselves are more than data centres.

3.1.2 Cloud and Grid

In [9], an extensive comparison between cloud computing and grid computing is performed taking into account multiple aspects including business models, architecture, resource management, programming model, application model, and security model. As stated in [9], cloud has evolved from grid, and relies on it as its backbone and infrastructure support. The evolution has been a result of a shift in focus from an infrastructure that delivers storage and computation resources (as in grids) to one that is economy-based aiming to deliver more abstract resources and services (as in clouds) in a

http://en.wikipedia.org/wiki/Point-to-point_(network_topology)



dynamic manner. Major conclusions of this comparison are that cloud and grid share common vision and similar architecture and technologies, though they significantly differ in terms of security, programming model, business model, compute model, data model, applications, and abstractions.

3.1.3 Cloud and CDN

Comparing Cloud and CDN is not an easy task, due to the fact that many alternative deployments of both might exist depending on special environments and perspectives within the general notion of Cloud computing and CDNs, respectively. For instance, there are global and local CDNs, or CDNs owned by a dedicated entity and CDNs deployed by ISPs, etc.

As aforementioned, traditional CDNs aim at improving the performance of websites and content delivery services by deploying an extensive network of geographically distributed data centres worldwide. They were initially built to cache and serve static content, but have adapted to the increasing needs of the content delivery market and manage to serve dynamic content using dynamic content acceleration techniques such as content pre-fetching, as well as on-the-fly compression techniques. A CDN is a network with static topology, and whose internal structure is transparent to its users. CDN servers’ locations are disseminated to DNS name servers and users’ requests are redirected via DNS to the most appropriate stored replica based on a combination of IP-based proximity and performance criteria, e.g., server condition, achieving low response times, high throughput, and service availability. CDNs display high content availability, reliability and QoS, whereas their main disadvantages include high operational and maintenance costs, difficult and expensive allocation of new resources to the network.

On the other hand, cloud computing is deployed in a dynamic manner and offers a wider range of services, such as storage, and computation, software and content delivery, whereas users of a cloud have access to cloud services via a specific interface made available by the cloud designer/operator. Cloud providers manage their infrastructure and services in one or more data centres, deployed worldwide based on economic, energy efficiency and often geographical criteria. In principle, cloud computing relies heavily on the sharing of resources and services, aiming to achieve lower costs and scalable performance. Users’ requests are redirected towards the most appropriate server based on internal procedures and policies, mostly dependent on servers’ and private network’s conditions. Advantages of clouds comprise of scalability, cost-efficient performance, service availability, hardware independence, dynamic allocation of new resources and virtualization. Their disadvantages include security and privacy issues in the sense that users' personal information, e.g., data and settings, are stored in their premises.

3.2 Cloud Models

In this section, we describe briefly main deployment and service models for cloud computing, which will constitute the background to describe in next sections (see Section 3.5) real cloud deployments, as well as to describe interesting application scenarios in Chapter 5.



3.2.1 Deployment Models

Depending on who owns and manages the cloud infrastructure, the following four deployment models are identified by NIST [1]:

i. Private cloud: The cloud infrastructure is provisioned for exclusive use by a single legal organization comprising multiple consumers (e.g., business units). It may be owned, managed, and operated by the organization, a third party, or some combination of them, and it may exist on or off premises. Private cloud deployment, w.r.t a public deployment, can provide cloud services with enhanced security features as dictated by user requirements, higher availability as resources are dedicated only to a certain group of users, and higher flexibility due to the fact that services can be tailored as per specific user requirements.

ii. Community cloud: The cloud infrastructure is provisioned for exclusive use by a specific community of consumers from organizations that have shared concerns (e.g., mission, security requirements, policy, and compliance considerations). It may be owned, managed, and operated by one or more of the organizations in the community, a third party, or some combination of them, and it may exist on or off premises.

iii. Public cloud: The cloud infrastructure is provisioned for open use by the general public. It may be owned, managed, and operated by a business, academic, or government organization, or some combination of them. It exists on the premises of the cloud provider.

iv. Hybrid cloud: The cloud infrastructure is a composition of two or more distinct cloud infrastructures (private, community, or public) that remain unique entities, but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load balancing between clouds).

Moreover, the relationships among different cloud operators, considered as different administrative entities, are ruled by the following definitions:

Single Cloud Model is the most typical model, followed by big cloud operators (i.e., Google, Amazon). who deploy different data centre infrastructures in geographical diversity, with connectivity on Tier-1 and Tier-2 domains. Connectivity among different data centres is provided based economic agreements with ISPs, but no agreements with other cloud operators are established.

Federated Cloud model is the model in which smaller cloud operators (with connectivity on Tier-2, and Tier-3 domains) join together to form a federation (a sort of super-cloud entity), in order to achieve economics improvements in terms of increasing capacity to serve more end-users and enlarging the platform of offered services. One major advantage of the federated cloud model is the possibility for smaller cloud operators to use resources located in data centres that belong to other cloud operators who have joined the federation. From user perspective, the differences between the cloud operators joining the federation are transparent. One of the major concerns of this model related to the protocols used to establish the federation as well as issues about user identity, security and privacy.

Interconnected cloud model is the last model of interaction among cloud operators. It’s similar to the previous model except for the fact that no federation is formed. The Interconnected Cloud Model foresees that each cloud operator maintain its administrative role, while also establishes economic agreements with



other partners to achieve service mobility, i.e., offload its computing and hosting capacity, with the final aim to ensure proper QoE to its own customers.

3.2.2 Service Models

According to [1], there are three major service models for the cloud which are defined w.r.t. the capabilities offered to the end-users:

1. Software as a Service (SaaS)

SaaS is defined in [1], as "the capability provided to the consumer is to use the cloud provider’s applications running on a cloud infrastructure. The applications are accessible from various client devices through either a thin client interface, such as a web browser (e.g., web-based email), or a program interface. The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings".

The deployment model can be either public, e.g., e-mail service such as Gmail, or private, e.g., a hosted product as in the case of Customer Relationship Management (CRM) services delivered to large enterprises.

2. Platform as a Service (PaaS)

PaaS is defined as "the capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages, libraries, services, and tools supported by the cloud provider. The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage, but has control over the deployed applications and possibly configuration settings for the application-hosting environment" [1].

PaaS is typically priced based on a utility billing model, so users pay only for what they use. Because PaaS provides a complete hardware and software infrastructure, it eliminates the need to invest in hardware or software and offers significant infrastructural savings. Some of the features that can be included with a PaaS offering may include: operating system, server-side scripting environment, database management system, server software, support, storage, network access, and tools for design and development.

The deployment model applied for PaaS can be either public, e.g., Google App Engine, Windows Azure, or private, e.g., a hosted product provided by Interoute to an enterprise customer.

3. Infrastructure as a Service (IaaS)

IaaS refers to "the capability provided to the consumer is the provision of processing power, storage, networks, and other fundamental computing resources, where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, and deployed applications; and possibly limited control of selecting networking components (e.g., host firewalls or machines with specific characteristics)" [1].

IaaS products are mainly offered over a private cloud deployment model to business customers who want to offload their IT structure.



Apart from their deployment and service model, the offered cloud services can be also classified based on their type to 7 major categories; these are as follows:

Dynamic servers which combine the flexibility and virtualization of cloud computing with dedicated hardware and computational power of traditional dedicated servers. Dynamic servers allow users to create VMs and to configure them by scaling up or down, while using private resources on dedicated hardware.

Content Delivery which involves content dissemination (through file sharing or video streaming) to multiple end-users with high availability and high performance, e.g., YouTube for video streaming.

Cloud Storage which is a model of networked online storage where data is stored in virtualised pools of storage usually hosted by third parties, e.g., Amazon's Simple Storage Service (S3). In fact, physical storage resource may span multiple servers.

Hosted email which is a model of email service provided by a mail-server that runs on the cloud, e.g., in the email-provider's or a third-party's data centre. Thus, the hosted email account is accessible by a 'light' client or the web browser, e.g., Gmail.

Hosted desktop which is a service within the broader cloud-computing sphere, delivered as a combination of hardware virtualization and processing capability. Processing takes place within the cloud providers data centre environment, while different works or tasks are allocated by the cloud provider, transparently to the end-user, to different servers.

Hosted telephony which refers to VoIP telephony where the server handling or terminating calls is on the cloud. Voice services over the cloud allow users to expand their options beyond local or regional carriers.

Finally, application engines which allow end-users to develop and run/host applications in cloud operator-managed data centres. Applications are then 'sandboxed' and run across multiple servers.

3.3 SLAs of Cloud Services and Indicative Parameters

A Service Level Agreement (SLA) is a formal definition of the relationship that exists between a service provider and its customer. An SLA is used to specify what the customer could expect from the provider, the obligations of the customer as well as the provider, performance, availability and security objectives of the service, as well as the procedures to be followed to ensure compliance with the SLA.

As for other managed network services, the typical SLAs between a cloud service provider and its customers should encompass at least the minimum and maximum service levels across all tiers of service provisioning. This should regulate:

Planned and unplanned downtime,

The process in which the cloud provider will notify the company of planned downtime, and mechanisms to accept or defer,

Contractual penalties for any unplanned downtime suffered outside of the agreed SLA,



Agreements for events, which the cloud provider has no control over; for example, natural disasters at the cloud provider’s data centre,

Operational Level Agreements (OLAs), which record engagement details between the cloud provider support teams including contact details during business and non-business hours, and

Definition of individual support tiers contained within OLAs, including individual responsibilities for service, process and delivery timeframes.

In addition, service providers should be able to offer SLAs based on application and user requirements. Moreover, in a context in which cloud operator provides services to a business customer the SLA may include application requirements like response time, application availability, minimum time to repair and issue resolution. Finally, in the context in which ISP offers connectivity services to a customer such as a cloud or data centre owner, an SLA may include customer requirements like upper bound for delay and latency, packet loss rate and minimum guaranteed throughput.

3.4 Financial / Economic Implications of the Cloud

Cloud computing provides some strong benefits to mid-sized and big companies who decide to outsource IT capabilities to the cloud. Main advantages of the IT migration to the cloud are summarized below.

Organizations that transfer their IT workload to the cloud enjoy reduced IT costs by transitioning from capital expenses (CAPEX) such as enterprise software licenses, and servers and networking equipment to more fluid, less expensive operational expenses (OPEX). Additionally, by leveraging on cloud services, the organization essentially pays for what it needs, when it needs it, following a pay-per-use model, whereas scalability provided by the cloud promotes a company's faster growth.

Moreover, the cloud allows businesses to rapidly alter products, processes, and services to align with shifting customer preferences, as well as the ability to respond quickly to changing both customer needs and market demands. Because complexity is veiled from the end-user, a company can expand its product and service sophistication without also increasing the level of user knowledge necessary to utilize or maintain the product or service. Furthermore, cloud’s ability to store more information, due to its expanded computational power and capacity, provides greater opportunities for customization of products and services. Because of cloud, businesses can create a more customer-centric experience by adapting to slight changes in a user-defined context.

Nonetheless, companies can achieve economies of scale increasing volume output or productivity with fewer people, monitor projects more effectively, and reduce costs for energy consumption, since the physical hardware is operated in the cloud.

In terms of efficiency achieved with cloud computing solutions, a study from [193] shows that when looking at organizations running their own data centre infrastructure we hypothesize that only 20% of the time and effort that goes into running applications, where all business value is concentrated. The diagram in Figure 3-1 illustrates the extent to which routine and non-core tasks, like patching operating systems and performing backups, impact upon the time of IT departments.



Figure 3-1: Application of 80/20 rule to Company operating his own data centre [193].

Cloud computing is a force that helps flip this ratio and gives IT departments the ability to spend 80% of their time and money on core business processes, like business application design.

3.5 Cloud Services and Real Cloud Deployments

In this section, we overview and briefly describe cloud services provided by well-known cloud operators, some of them also known as Internet Giants, e.g., Amazon, Google, Microsoft, etc., as well as services provided by Interoute (IRT), based on which most (if not all) cloud-based applications are built, e.g., Dropbox employs both Amazon's EC2 and S3 to provide its personal online storage service. This description is highly useful to understand later the architecture and operation of the cloud-based overlay applications in Chapter 4, well as to define realistic scenarios of cloud service provision and identify possible stakeholders corresponding to real operators in Chapter 5.

3.5.1 Amazon

Amazon Web Services (AWS) provides a complete set of cloud computing services that enable you to build sophisticated, scalable applications. Below, the most significant web services are summarized.

Amazon Elastic Compute Cloud (Amazon EC2) [170] is a web service, in particular a PaaS solution, that provides resizable computational capacity in the cloud. Furthermore, EC2 reduces the capacity scaling time, and allows paying only for capacity that is actually being used. Last but not least, Amazon EC2 provides tools to build failure resilient applications.

Next, Amazon Elastic Block Store (Amazon EBS) [171] provides block level storage volumes for use with Amazon EC2 instances. The storage volumes can be attached to a running Amazon EC2 instance and exposed as a device within the instance. Amazon EBS is particularly suited for applications that require a database, file system, or access to raw block level storage.

Another important web service offered by Amazon is Amazon's Simple Storage Service (Amazon S3) [172], which provides a web services interface that can be used to store and retrieve unlimited data on the web. Amazon S3 is an IaaS solution and aims to make web-



scale computing easier since it claims that the developer has access to the same infrastructure that Amazon uses to run its own global network of web sites.

Finally, CloudFront [173] is Amazon’s CDN, seamlessly integrating with Amazon’s cloud services (Amazon S3 cloud storage service, EC2 cloud computing platform, etc.). The service was initially deployed in November of 2008, after the release of the S3 cloud storage service in 2006, which led to massive amounts of data delivered to end-users worldwide. A CDN approach, optimized to work with AWS, was adopted to improve scalability and performance. Content creators or providers are able to access and manage the service either through AWS’s website or CloudFront’s exposed API, initially declaring their content servers which might be either within (e.g., an S3 storage server) or outside (e.g., provider’s private cloud or external cloud storage solution) Amazon’s AWS. In addition, the service provides a namespace, and subscribed providers can share their content to their websites or dedicated applications. The load balancing and redirection mechanisms handle the end-users’ requests and redirect them to the closest content replica. Besides traditional HTTP and HTTPS distribution, since 2012 CloudFront also supports streaming distributions using the Real Time Messaging Protocol (RTMP). The Amazon CloudFront Global Edge Network consists of around 40 data centres in 14 countries worldwide and offers reliable and fast content delivery. Tests conducted using the Compuware Corporation performance network, indicated that CloudFront was 10% faster than the average latency of 3 major traditional CDNs (Akamai, Limelight and Level3) for 1 MB objects and 20% faster for 12 KB objects [10]. In addition, results from “Last Mile” tests indicate CloudFront’s comparable performance to Akamai’s CDN.

3.5.2 Google

In addition to the well-known search and mail services, the hyper giant Google offers a considerable number of other cloud services including office software, storage, computation, web application deployment and even data mining and translation tools. In the following we give a very brief overview of them with information mainly taken from the developer web page of Google [148].

The Google App Engine offers to deploy and run web applications on the infrastructure of Google. This permits the applications to scale with increasing user demands without setting up new servers. Such tasks are automatically handled by the app engine, which balances the load within the Google-owned infrastructure. However, application developers need to base their applications on the application framework provided by Google. The framework supports application development in Java, Python and the Go programming language.

The Google Compute Engine can be classified as a PaaS offer, where customers can rent virtual Linux machines. The machines are priced on a per usage basis and can have different numbers of virtual CPU cores and varying amounts of memory and disk space. Within the virtual machine the user can install and run basically every piece of software that is available for the given operating system. At the moment, CentOS and Google Compute Engine Linux are supported operating systems.

Google Cloud Storage also targets at application developers and enables them to store, move, retrieve and download data and share them with other users within the Google cloud. The data is identified with unique keys, which are in turn organized in buckets. The service is based on a RESTful architecture, which is stateless and defines clients that send requests to store data at servers, i.e., in the Google infrastructure, or retrieve them



from there. Note that Representational State Transfer (REST) is a programming paradigm for web applications, which incorporates the idea that the URL of a website already defines the content of the website. Then, access control in the Google Cloud Storage is possible using Google account email addresses for example. Finally, it should be noted that this service is different from Google Drive, which targets end users and not application developers.

Google BigQuery and Google Cloud SQL facilitate to manage relational databases and to execute SQL-like queries on these databases or even on public or shared data sets. The focus of Google BigQuery is on the execution of database queries on very large data sets. The structure of such data sets is limited to a single table, even if this table might be huge. If entire relational databases are required, Google proposes to use Google Cloud SQL. This service is based on MySQL and provides all features that relational databases typically have. The main difference is that Google Cloud SQL runs in the Google cloud.

Furthermore, Google offers machine learning and data mining tools to application developers. As possible use cases, Google mentions the prediction of movies that a user might like based on the movies he watched in the past. Another use case is the identification of spam emails. This service is called Google Prediction API.

The Google Translation API service is able to translate text from one language into another one covering 64 different languages. Like all aforementioned services, it is charged on a per usage basis and build based on a RESTful architecture. In addition, the service also offers language recognition, i.e., it can detect the language of a given text.

Finally, the services Google Docs and Google Drive target at home and business users instead of application developers. These services provide office tools such as word processing, table calculation or presentation tools to the users and permit the collaborative editing and sharing of such documents. Therefore, they can be categorized as software-as-a-service (SaaS).

3.5.3 IBM

IBM’s SmartCloud [147] is a line of enterprise-class cloud computing technologies and services which was launched in April 2011. It can be used for building private, public and hybrid clouds. IBM Smart Cloud offers all three types of cloud services, IaaS, PasS and SaaS, as the following three product scopes:

SmartCloud Foundation consists of software and hardware infrastructure. It gives needed functionalities to set up clouds on virtualized hardware, manage virtual machines and monitor the performance of virtual and physical environments;

IBM SmartCloud Application Services is a PaaS environment for development and deployment of applications to the cloud. It provides cloud-based development tools, workload patterns, middleware and databases;

IBM SmartCloud Ecosystem is a set of new services for software vendors to support and improve their products. It helps their clients to adopt cloud models and manage cloud-based actions.

3.5.4 Microsoft

Microsoft declares that its cloud services create a correlated family of solutions and are dedicated to small businesses in order to make available resources which are scarce for



companies using Microsoft products and being familiar with applications [118]. Cloud services are split into two categories: as “commercial” (12 services, e.g., Windows Azure, Windows Azure Platform Appliance, SharePoint Online, Office Live Meeting …) and “consumer” (10 services, e.g., Windows Live, Windows Live Hotmail, Windows Live Messenger, Bing, Zune, MSN, Xbox Live, etc.).

Windows Azure [119], delivered 1 February 2010 is a cloud services platform and infrastructure, for building, deploying and managing applications, storing data across a global network of Microsoft-managed data centres. It provides both PaaS and IaaS, and supports many different programming languages, tools and frameworks. Windows Azure allows to easily scale the applications to any size. It is available in multiple data centres around the world, enabling to deploy the applications close to the customers. As an open platform, Windows Azure offers choices to developers. It allows them to use multiples languages (.NET, PHP, Ruby, Python or Java) and development tools (Visual Studio or Eclipse) to build applications. The Windows Azure supports multiple popular Internet standards and protocols including HTTP, XML, SOAP and REST.

The Windows Azure CDN [120] is Microsoft’s CDN solution, to support its cloud computing platform, i.e., Windows Azure. It was available since the beginning of the cloud platform’s deployment in 2008, targeting the caching of Windows Azure content in strategically-placed physical servers around the world in order to improve content delivery, performance and scalability.

Content creators and providers may register and access the service via the Windows Azure website, where they can either create their personal cloud storage space within Windows Azure services or define their external sources (other cloud storage services or other CDNs). By enabling the CDN service, any publicly available content will be cached to the CDN nodes and redirected to end-users, upon request. In addition to static content, the Windows Azure CDN caches dynamic content from Azure-hosted applications, enabling hosted applications to scale and perform better. It also provides all the necessary tools for multiple programming languages to create applications that use their services.

24 nodes in 17 countries worldwide constitute the Windows Azure CDN network. Each data centre contains more than 300,000 servers [15], organized in 400-2500 server containers, including racks of 96 servers with estimated power of 16 kilowatts per rack [16]. The Dublin data centre is estimated to generate up to 5.4 megawatts of critical power, with the potential to expand to a total of 22.2 megawatts of critical power. Windows Azure ensures that the service has an uptime 99.95%.

The Windows Azure SQL Database is a highly available, scalable, flexible database service hosted by Microsoft in the cloud [158], [159], [160]. It is offered by Windows Azure platform, formerly known as SQL Azure Database. It is relational cloud databases built on the Microsoft SQL Server and provides a high-level of interoperability, enabling customers to build applications using many of the major development frameworks. The latest release of SQL Databases was completed in November 2012 and offers several enhancements.

The Windows Live [161], launched 1 November 2005, is a collection of online services and applications from Microsoft for individuals. It combines advantages of Windows system with cloud computing, provides online services (e.g., Windows Live Hotmail, Windows Live SkyDrive), software applications (e.g. Windows Live Mail, Windows Live Messenger, Windows Live Movie Maker) and search services (e.g., Bing). As official statistics show Microsoft over 330 million registered users.



The Microsoft Zune [162] is a free digital media platform, launched by Microsoft. It provides a multiple services (e.g., Zune Application, Zune Media Player, Zune Xbox Music Pass, Zune Marketplace, Zune Software).

3.5.5 Interoute

IRT owns and operates 8 data centres across Europe, directly connected through the Interoute network. This pan-European platform is designed for the delivery of enterprise IaaS and virtualized services. Each data centre is deployed with a network fabric deeply embedded in the IRT’s infrastructure to support effective access, geographic fallback and full product range.

IRT’s Virtual Data Centre (VDC) is a multi-tenant, IaaS platform that offers to the customers on-demand computing and cloud hosting with integrated applications. It creates the ability to offer either private or public cloud computing, offering public simplicity with private cloud security. Each of the components selected by a customer for its VDC can be provisioned and bound to its existing IRT VPN network, a private VDC network or an Internet routable network. Moreover the customer is allowed to subscribe to two or more IRT sites for resiliency purposes. Figure 3-2 illustrates IRT’s VDC both for public and private deployment models.

Figure 3-2: Interoute’s VDC.

From a customer perspective, IRT’s VDC is seen as a pool of compute and storage resources available on an Interoute provided network and hosted within geographically separated zones; additionally, the customer is presented with a VDC Control Centre graphical http-based interface, i.e., the “Hub”, which allows operations such as configuration of services, backups reporting, and device performance reporting. After logging on the portal, VDC resources are instantiated for the customer who is allowed to perform the following operations: create storage for VMs, select image, create storage volume for VMs, and further deployment operations.



The VDC product and delivery relies on a modular and scalable design that is vendor agnostic. Each of the components selected are physically connected and configured in a standard organized group, referred to as a “Pod”. A pod must contain all three components: compute, storage and network. Each pod is integrated into the existing platform by the “Orchestration” software. Thus, a VDC can be defined as 4 main components:

1. Cloud Orchestration: Software that orchestrates the delivery and management of VMs for a customer/service provider, and the interfaces into the system (VDC Console & API),

2. Compute: Provides CPU and Memory resources through the use of Type II Hypervisors on commodity hardware,

3. Storage: Provides shared disk resources for hosting virtual appliances as well as block storage to the appliances,

4. Networking: Provides pre-provisioned multi-tenanted network and connectivity between all components, customer external and internal LANs and the Internet.

IRT leverages on a suite of IT infrastructure that enables the customers to configure proper services in terms of hosting, computational capacity, security which is totally automated providing fast service provisioning, and rapid delivery model.

3.6 Traffic Characteristics of Clouds and Data Centres

In this section, a brief survey on overall IP and cloud traffic analysis and trends is presented based on data mainly taken from Cisco Virtual Networking Index [23], Global Cloud Index [20] as well as other publicly available statistics. Additionally, we address and compare traffic statistics and the evolution of traffic generated by inter data centre and inter-cloud communications, while finally, we characterize inter data centre communication based on major services categories and critical factors affecting it. Moreover, the cloud traffic survey provided in this section will provide input to the overall assessment of cloud-based applications to be performed in the next chapter, Chapter 4.

3.6.1 Global IP Traffic Statistics and Trends

The development of IP traffic over the last decade is visible in official statistics issued by administrative bodies of several countries, e.g., for Germany [17], Australia [18], Hong Kong [19], and as well as periodical announcements on traffic trends by vendors of routing and measurement equipment, e.g. the well-known reports by Cisco [23] and Sandvine [21]. Additionally, measurement results from links on TDG’s IP platform reveal major differences in the traffic mix over a daily load profile and in the up-/downstream direction [22]. Consequently, meaningful traffic statistics should indicate if results are valid for busy hours or as mean value over weekdays or also including weekends, as well as where the statistics has been taken from (region(s), organization(s), core/aggregation network/LAN, up-/downstream).

Figure 3-3 summarizes the outcomes of the aforementioned studies and illustrates the common main trend, indicating traffic growth by a factor of the order 100 for fixed IP network traffic over the past decade, although there are different phases of steep and slower annual increase. Moreover, traffic in mobile networks is included (dashed lines), which currently accounts for only a few percentage of the total IP traffic but is catching up



at higher increase rates of 50%-150% per year as compared to 25% - 50% annual increase as reported from fixed networks.

1

10

100

1000

10000

100000

1000000

2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011Year

IP T

raff

ic G

row

th S

tati

stic

s [P

etaB

yte

/Yea

r] Global Internet, Cisco VNIN-America, Cisco VNIW-Europe, Cisco VNIGermany, BNetzAHong Kong, OFTA StatisticsAustralia, Bureau of StatisticsGlobal, Cisco VNI, Mobile IPGermany, BNetzA, Mobile IPHong Kong, OFTA, Mobile IP

Figure 3-3: Trends in IP traffic growth reported from different sources.

3.6.2 Data Centre Traffic Statistics and Tends

The cloud traffic statistics presented in this section are derived based on aggregated statistics related to services delivered to end-users via both CDNs and clouds [20].

Although the Internet is forecast to reach the ‘Zettabyte’ era in 2016, the data centre has already entered it. While the amount of traffic crossing the Internet and IP WAN networks is projected to reach 1.3 ZBs (i.e., 10

9 TBs) per year in 2016, the amount of data centre

traffic is already 1.8 ZBs per year, and by 2016 will nearly quadruple to reach 6.6 ZBs per year. This represents a 31% Compound Annual Growth Rate (CAGR). The higher volume of data centre traffic is due to the inclusion of traffic inside the data centre (typically, definitions of Internet and WAN stop at the boundary of the data centre). Figure 3-4 shows the growing rate of global data centre traffic including traffic generated within the data centre by 2016. Data centres traffic can be broadly categorized into three main areas w.r.t. the destination of the traffic flows, i.e.:

Traffic that is generated and remains within the data centre,

Traffic that flows from data centre to data centre, and

Traffic that flows from the data centre to end users through the Internet or IP WAN.

The portion of traffic residing within the data centre will remain the majority throughout the forecast period, accounting for 76% of data centre traffic in both 2011 and 2016. Factors contributing to traffic remaining in the data centre include functional separation of application servers, storage, and databases, which generates replication, backup, and read/write traffic traversing the data centre. Furthermore, parallel processing divides tasks and sends them to multiple servers, contributing to internal data centre traffic.



Figure 3-4: Global data centre traffic growing rate [20].

Figure 3-5: Global data centre traffic by destination [20].

Table 3-1 summarizes the expected evolution of traffic by data centres from 2011 to 2016, where data centre traffic is categorized by destination (i.e., within data centre, inter data centre, etc.), or segment (business vs. consumer), or type (traditional vs. cloud).

We observe that traffic within data centres is about 4.5-fold higher than traffic originating from data centres towards the end-users, and 11-12-fold higher than traffic between data centres, i.e., inter data centre traffic. The volume of 299 Exabytes (EBs) (i.e., 106 Terabytes (TBs)) of data centre to user traffic makes already most of 360 EBs of the global IP traffic [20], [23]. Thus, it can be inferred that the Internet traffic is dominated by more than 70% by traffic from data centre to user; remaining 25-30% of the global IP traffic is generated by P2P applications, whereas other important applications’ share like VoIP is negligible. Global data centre IP traffic is forecast to nearly quadruple and grow by 31% CAGR over the next 5 years.

Table 3-1: Data centre traffic trends for 2011-2016 [20].



Moreover, the three categories of traffic within, between and from the data centres to the users are estimated to grow at about the same pace, whereas virtualization is foreseen as a main driver of the traffic between data centres provided by the same or different entities, e.g., cloud operators. Therefore, a main challenge for energy efficient traffic management lies in an optimized distribution of data over traditional and cloud data centres with regard to short transport paths from data centre to the end-user and between data centres themselves. Note that traditional data centres are defined in [20] as those whose operation is associated with non-cloud consumer and business applications.

Furthermore, w.r.t. segment and type categorization, it can be observed that consumer traffic is driven by CAGR of around 30%, while a lower growth rate of 23% is assumed for business traffic, such that consumer traffic for the mass market of private Internet access subscribers will exceed business traffic 6-fold in 2016. Moreover, a comparison of cloud and traditional data centre traffic attributes an essentially higher growth of 44% for clouds, whereas only 17% for traditional data centres. As a consequence, clouds are expected to overtake traditional data centre traffic almost 2-fold until 2016.

3.6.3 Cloud Traffic Statistics and Trends

The Cisco Global Cloud Index forecasts the continued transition of workloads from traditional data centres to cloud data centres. Due to the workload transmission, the traffic generated by clouds is expected to scale at a higher rate of 44% CAGR compared to 31% for global data centre traffic, while traditional data centre traffic growth is significantly smaller (see Figure 3-6). Global cloud IP traffic is forecast to increase 6 times and grow by 44% CAGR over the next 5 years. Moreover, cloud traffic crossed the Zettabyte threshold in 2012, and by 2016, nearly two-thirds of all data centre traffic are expected to be based in the cloud. Significant drivers of cloud traffic growth are the rapid adoption of and migration to cloud architectures, along with the ability of cloud data centres to handle significantly higher traffic loads.

Cloud data centres can be distinguished in two categories based on services delivered to the end-users, i.e., business and consumer ones. Business data centres are typically dedicated to organizational needs and handle traffic only for their company's own needs that may adhere to stronger security guidelines. On the other hand, consumer data centres typically cater to a wider audience and handle traffic for the mass consumer base.



Figure 3-6: Cloud vs. traditional data centre traffic growth 2011-2016 [20].

Considering forecasts of cloud traffic generated by data centres, consumer data centres' traffic is expected to grow with a CAGR of 46%, and to reach 3,659 EBs by 2016, while business cloud traffic is expected to grow at a CAGR of 37%, increasing to 596 EBs by 2016. Table 3-2 summarizes details for global consumer and business cloud traffic CAGR.

Table 3-2: Cloud IP traffic trends 2011-2016 [20].

Concerning the business segment, the necessity to provide fast and flexible access to large data archives is an important objective for IT organizations considering cloud-based solutions. In addition, enabling advanced analytics to tap into the wealth of information contained in largely unstructured data archives can create a valuable competitive business advantage. Moreover, enhanced collaboration services delivered through the cloud can increase employee productivity and customer satisfaction.

On the other hand, regarding the consumer segment, applications such as video and audio streaming are strong factors in cloud traffic growth, while emerging services such as personal content lockers are also gaining in popularity. In personal content lockers, i.e., online storage applications such as Dropbox, users can store and share music, photos, and videos through an easy-to-use interface at relatively low or no cost. Especially related to personal cloud storage, Cisco GCI [20] forecasts that personal cloud traffic will increase from 0.6 EBs annually in 2011 to 25 EBs in 2016, at a CAGR of 111%. Furthermore, the proliferation of tablets, smart phones, and other mobile devices which allow access to such applications in a manner convenient to the end-user, further contribute to the high growth rate of the global cloud traffic.



Finally, Ipoque statistics [141] have also been considered in order to have a wider overview of internet traffic characterization; though since they are pertaining only up to 2009, they will not be considered as relevant for our current investigation.

3.6.4 Characteristics of Communication between Cloud Data Centres

Traditional network infrastructure between data centres is organized in a hierarchical structure which is composed of at least three levels: access level, distribution level and core level. In this structure, the typical traffic flow is north/south-based, i.e., from server racks (access layer) to L3 high capacity core switches (acting as root bridges for spanning tree protocol). However, within the last years, data centres' inter-connection is undergoing a deep shift towards new design due to the rise of a number of issues, i.e.,: rapidly rising traffic volumes associated with data distribution and synchronization operations, data transfer operations in service contexts such as content delivery or storage, high variability of bandwidth requirements between peak and mean traffic rates, and increased number of east/west traffic flows (i.e., server-to-server). The changing in the actual data centre inter-connection is led by the rising of new services that the new generation of data centres must be able to provide, e.g.,:

Storage and data replication for IT disaster recovery strategy based on a geographical diversity approach. Such services initiate relatively small number of flows or connections, though they generate very high traffic throughput per flow.

VM migration, with VM disk space either asynchronously replicated to the new data centre or accessed from the original data centre over the ISP's WAN.

Synchronous replication between data centres (active-active storage replication), which allows data to reside at multiple locations and to be actively accessed by VMs at all sites.

Within a single ISP and data-centre provider, inter data centre communications generally rely on traditional L2VPN or IPMPLS/VPN services. The new services offered imply a massive amount of data moved across multiple and geographical distributed data centres. The peak traffic volumes between data centres are dominated by background, non-interactive, bulk data transfers due to backup and replication applications running to transfer bulk data between distributed data centres.

Additionally, the inter data centre communication is significantly affected by some critical factors; one of which is the bandwidth allocation of the underlying transport structure. Most cloud operators use a fixed bandwidth allocation (mainly tailored on an upper limit of their transmission needs), which may result in non-optimal resource allocation since the variability of bandwidth requirements between peak and mean traffic rates is rapidly increasing over the years. The actual mitigation actions mainly deal with load balancing over multi-path, multi-hop Store-and-Forward (SnF) scheduling techniques.

Apart from bandwidth, latency is also a crucial metric for the east/west traffic. Many Internet services and application provided over the cloud are deployed in highly distributed environments where data are stored in different locations. Usually they have a limited time window to fetch requested data, process them and send them back to the requesting entity. Delay and other problems related to bandwidth limitations may disqualify such services and applications.

Furthermore, another issue to be considered is related to cloud traffic patterns due to virtual switching operations. These operations are performed by hypervisors to forward the



traffic between VMs and external network switches. It is highly efficient (i.e., in terms of low latency), when VMs communicate with each other or external switches without involving external switching. However, there are also some drawbacks; for instance internal switching consumes CPU that may limit capability to host more VMs. Nevertheless, the most significant drawback which is also related to SmartenIT’s scope is the inconsistency of internal switching with network management and monitoring of external network infrastructure by network providers; such an inconsistency may lead to conflicts in policies and security issues.

Finally, the rapidly growing number of mobile devices (e.g., smartphones and tablets) used to access new overlay applications has an important influence on how application ecosystems are implemented nowadays. For all devices that contain only thin graphical user interfaces, all heavy operations and data storages are placed in the cloud, while the allocation of resources is transparent for users. Because of shifting the workload toward the cloud infrastructure, computation needs as well as the power consumption are optimized. However, high bandwidth Internet access is required for the end-users’ mobile devices, while the shifted workload further burdens the cloud, and highly affects data centre to user, as well as inter data centre communication.

3.7 Characteristics of Energy Consumption by Cloud Services

Energy efficiency is fundamental as a key objective of SmartenIT; i.e., we aim to design traffic management mechanisms that will achieve low energy consumption, or else energy efficiency in data centres, ISPs' networks or in mobile devices. In order to address the energy efficiency of cloud, we first need to capture the current state. Thus, the following sections will analyse the energy consumption issues related to three main areas, i.e., data centres, ISPs' networks, and mobile devices, describing also measurements methods at different stages and energy wasting, thus highlighting possible points on and opportunities for intervention for SmartenIT purposes. Moreover, the energy consumption survey provided in this section will provide input to the overall assessment of cloud-based applications to be performed in Chapter 4.

3.7.1 Data Centres

Cloud computing has proven to be a highly scalable and cost-effective solution for the ICT industry. Hence, cloud service providers are making significant investments in data centre infrastructure so as to provide a wide range of cloud services, business and commercial applications for their customers. However, clouds are composed essentially of data centres, which are being built at ever-larger scales and with increased server density, resulting in greater energy consumption. High energy consumption implies though high energy cost, which constitutes a significant part of a data centre's operating cost as reported in the next section, leading thus to an erosion of cloud providers revenues. As expected, cloud operators are highly interested in energy efficiency techniques to minimize incurred costs, and maximize their benefit.

Nevertheless, the energy efficiency in data centres and clouds is being investigated by various EU-funded research projects as energy awareness is one of the priorities of the European Union. Some of those that may also provide valuable input for SmartenIT are presented in Table 3-3; some of the solutions proposed by the projects presented below are also briefly overviewed in Deliverable D2.1 [203], while Energy Efficiency is also to be addressed in Deliverable D1.2.



Table 3-3: Summary of initiatives for energy efficiency in data centres.

Project name Goal of the project Description

GAMES - Green Active Management of Energy in IT Service centres

Energy-efficient adaptive data centres

The project works on energy efficiency along with QoS and IT resource utilization and performance.

FIT4Green - Energy-aware ICT optimization policies

Allocation of ICT resources in data centres to minimize energy consumption.

The project has created an energy-aware layer of plug-in on top of the current data centres’ management tools.

CoolEmAll Tools and knowledge to enable data centre designers and operators to plan and run facilities more energy efficiently.

The project works on advanced simulation, visualization and decision support tools along with blueprints of computing building blocks for modular data centre environments

All4Green Energy saving policies and energy demand patterns in data centres.

The project works on a definition of suitable SLA including energy-efficiency aspects. A key element of the investigated architecture, procedures and functionalities is the collaboration (interface) with energy producers/providers for energy consumption optimization.

Thus, energy consumption is a critical concern for IT organizations worldwide as the cost of operating data centres increases due to the growing use of computing devices and rising energy costs. In the sections below, we provide measurements, future projections and break-down of the energy consumption by data centres. Additionally, and we address potential for improvement, i.e., reduction of the energy consumed due to typical data centre operations.

3.7.1.1 Measurement Studies

In 2006, data centres consumed in the US 1.5% of the total electricity [151]. Moreover, it was estimated that the energy consumption would double in the next 5 years. The reason for this prediction is an increasing use of Internet services such as online banking, or media services. However, a new study [152] from 2011 shows that the energy consumption for data centres did not increase as predicted. In [153], although the IP traffic was expected to grow by 46% per year, the energy consumption is estimated to increase by up to 36% in the US in 5 years, from 2005 to 2010. The main reasons according to [152] are the use of virtualization that leads to better hardware utilization. Furthermore, the economic turbulences in 2008 led to a fewer investments in new servers, and the overall electricity usage did not grow as fast as predicted in the US. Figure 3-7 depicts the predictions by [151] and [152] (indicated with red arrows).

Another observation by [152] is that Google, although having many servers, only contributes 1% to the electricity used by data centres worldwide. However, since Google does not publish the number of their servers, these numbers are estimations. One reason for this low value is that Google assembles its own servers and can optimize its servers and data centres to reach higher infrastructure efficiency. Infrastructure efficiency is typically measured by Power Usage Effectiveness (PUE), which is the ratio of total amount of power used by a data centre to the power used by the IT equipment. Google's data centres have a PUE of 1.13 in 2012, while a value of 1 is ideal [153].



Figure 3-7: Predicted US electricity use for data centres from the EPA report to Congress [151] and the range estimated by Koomey [152].

According to [139], power is wasted at different stages of data centre operation. A typical breakdown of data centre energy overheads is (also depicted in Figure 3-8):

Network-Critical Physical Infrastructure (NCPI) equipment is responsible for approximately the 70% of total data centre energy consumption; 45% of which is due to cooling infrastructure and 25% for power delivery units.

The remaining 30% of total energy is delivered to IT equipment for critical operations such as hard disk drive operations, networking, memory, CPU, motherboards, fans, etc.

3.7.1.2 Potential for Improvement of Energy Efficiency in Data Centres

In [139], it is stated that PUE of data centres in 2006 was on average around 2.0, namely 2.0 times more energy is consumed in total by a data centre than the amount of energy delivered to IT equipment. Moreover, in [139], it was suggested that the energy consumption can be saved by continuing the trend of consolidation and virtualization, enabling power management on servers and equipment, and improve data centre cooling.

Typically, facility equipment power load can be monitored, installing energy meters into the data centre. They return the total electrical energy utilization by the units to which they are intended for. The meters are usually located at the distribution panel supplying power to the Precision Air Conditioning PAC units.

Facility networking in a data centre, is widely acknowledged as a promising technology for energy saving. For this reason, another value which must be controlled is the network energy consumption. It is associated to the network infrastructure cabled inside the IT Service Centre, composed of switches, routers, etc. In this case, the kind of data which must be monitored could be the traffic rate of the network, its maximum and minimum speed, along with the measurement of the energy consumption by means of appropriate physical sensors.



Figure 3-8: Breakdown of data centre energy overheads [139].

The recently emerged cloud computing paradigm leverages virtualization technology and provides the ability to provision resources on-demand on the pay-as-you-go basis. Organizations can outsource their computation needs to the Cloud, thereby eliminating the necessity to maintain their own computing infrastructure.

Virtualization perfectly addresses the requirements of energy efficiency in data centres. The resources of a VM can be provisioned dynamically as the workload demands change for an application. Virtualization is the most adopted power management and resource allocation technique used by the data centre operators. It is implemented in both the server and switch domain but with different objectives. Server domain virtualization usually achieves energy efficiency by sharing limited resources among different applications. On the other hand, virtualization not provide for energy efficiency in the network domain. In effect, network resources are burdened by the virtualization techniques, since live migration of VMs between physical hosts generates a significant amount of traffic.

The overall key benefits for a data centre owner can be summarized as follows: energy costs reduction by up to 80%, dynamic power off servers without affecting applications or end-users, and greening of IT infrastructure, while also improving reliability, availability and service levels.

SmartenIT's potential of intervention can be mainly related to the management of traffic generated by live VMs migration from data centre to data centre, migration that is strictly tied to data centre resources orchestration and energy efficiency achievements.



3.7.2 ISPs' Networks

In ISP networks, the energy consumption has increased with traffic volume and popularity of Internet applications and meanwhile is a critical issue for further traffic and bandwidth growth [196]. Fiber optic links provide high bandwidth in fixed networks without electrical power supply but the switching and routing still requires more and more energy. In core routing facilities, the energy demand is becoming a limiting factor for bandwidth and traffic increase, if upgrades of IP transport capacity imply a similar increase factor in capacity as well as in energy consumption [197]. Moreover, mobile communication is also subject to a tremendous increase in traffic and energy consumption with regard to network elements and devices. Section 3.7.3 is discussing the latter area in more detail.

Therefore traffic management and options to reduce the traffic load are crucial as well as energy saving technologies. Higher resource utilization means service for more users based on the same equipment and OPEX, part of which is energy costs [194]. Caching infrastructure within ISPs is useful to shorten traffic paths in content delivery [195]. Their benefits in transport capacity savings and lower delays have to be balanced against required data centre and server resources, which again consume energy. Flexible control and (re-)direction of traffic flows can also reduce over-provisioning. QoS differentiation can help to run a network at higher load and higher risk for congestion while sufficient service quality is still met for the most important and critical applications.

Moreover, there have been proposed options for switching off parts of the network equipment in phases of lower load and reactivating them on demand. This constitutes an another way of energy saving, which is often not feasible currently due to long set up periods of networking equipment. On the other hand peak-to-mean ratios of daily traffic profiles are often in the order of 2 or larger and resilience in core networks is often achieved by doubled network topologies which could be kept in a less hot standby mode at lower energy consumption. When usage is highly varying over time e.g. between office, shops and residential areas or for mobile ad hoc network, hotspots for event sites etc. then the energy saving potential by on demand provisioning is high as well.

3.7.3 Mobile Devices

Measuring energy consumption of mobile devices and smartphones in particular is a challenging task. On the one hand, the built-in hardware, which offers a low precision and sampling rate, might be sufficient in some cases. However, for low-level experiments, e.g., frame-based measurements of the wireless interface, more precise measurement equipment is needed. On the other hand, external measurement hardware is expensive and counteracts the idea of mobility, preventing devices from moving as intended and probably needed by experimenters. Additionally, both methods do not allow to measure single components like the WiFi chip or the consumption of single applications (apps).

In the next sections, we briefly overview methods for measuring energy consumption and provide results of a number of recent measurement studies on the aspect of energy efficiency in mobile devices.

3.7.3.1 Measurement Methods

Recently developed energy models as those described in [98], [99], [100], and [101] try to fill the gap. The common idea of these models is to estimate the current energy consumption of the device based on a number of data sources provided by the Operating System (OS). These sources can comprise the actual CPU frequency, the data rate of the



radio chip, and LCD brightness [98], [99] or are based on system calls, e.g., calls related to network sockets, which are triggered by an application [100], [101]. Thus, the estimate can be split up to components and apps, allowing a precise assessment of code quality with respect to energy consumption.

The model parameters are obtained by correlating the data sources and the actual energy consumption as measured by external measurement hardware and are thus device-specific. However, in test beds with homogeneous resources, energy models provide the unique possibility to measure a larger number of devices without the need of additional hardware. In [102], a system is proposed to measure, collect, and visualize the energy consumption of P2P streaming processes of mobile devices in a precise manner at runtime, using the model presented in [99][99]. Figure 3-9 shows the interaction of the developer and measurement platform, where the lower half is mapped to a central server entity collecting the data from a number of Android devices and allowing statistical evaluation using a web frontend.

Figure 3-9: Interaction of measurement platform and developer [102].

3.7.3.2 Measurement Studies

Energy is a scarce good on mobile devices. Thus, a number of recent measurement studies focus on the aspect of energy efficiency. The following paragraphs discuss the relevance of the consumption of network related hardware components. This is done in order to substantiate the necessity of taking energy efficiency into account, when designing network related algorithms as planned within the scope of SmartenIT.

Besides CPU, RAM and GPS, the network interface is particularly relevant, as mobile users heavily rely on network access. In [103], a study of the user behaviour of 255 users measures nearly 50% of all user/smartphone interactions being related to communication, i.e., to email, SMS, instant messaging and voice calls, not even including browsing, i.e., 12%, and media consumption, i.e., 9%. The relationship of energy-efficiency and usage pattern is not evaluated in this study. However, in [98], the power consumption of week-long traces of 20 users is broken down by components using a model-based measurement approach. The average aggregated consumption of networking components (WiFi, edge, voice) accounts for roughly 25% of the overall consumption (see Figure 3-10), while the maximum is at ~40%.

A non-user centric insight into the energy consumption of smartphones can be given by analyzing models to estimate the energy-efficiency as described beforehand. These have shown communication related hardware to belong to the main consumers of energy ([99], [98], [100], [101]).



Figure 3-10: Power breakdown of 20 users by hardware component, taken from [98].

Figure 3-11: DFA representing the WiFi energy consumption as modelled in [100].

The models presented in [98] and [99] are rather simplistic linear regression models. The authors find the WiFi/GSM chipset to be a main contributor. In [100], [101], and [102], the energy consumption of a complete smartphone is modeled and validated, using a system call based approach. An interesting aspect of this work is the modeling of wireless networking activity as Deterministic Finite Automatons (DFA); an example is depicted in Figure 3-11. These automatons reflect the interface’s power states, which dominate the energy consumption, while the actual data rate has minor influence. Another interesting finding is the existence of so-called tail states: to avoid the frequent oscillation of power states, the hardware stays in a high power state to for a defined amount of time even though no data is sent [101].

3.7.3.3 Methods for Improvement of Energy Efficiency of a Network Application

Currently, the literature on methods for energy-efficient network access on mobile phones can be divided roughly into two categories. Offloading/prefetching approaches try to minimize energy consumption by doing a handover between transmission technologies. These approaches utilize the tradeoff of transmission range and energy consumption: cellular technologies like HSDPA or LTE offer wide range communication at a high price in terms of energy, whereas low range technologies like WiFi or Bluetooth have a comparably low energy-footprint and can often provide higher data rates. An example of such an approach is given in [104].



Figure 3-12: Energy consumption of browser search; the green indicates the percentage of energy wasted in tail states [100].

As opposed to offloading approaches, hardware-aware approaches embed knowledge of hardware properties into the application layer, e.g., power models of the WiFi or 3G chip. In [100], several applications are studied for energy consumption in tail states. Depending on the application, this amount is remarkably high (e.g., for Google search on a browser, about 30% of energy is wasted in tail states, see Figure 3-12). Sophisticated aggregation schemes as presented in [103] can help to avoid the frequent occurrence of these states.

3.8 Summary of Cloud Characteristics

In this chapter, we provided cloud basics, i.e., definitions of major terms related to cloud services and applications. Additionally, we described fundamental deployment models and economic and QoS aspects of cloud computing, to develop a common background on the impact of cloud computing on the various Internet stakeholders. As a next step, in Section 3.5, we overviewed a variety of cloud services offered by well-known major cloud operators, as the so-called Internet giants, such as Amazon and Google, as most cloud-based applications are built upon those cloud services, e.g., Dropbox employs both Amazon's EC2 and S3 to provide its personal online storage service. This description will allow us to understand the architecture and operation of the cloud-based overlay applications described in Chapter 4, to define realistic scenarios of cloud service provision, and last to identify possible stakeholders corresponding to real operators described in Chapter 5.

Next, in Section 3.6, we overviewed various sources addressing traffic generated by data centres and cloud both as a share of global IP traffic, and in terms of absolute traffic volumes, popularity and future trends. The main conclusions and future projections are as follows: Global data centre IP traffic will nearly quadruple, while it will grow by 31% CAGR over the next 5 years. On the other hand, global cloud IP traffic will increase 6 times and grow by 44% CAGR over the next 5 years, while global cloud IP traffic will account for nearly two-thirds of total data centre traffic by 2016. Moreover, the rapidly growing number of mobile devices (e.g., smartphones and tablets) used to access new overlay applications constitutes a key driver of the cloud traffic increase. Specifically, it has an important influence on how application ecosystems are implemented nowadays, namely, applications 'run' in the cloud (i.e., computation, data storage), while only a 'shell' providing an interface to access the application remains at the mobile device.

Then, in Section 3.7, we overviewed studied addressing the characteristics of energy consumption by data centres, ISPs' networks and end-users' mobile devices. Our major



conclusions for data centres are as follows: Although the IP traffic was expected to grow by 46% per year in the last 5 years, the energy consumption for data centres did not increase as predicted. In particular, it has been estimated to have increased by up to 36% in the US, due to the use of virtualization that leads to more efficient hardware utilization. Moreover, currently, only, 30% of total energy is delivered to IT equipment for critical operations such as hard disk drive operations, networking, memory, CPU, motherboards, fans, etc., implying PUE equal to 3.3. On the other hand, the remaining 70% of total data centre energy consumption splits to 45% for cooling infrastructure and 25% for power delivery units.

Concerning ISPs' networks, the energy consumption has increased with traffic volume and popularity of Internet applications and has become a critical issue for further traffic and bandwidth growth. Fibre optic links and equipment consume more energy than older equipment for switching and routing, while caching infrastructure within ISPs, which is increasingly adopted to shorten traffic paths in content delivery, leads to installation and operation of small data centres and server resources in ISP's premises, which again consume energy. Moreover, energy saving by switching off parts of the network equipment in phases of lower load and reactivating them on demand is often not feasible currently due to long set up periods of networking equipment.

On the other hand, considering energy implications on users' mobile devices of all user/smartphone interactions have been recorded as being related to communication, a) email, SMS, instant messaging and voice calls, nearly 50%, b) browsing, i.e., 12%, and c) media consumption, i.e., 9%. Proposed approaches that aim to optimize the consumed energy utilize the trade-off of transmission range and energy consumption. For instance, cellular technologies like HSDPA or LTE offer wide range communication at a high price in terms of energy, whereas low range technologies like WiFi or Bluetooth have a relatively low energy-footprint and can provide higher data rates.

The investigation in Section 3.6 and Section 3.7, along with the studies performed in Chapter 4 addressing specific characteristics of various application categories, e.g., online storage, and specific examples of them, e.g., Dropbox, will allow us to assess the various cloud applications (in the end of Chapter 4) w.r.t. to their suitability to be selected as SmartenIT's key applications, whose traffic is to be handled by mechanisms to be developed in WP2.



4 Characteristics of Cloud Applications

In this chapter, we provide an overview of a multitude of overlay applications, both highly popular and well-established and new emerging ones, which all share the characteristic, i.e., they are provided over the cloud, namely their entire functionality or part of it is based on some cloud service such as cloud storage or cloud computation. Specifically, we describe these applications from multiple viewpoints, in terms of their operation, traffic characteristics and energy consumption. Additionally, taking also into account the terminal, i.e., the device of the end-user, allowing the access to such cloud-based applications, we explicitly address categories of applications that are specific for mobile devices, such as mobile video streaming and mobile P2P file sharing.

Our ultimate goal is to highlight relevant to SmartenIT characteristics of new overlay applications, i.e., related to the main concepts and scenarios of applications, traffic generated by them, and potentially the social information that can be extracted from them. These characteristics will play fundamental role in the selection of those applications that are to be addressed by SmartenIT (in WP1), the description of interesting scenarios and use-cases, the design of new socially-aware, energy-efficient traffic management mechanisms (in WP2), and the development of SmartenIT's architecture (in WP3).

Identification of traffic characteristics, mainly referring to traffic volume generated, as well as traffic patterns, e.g., variability and burstiness, and QoE requirements, where available, is an important task done permanently by network operators and administrators. Besides theoretical aspects of traffic analysis, carried in order to study network-related phenomena and to follow evolution of services and user behavior, such analysis helps to solve urgent problems (such as failures or lack of resources) and support traffic management and long-term network planning. There are long-lasting efforts to characterize traffic in the Internet. These investigations profit from previous and ongoing efforts to characterize traffic in the network. The goal is to recognize traffic flows, characteristics of single flows and also characteristics of multiplexed traffic composed of many flows.

The set of applications presented below is representative of the contemporary overlay and social networking and in some cases such applications were used by SmartenIT partners for thorough investigations, based on studying application architecture, service scenario, possibility to selectively monitor or register relevant traffic flows for immediate inspection and also for off-line analysis. The description of an application should give a comprehensive view of its main features, popularity, opportunity for intervention and optimization. Defining such application’s features more precisely we aim at describing its:

Category: to which service/application it belongs, e.g., online storage, search engine, etc.

Operation: how the application works, what other cloud services it employs,

Popularity, dynamics: known data about evolution, number of users, perspectives.

Openness: open code, possibility of intervening,

Potential for SmartenIT intervention/optimization: jointly consideration of data and details concerning application with openness of source code (or protocol) suggestion possibility to influence the service.

Applicability for SmartenIT: expression of interest for SmartenIT with some hints about level of relevance and the way of influencing the application performance.



4.1 Video Streaming

According to Cisco traffic measurement studies [23], video traffic constitutes 51% of global consumer IP traffic in 2011, and it is forecast to reach 55% of global IP traffic by 2016. Moreover, according to recent studies (e.g., [23], [68]), mobile video streaming traffic accounts for up to 50% of the aggregated traffic in mobile networks at peak hours in North America and the Asia-Pacific region. In Europe, its share is lower with around 35% of aggregated traffic. In all three regions, YouTube is either leading the ranking in total downstream traffic or is among the top three traffic sources. This does not include video exchanged through P2P file sharing. The sum of all forms of video (TV, Video on Demand, Internet, and P2P) will be approximately 86% of global consumer traffic by 2016. In particular, it is said that it would take over 6 million years to watch the amount of video that will cross global IP networks each month in 2016. Every second, 1.2 million minutes of video content will cross the network in 2016.

In this section, we focus on Internet video streaming, which was measured to generate up to 72% of total video consumer traffic in 2011, where both short, i.e., UGC and video clips up to 7 minutes, and long videos, i.e., generally videos exceeding 7 minutes in duration, have been considered. Moreover, video streaming traffic will constitute up to 46% of total consumer video traffic by 2016. Note that although video streaming traffic is forecast to present positive CAGR, i.e., 34% for short video and 22% for long, the share w.r.t. to total traffic will be reduced due to the fact that other types of video traffic will increase with much higher CAGR, i.e., 45% for Internet to TV traffic, and 90% for mobile video traffic.

4.1.1 YouTube

In this section, we provide a description of the functionality, traffic characteristics and QoE implications of YouTube, as the most popular application both for PCs and mobile applications, which generates enormous traffic volumes in the Internet. Moreover, an in-depth analysis of YouTube is provided in Appendix A.

YouTube is a video streaming platform and its portal (i.e., YouTube.com) is one of the main Internet portals for Video-on-Demand (VoD), as well as one of the most popular sites globally. Currently, only Google (i.e., Google.com) and Facebook (i.e., Facebook.com) are more popular than YouTube according to the Alexa Ranking [24]. The video content that is available on this platform is generated by users. This means that YouTube does not distribute its own videos. Instead, it offers a platform where users can upload videos and share them with their friends or the public.

This service is free of charge for the users, i.e., every user can upload and watch the videos without any costs additional to the Internet access. Therefore, a large number of people and companies upload short videos clips, for example product advertisements. In contrast, currently popular movies or songs are not available at YouTube due to copyright protections. For that purpose, other VoD portals exist, which charge a fee for their service and for the copyright owners.

In the YouTube service, three different stakeholders are involved. These are the YouTube platform owner, the ISPs and the users. While the ISPs and YouTube aim at increasing their profit, the users are interested in a high QoE when watching the video. The ISPs sell connectivity to their customers and this is their main source of revenues. However, YouTube (or Google) try to establish peering agreements with other ISPs so that they can exchange traffic without additional fees.



The user experience when watching a video might be degraded by long startup delays or stalling events [55]. Therefore, these two aspects are the main QoE indicators. In order to guarantee a high QoE, the connection between YouTube servers and end-user clients’ needs to support the required bandwidth. YouTube videos are mostly watched in 360p resolution (which is the default setting) and have a bit-rate of 0.1 and 1 Mb/s. Since YouTube is a VoD service, latency and jitter or not so important. The reason is that the client software tries to download the parts of the video up to 60 seconds before they are watched.

Recently, YouTube gets also more and more popular on mobile devices; in particular, according to [21], 27.8% of all YouTube traffic is consumed on a smartphone or tablet. There are YouTube apps for the Android operating system as well as for Apple’s iOS. In such mobile scenarios, the users might accept some quality degradations compared to watching the video on their computers at home. However, energy efficiency becomes a major issue for the QoE since users can get dissatisfied if watching a video clip on their smartphone drains the battery very fast.

Possible points of intervention for optimizing the YouTube service are rather limited since the whole system is proprietary. From outside, it is neither possible to change the sending behaviour of YouTube nor to change the distribution of videos, i.e., which videos are stored on which servers. Furthermore, the client playback software is not open source and cannot be modified. Hence, the only point of intervention is the delivery path of the videos. On this path, the ISPs can intervene, e.g., by caching popular videos or by smoothing the traffic. The latter one could also be performed by the TCP/IP stack of the operating system but this is rather a complicated task.

Energy efficiency plays mainly a role in the YouTube data centres and on mobile user devices. However, there is only little information available about the concrete structure of YouTube data centres and how their energy efficiency could be improved. Therefore, it might be difficult to investigate energy efficiency issues in the context of YouTube.

4.1.2 Netflix

Netflix is a major American streaming video provider, operating in both Americas, and a few European countries [189]. Over 23 million users account for 29.7% of peak downstream traffic in the United States, making it bigger than YouTube.

Netflix architecture includes data centre, services that track registration process, hold payment information and redirect users to respective servers for other Netflix functions. Amazon cloud handles many key functions like sign-in process, content ingestion, CDN routing, Digital Rights Management (DRM), logging and mobile device support. For main functionality – content delivery, three CDNs are used: Akamai, Limelight and Level-3. To receive and watch video content, a dedicated player Silverlight is used.

Data streaming is performed by Dynamic Streaming over HTTP (DASH) protocol. The protocol encodes video content and splits it into a few second-long chunks. Each download is measured in order to determine appropriate quality for the next fragment. While video streaming is done by CDNs alone, the user experience is reported periodically (once every 60+ seconds) to a control server within the Amazon cloud

Choice of a CDN is based on a ranking in a manifest file received over SSL and is independent of content type, length or location. Preference seems to depend on user account and can remain unchanged for days [188]. Once the CDN is selected it remains



unchanged, even if a different CDN could offer a higher bandwidth, change occurs only in case when bandwidth drops below the threshold value that can support a lowest quality video stream. While it reduces frequent changes in network topology it seems that bandwidth usage is sub-optimal. This can be seen as opportunity for SmartenIT solutions.

4.1.3 Hulu

Hulu is a video streaming service that operates in United States and Japan [191]. It offers a variety of video content ranging from movies and TV shows to web series, trailers and behind-the-scenes materials from popular TV networks. Majority of Hulu download is the free video streaming for desktop users, but paid subscription service is also available, it enables HD quality of videos and support of transfer to mobile devices.

Most of the video content is streamed at 480kbps and 700kbps, but some of the videos can be streamed at 1000Kbps. HD quality is limited to subscribers of HuluPlus. The users are assigned to one of three CDNs (Akamai, Limelight or Level3), then IP address of server is selected via hostname and DNS. Content is delivered via encrypted Real Time Messaging Protocol (RTMP) on port 1935 – when using Level3, or tunnelled over HTTP (RTMPT) – when using Akamai, Limelight or when port 1935 is blocked.

The choice of a specific CDN depends on the manifest file, obtained from s.hulu.com, which contains arranged preference list, server location, available bitrates and other information. There are very frequent preference changes between three CDNs but they are unrelated to the content, user location or time slot. It appears that the preferences are randomized in a way that directs 25% of traffic to Limelight, 28% to Akamai and 47% to Level3 [190]. Preferences of CDN are subject to frequent changes. Change of the CDN takes place only when the currently used one cannot support even the lowest requested quality.

Once a CDN is selected, a specific server must be selected. All CDNs serve desktop users while mobile users are served only by Akamai and Level3, finally Hulu advertisement service is served only by Akamai and Limelight. All CDNs use locality-aware DNS resolution, so general hostname (e.g., cp39466.edgefcs.net for Akamai hostname will return one of 1178 IP addresses depending on user location).

It is important to note that CDN infrastructure and operation involves significant costs and necessary expertise. That’s why Hulu, like many other service providers, rely on third-party content distribution networks. Three CDNs used by Hulu are used by other online video services with the same streaming technologies.

Hulu is already managed to some degree, by influencing user preferences towards CDNs. That can limit SmartenIT potential influence. However energy efficiency aspect may be of some interest.

4.1.4 Relevance for SmartenIT

As already pointed out in the beginning of Section 4.1, video streaming traffic constitutes a large share of total consumer traffic, and thus an excellent candidate for SmartenIT to address. Specifically, YouTube, as the most prominent example of this category of applications, is provided over a proprietary platform, thus a closed system, we need to consider possible points of intervention for SmartenIT and the potential for optimization for such traffic.



Since video streaming has such a large share of traffic, the prior target for optimization is traffic management. As stated in the previous subsections the streaming service providers YouTube, Netflix and Hulu already use CDNs to efficiently distribute their content. Therefore the traffic optimization potential has to be investigated for each platform. Although the CDNs bring content already close to users, local caches at ISPs, hybrid peer-assisted distribution of popular videos or intelligent replica placement strategies could further reduce global video streaming traffic and reduce transit costs for ISPs. Therefore the current Internet structure with reduced upload bandwidth of end-users and peering agreements with access networks have to be considered.

As described in [188] Netflix directs video requests to CDNs independent of content type, length or location. Hence, there is still a high potential for optimization, since the type of content determines the propagation by social media and locality is not taken into account. Both Hulu [190] and YouTube (see Section 4.1.1) already use locally aware DNS-resolution. Hulu uses third party CDNs for content distribution, whereas YouTube uses primarily the global CDN of Google, but also offers ISPs to integrate YouTube caches in their premises.

The options to intervene in the video delivery of YouTube are limited, if there is no collaboration with YouTube. Without access to the content delivery network of YouTube, intervention is for example possible for the ISPs carrying the data or in the operating system of the clients. For ISPs possible interventions could be to prioritize the video traffic over file downloads, where the QoE is less sensitive to bandwidth variations. In addition, dynamic resource management strategies might be possible. Concretely, ISPs could monitor the YouTube traffic flows and estimate the current fill state of the playback buffer at the client based on the knowledge about the YouTube flow control detailed in A.3. In case that this buffer is about to become empty, the ISPs can dynamically allocate more resources to this particular flow and avoid stalling in this way. Furthermore ISPs could exploit information of social networks to predict the demand of specific content and optimize the hit-rates of their local caches.

The QoE findings could be exploited in the following way. In case of high network load and scarce bandwidth resource, the initial delay (which is less QoE critical) could be increased in order to avoid the QoE critical stalling during the video. However, this requires the cooperation of the YouTube player and the servers as well as the ISP. Finally, in the operating system of the client a possible point of intervention might be to change the TCP flow control.

4.2 Mobile P2P Video Streaming

The streaming of video data in mobile environments, in comparison to fixed network access to this data-intensive type of content, imposes special challenges for the content delivery process. Over the last years, the delivery of video or more general of real-time entertainment data has seen a large adoption in this environment. As mentioned in Section 4.1, mobile video streaming traffic accounts for up to 50% of the aggregated traffic mobile traffic at peak hours in North America and Asia-Pacific region, and 35% in Europe.

Moreover, for the Asia-Pacific region [68] documents an additional important contributor to mobile traffic, namely the usage of P2P-based streaming applications. Here, the upload traffic is dominated by PPStream, while regarding the download traffic, PPStream is beyond the top three contributors [69] (see Table 4-1). Other examples of widely used P2P-based streaming applications are PPLive [70], CoolStreaming [71], and SopCast [72].



Table 4-1: Top 10 applications in the Asia-Pacific region at peak time [68]

In this section, we focus on two prominent examples of mobile P2P video streaming which are among the top three traffic contributors in this category, i.e., PPLive and PPStream.

4.2.1 PPLive

PPLive along with PPStream, a very similar application which is introduced in the next subsection (see Section 4.2.2), is one of the most well-known applications for streaming video in a P2P fashion. Both are based in Asia and are available for both streaming to PC as well as mobile devices. The user interfaces are mostly in Chinese and so is a large part of the delivered content. This might be two important reasons why both applications see a rather limited adoption in other parts of the world and currently have a mostly Asian customer base. In the following, major characteristics of, first, PPLive are discussed to understand general requirements and implications of P2P-based streaming applications that could be used to define according traffic measurement and management mechanisms in the context of SmartenIT. In the next Section 4.2.2, PPStream is discussed as it has become a major contributor to the upload traffic of mobile networks, as introduced before (see Table 4-1).

Spoto et al. [73] conducted a measurement study of the PPLive network and showed that during their measurements in 2008 on average 1 million users were connected to the network every day. PPLive offered several hundred different channels at this time, including traditional broadcasting channels and scheduled videos. For each channel of the network, an own P2P mesh-based overlay was constructed among the active participants. As PPLive uses a proprietary protocol, not all mechanisms of the application are understood. In the following, the results of a number of measurements studies, i.e., [74], [75], and [76], are used to shed some light on the internals of the application mechanisms.

For the content distribution, PPLive divides the video streams into blocks that can be requested individually in a pull-based manner. The streams are provided by a content server that injects the video blocks to a channel’s overlay and thereby acts as source of the stream. To retrieve the video stream, each client connects to a number of other clients that it requests from an entity that acts like a tracker similar to those known from content distribution protocols such as BitTorrent. Some of the contacts are used as active contacts to request video blocks from, others are kept as candidates to replace active contacts if



required. The number of contacts depends on the popularity of the channel and the resources of the client itself. With this connection behaviour, the clients form a mesh-like topology, where, potentially, every client can become a major traffic source and act as multiplicator of the video traffic. Managing such client initiated traffic, therefore, can become important for network providers, especially in mobile environments where resources are generally scarce.

Conducting active measurements studies with distributed crawlers running on PlanetLab, Vu et al. [76] observed that clients can react very fast to the arrival or leave of other clients and include or remove contacts to/from their neighbourhood set within less than 15 seconds. They, furthermore, identified three different classes of packets that are used by the PPLive client. One for class with rather small packets that are used for the management of the overlay, one for the exchange of buffer maps to announce video block availabilities with about 800 Bytes per packet, and another one for the actual video data transfer with more than 1000 Bytes per packet. Another measurement study, by Ali et al., [77], shows similar results and compares them to the messages exchanged in SopCast, another P2P-based streaming application.

Hei et al. [78] also conducted active and passive measurements of the PPLive network and identified several shortcomings of the application. First of all, it causes very long start-up delays due to its initial buffering of several ten seconds to mitigate later problems in the distribution process, making channel switching costly in terms of playback experience. Second, clients experience playback lags of up to several minutes, meaning that the playback of different clients watching the same channel can be far away from each other. This usually is an important quality criterion for the streaming of live events where users might want to be as live and in sync as possible.

4.2.2 PPStream

The previously presented characteristics of PPLive streaming traffic (see Section 4.2.1), in general, also apply in the same way for PPStream, a second Asia-based P2P streaming application. Like PPLive, PPStream used a mesh-pull overlay for the data dissemination process. In contrast to PPLive recent studies (e.g., [68]) show that PPStream has become an even more important traffic source in mobile networks. Therefore, in the following, some key characteristics of this application are presented.

In [156], the authors present the results of a measurement study of PPStream that was conducted during the 18 days of the 2002 Olympics in Beijing. They deployed five own instances of the application in different networks and used passive measurements of the generated traffic, focusing on the management traffic of the overlay application. Therefore they captured and analyzed all transferred peer lists, buffer-maps, and data request messages. For their investigations, the authors focus on the playback delay, the data scheduling mechanism and its impact on playback quality, as well as the control overhead of the application. While works such as [157] report an average number of users per channel of 6,000 for an average day, during the opening session of the Olympics about 7 million users were observed in this study.

Furthermore, an interesting observation was made concerning the used transport protocol: In contrast to previous measurements in [155] and observed traffic in regular channels of the application, for the broadcast of the Olympics, it was observed that almost exclusively UDP was used for both video and signaling traffic. In addition, a switch to smaller scheduling units, so called small-chunks of 1KB size (in comparison to 8KB sub-chunks in



regular channels) was observed. While an increase of control overhead due to the necessary additional pull requests was measured, this also reduced the data miss ratio, resulting in a better playback experience for the user. The video bitrate of the Olympic channels was measured in [156] as being 391 Kbps, whereas normal channels of PPStream are reported to be able to have bitrates up to 1024 Kbps [157].

Key findings of [156] are that there are immense playback delay differences between clients for the same channel, even in same local network. Differences of more than 80 seconds or even several minutes were observed for the Olympic channels. A previous study [157] reports an average of 20 seconds for the average day streaming cases. Nevertheless, the findings of [156] show that PPStream uses a mechanism to prefer intra-ISP overlay contacts but does not implement mechanisms for more fine-granular locality information of peers. The authors see a potential for improvement in this case.

The findings concerning large-scale streaming events, such as the Olympics, provide valuable inputs for the understanding of traffic characteristics of popular applications used in fixed as well as mobile environments.


In the context of SmartenIT, it is important to understand the specific traffic characteristics and requirements that such P2P-based streaming systems impose. As Table shows, they cause a large amount of traffic both on the uplink as well as on the downlink of today’s mobile networks. The increased utilization of the uplink is characteristic to the application of P2P techniques as clients not only consume but also contribute bandwidth to the system. It has to be investigated how traffic management mechanisms can be used to support such kind of applications. This is especially interesting for P2P-based streaming in mobile environments where the limited resources and battery lifetime of the devices impose additional challenges.

Liu et al. [79] showed that different mechanisms on client-side can have an impact on the energy-efficiency of the data transmission. In the context of SmartenIT it has to be seen how the traffic management in mobile and fixed networks can contribute to a more energy-efficient streaming process. Here, in analogy to the client side, techniques such as traffic aggregation on network side could be used to avoid long client-side tail states. Furthermore, there is a great potential to improve the energy-efficiency and QoE at the mobile clients by offloading the mobile network. This could be done, e.g., using nano data center [192] concepts to take over parts of the expensive upload process of the mobile device or using direct ad-hoc-based communication between clients. For the latter, support by the mobile network side could play an important role for the discovery and initiation of ad-hoc communications in spatial proximity of a client. In a broader sense, this can be seen as part of the mobile traffic management process for the dissemination of streaming data to mobile clients.

4.3 Online Storage

Online storage applications allow large data transfers using between the end-user's device and the cloud. Generally, online storage services include file-hosting, network back-up, and one-click downloads, however, in this section, we focus on large-scale personal back-up for disaster recovery or remote access, e.g., Dropbox, Google Drive, etc.



According to [174], a significant portion of traffic is associated with online back-up and file storage sites. Until 2011, no single site had secured dominance, as the popularity of services varied between regions and new competitors were still entering the market. Additionally, it was expected that online storage and back-up usage would increase in tandem with smartphone and tablet adoption, with users relying on these services to access their files remotely, while it was concluded that cloud storage has the potential to become a major source of traffic on the internet.

Moreover in [21], the by then low percentage of traffic generated by online storage applications was noted, while in [68] this low traffic share was attributed to the fact that normally, a large and increasing amount of this type of traffic is carried via secure tunnels, so it is frequently indistinguishable from things like mobile banking and other encrypted applications. In fact, on many mobile networks, a scenario was observed, in which a major mobile OS update triggers a step-function increase in levels of SSL traffic by introducing features like file synchronization over SSL. Nevertheless, it was forecasted that online storage and back-up services will see dramatic growth as transparent automated back-ups become commonplace.

Below, we describe certain popular and also some less known applications for online storage which might be of interest to SmartenIT for different reasons, which are highlighted respectively.

4.3.1 Dropbox

In this section, we describe the operation of Dropbox, an online personal storage system. Additionally, we briefly overview the outcome of the measurements reported in [29], whereas an extensive analysis of Dropbox usage on the Internet is given in Appendix B.

Dropbox [27] is a cloud-based file hosting service that offers cloud storage, file synchronization, and the necessary client software for the aforementioned operations. It allows users to create a designated folder on each of their devices, e.g., desktops, laptops or smart phones. In this folder, users are enabled to drop any file, or to upload it through a web browser. Then, Dropbox synchronizes the designated folder so that the same folder and its sub-folders including the same files with the same content appears to the end-user, regardless of the device that the end-user is accessing the folder from. Files placed in this folder are accessible through a website and smartphone/tablet applications.

While Dropbox functions as a storage service, it focuses on synchronization and sharing. In particular, it supports revision history, so that files deleted from the Dropbox folder may be recovered from any of the synched devices. Additionally, Dropbox supports multi-user version control, enabling several users to edit and re-port files without overwriting versions. The version history is kept by default for 30 day for the free version, while an unlimited version is also available for purchase.

The basic object in the Dropbox system is a chunk of data with size of up to 4MB. Files larger than that are split into several chunks, each treated as an independent object. Each chunk is identified by a SHA256 hash value, which is part of meta-data descriptions of files. Dropbox reduces the amount of exchanged data by using delta encoding when transmitting chunks. It also keeps locally in each device a database of meta-data information (updated via incremental updates) and compresses chunks before submitting them. In addition, the client offers the user the ability to control the maximum download and upload speed.



Dropbox also provides some form of localization; more specifically, a technology called LANSync is employed, which allows devices on the same local area network (LAN) to securely download files locally from each other, instead of always 'hitting' the central servers.

Concerning technical details, Dropbox uses Amazon's S3 platform to store the files, and Amazon's EC2 for control and synchronization handling. Synchronization is performed with SSL transfers, while data are stored encrypted via AES-256 encryption. Both the Dropbox server and client are written in Python. Registered by IANA ports for communication are TCP 17500 and UDP 17500 ports.

Even though Dropbox states that files are stored encrypted on their servers, it seems that Dropbox has access to all the files, which makes the statement about encrypted data questionable. This finding is reported in [28], where same file of several MB was added to two different Dropbox accounts. When the file was added to the second account, the Dropbox client generated however only about 16 KB traffic to the storage server, which means that the file content needs to be accessible to Dropbox. Otherwise, a comparison would not be possible.

As with all proprietary cloud application, the control or optimization of such systems is difficult unless we assume collaboration with the cloud provider, which is Dropbox in this case. Drago et al. [29] proposed improvements based on their network measurements, but only the Dropbox software developers and engineers are able to implement such proposals. Hence, although the offered service and the used technology might be of high interest for the SmartenIT project, we can also only propose optimizations, but we are not able to test actual implementations.

Main findings of [29] show that the Dropbox service performance is highly impacted by the distance between the clients and the data centres, which are currently located in the U.S. However, geographically dispersed sources as well as longitudinal data are expected to set the need for distribution of Dropbox's storage servers around the world. The dispersion of Dropbox servers around the globe will though set new challenges to the Dropbox provider, the cloud operator hosting it (i.e., offering storage and control servers) and eventually network providers whose links will be utilized for the inter-connection of Dropbox servers. Additionally, the usage of per-chunk acknowledgments in the client protocol combined with the typically small chunk sizes deeply limits the effective throughput of the service. Nevertheless, as also highlighted in [29], there is space for Dropbox (i.e., the application provider) to optimize both performance and throughput to end-users.

4.3.2 Google Drive

Google Drive is a cloud-based storage service integrated with Google Docs – providing a free cloud-based SaaS office suite: Documents, Spreadsheets, Presentations and Drawings. Google Docs has its own document formats; namely, when a user uploads his own documents to the Google Docs, those documents are converted to the Google’s format. Google Drive was released on April 24, 2012 [121].

Google offers to its customers 5 GB free cloud disk space. A user can get additional quota for adequate monthly fee. Available disk sizes are: 25 GB, 50 GB, 100 GB, 200 GB, 400 GB, 1 TB, 2 TB, 4 TB, 8 TB, 16 TB. When user get extra disk space its Gmail quota increases additionally from 10GB (free account) to 25 GB [122].



Data on the Google Drive can be stored in three ways: user can create documents via Google Drive web panel, user can import documents to the Google Drive via web panel or user can upload files with the Google Drive client software. Google Drive client software also provides files synchronizations. It is available for PCs with Windows, Macs, Android smartphones and tablets and works on iPhones and iPads with iOS 5.0+ [123].

Documents, spreadsheets and presentations created inside Google Drive or imported thought web interface or sent via email can be shared, opened and edited by many people at the same time. User or owner of the document cannot be notified of changes but can see revision history. Additionally, when users cooperate with document they can see that another person also opened the document. Users also can communicate via integrated chat.

Created or uploaded documents, spreadsheets and presentations have some limits. Document can have maximum of 102400 characters and can have maximum 2MB size after uploading and converting to the Google document format. Google spreadsheet can have maximum of 400000 cells with maximum of 256 columns per sheet. After sending to the server can have maximum of 20MB. Presentations can take up to 50MB (about 200 slides). There is no restriction on the drawings but other files (not converted to Google’s format) can be up to 10GB [124].

Google Drive allows user to preview documents online but only files less than 25MB. Supported file types are listed in [125]. Because Google Drive is directly connected with Gmail, users can in simple way (using the Google Drive viewer) preview attachments form email. Additionally, Google Drive allows user to enable free two-factor authentication for better security. To login, user in addition to the regular login name and password must enter short random code sequence generated by the Google Authenticator application from Android or iOS phone or sent via SMS to their phone.

Table 4-2: Typical energy reduction in company after migrating to Google Apps [202][128].

Source of energy

consumption

Energy consumption

today (kWh per employee per year)

Energy consumption

after Google Apps (kWh per employee per year)

Savings from

switching to

Google Apps

Server direct energy 18–175 2–53 70–90%

Server cooling 18–88 2–26 70–90%

Total energy consumed

per user

36–263 4–79 70–90%

Additional cloud-based energy consumption (Google

+ networking)

-- 1–5 2–3% increase

Total required energy 36–263 5–84 68–87%

As Google reported in [202], the migration from typical IT solutions in companies can reduce energy consumption by 68-87%. For example, migration to cloud Google Apps (operating within Google Drive) in the U.S. General Service Administration (GSA) in 2012 has reduced the total number of servers operated by GSA for email and collaboration (Google Docs) from 324 to 61. Therefore, total direct power reduction of GSA servers achieved 87%. Additional cloud-based energy consumption (Google + network) increased by 2-3%. Typical energy saving resulting from reduction of energy used directly by servers and servers’ cooling are presented in Table 4-2.



4.3.3 PiCsMu

PiCsMu (Platform-independent Cross Storage for Multi-Usage) [30] is a cloud storage system implemented with a cloud-based overlay by the University of Zurich (UZH). The built overlay is used in order to store, retrieve, and share files – either private (not sharing) or in a public fashion (sharing). In the private scope, PiCsMu users can upload files to their own use. Just who uploaded can download the file. In the public scope, PiCsMu users can choose if files should be available to a set of other PiCsMu users (private share), or if everyone part of the PiCsMu network can access them (public share).

The system is divided into the overlay and the underlay. In the PiCsMu underlay, PiCsMu users need to add Cloud Services credentials, e.g., Google Picasa, Facebook, SoundCloud, ImageShack, Dropbox, Amazon EC2/S3, Apple iCloud, OwnCloud, or any Cloud Service that provides an API, in order to store files. The overlay holds the index information about files, e.g., in which Cloud Service the file was stored, if the data was encrypted, how the file was split into several small chunks, etc.

The overlay is divided in two parts: private index and shared index. The private index is established in a centralized server and it stores information about private files – i.e., files that are just inherent to one single PiCsMu user. The shared index is established in a P2P network to share files with a set of PiCsMu users (requiring a PiCsMu credential), or for the whole P2P network (thus, not requiring a PiCsMu credential). Within the shared index, PiCsMu application uses encryption (public/private keys) to share files to a limited set of PiCsMu users. One important aspect is that the shared index established in a P2P network uses social capabilities (expressing friendship relations) to avoid attacks (e.g., flood an users) and mistrust (e.g., share a malicious resources to someone).

The PiCsMu application has an innovative manner to store data due to the capability to fragment files into several chunks (file parts) of different file types, and de facto store these file parts into different Cloud Services (independent of which file type is accepted by such Cloud Service). Each file part is encrypted with a different encryption method, with a different salt, Initialization Vector (IV), and key. Moreover, each file part is encoded using an encoder which transforms the file part data into a specific file type – e.g., data can be encoded into an image through the means of Steganography. Several encoders are provided in the scope of PiCsMu application.

In order to illustrate the PiCsMu system, the example of a PiCsMu user uploading a PDF file X for private use is given. The first step is to check which Cloud Services credentials the PiCsMu user has registered for its use. Assuming that Google Picasa, SoundCloud, Dropbox, and Facebook credentials are added to the PiCsMu application, the fragmentation process is able to calculate how many file parts the PDF X can be split – based on the maximum upload size of each Cloud Service. In this example, it is possible to assume that the PDF X was divided into 6 file parts. Each of these file parts should be first encrypted and go through the encoding process. The encoding process will consider each Cloud Service available and check which file type is accepted to upload to the given service. For example, Google Picasa just accepts images or videos (JPG, PNG, AVI, etc), SoundCloud just accept audio files (MP3, WAV, OGG, etc), Facebook accepts either images or video files, and Dropbox accepts any kind of file type. Therefore, the PDF X is fragmented to 6 parts, in which each of these 6 parts are different encoded files (of different file types) carrying the PDF data: 2 JPG images, 1 PNG image, 1 MP3 audio file, 1 WAV audio file, and one plain text file. As the last step, these files are stored into the Cloud Services and the PiCsMu central server that hosts the private index is informed about the process. When the PiCsMu user wants to download the PDF X, the PiCsMu



application first contacts the central server and checks the index for the specific PDF file. Based on the index, the PiCsMu application is aware of where the file parts are located (in which services), how the file parts were encoded, how the file parts were encrypted, etc., and it is able to join the PDF X file as in its original format.

The main advantages of having the fragmentation process are:

Distribute file parts in multiple Cloud Services around the globe, enabling a network of entities that store data being managed in an overlay.

A single entity (Cloud Service) is not able to reconstruct the whole file data, which is a measure to enable a higher degree of security.

A scheduler (PiCsMu Scheduler) can interfere in the fragmentation process in order to split files using an error correction code (e.g., Reed Solomon). Therefore, if one or more Cloud Services loose the data, such part can be reconstructed using the same error correction code.

The main advantages of having the encoding process are:

Higher security and privacy. The data is encoded into files that do not seem to carry any information. Therefore, the encoding process adds another layer of complexity in order to extract the original data from an encoded file. For example, data is encrypted and encoded into image files (e.g., JPG or PNG). If a malicious person wants to extract the data from the JPG file, first it is necessary to identify how the data was encoded. After that, the data should be unencrypted, and then, finally, the malicious person would have the original data. It is important to highlight that the encoding process combined with the fragmentation process gives even a higher degree of security, since parts of data will be encoded into files and spread into different Cloud Services – thus, diminishing the probability that a malicious person would have the possibility to reconstruct the whole file data.

Data can be stored anywhere, even if there are file type restrictions. This enables the notion of homogeneity: all Cloud Services can store the data, in the same manner, independently of file type restrictions.


As forecasted in [68], online storage is constantly gaining more popularity, whereas the services overviewed in Section 4.3 are expected to generate a considerable large amount of the global traffic in the future. Therefore, due to the increasing popularity of and significant traffic volumes generated by such applications, SmartenIT should consider this category of applications by designing traffic management mechanisms that will efficiently address the generated traffic.

Additionally, research work related to the described applications, e.g., [29], stress out the lack of efficiency in certain cases and the potential for improvement for such applications. Specifically, the Dropbox application is of interest to SmartenIT due to its increasing popularity and the very high amount of traffic created by this (so far) limited percentage of users; these two, i.e., popularity and data volumes generated, motivate our expectations that cloud storage systems will be among the top applications producing Internet traffic soon. However, the key question regarding the selection of Dropbox as candidate application to be further addressed by developing mechanisms in the context of SmartenIT is yet to be answered; unfortunately, the proprietary nature of the application



and the used infrastructure is rather dissuasive due to the low level of intervention allowed. Moreover, as online storage applications are becoming increasingly popular also on mobile devices [21], they should be considered by the SmartenIT traffic management solution in respect to mobility and energy efficiency aspects of the project.

4.4 Search Engines

Although a query sent to search engine generates insignificant traffic volumes, the enormous number of queries performed daily incurs great traffic volumes and also significant energy consumption. In particular, according to Google [126], [127], the energy consumed to perform one query in Google’s search engine, i.e., Google Search, is about 0.0003 kWh; thus the energy to conduct 100 searches on Google’s site is equivalent to a 60-watt light bulb burning for 28 minutes. Below, in Table 4-3, the 5 most popular search engines are presented. In the sections to follow, we describe the top 3 and provide traffic and energy characteristics, where available.

Table 4-3: Top 5 search engines [128].

Websites Total visits Visits share Rank 03/16 Rank 03/09 Rank 03/02

Google 3,047,369,237 72.79% 1 1 1

Yahoo! Search 330,764,881 7.90% 3 3 3

Bing 316,318,917 7.56% 2 2 2

Ask 137,104,317 3.27% 4 4 4

AOL 71,810,730 1.72% 5 5 5

4.4.1 Google Search

Google Search [175] is the most frequently used search engine developed by Google Inc. Google uses thousands of low-cost computers from their server farm to provide parallel processing. Google search engine consists of: Googlebot, indexer and query processor.

The Googlebot [176] is a special program that finds and downloads web pages and provides their content to the Google indexer. The Googlebot can find web pages in two ways: when user fill in special add URL form, i.e., www.google.com/addurl.html, or when Googlebot finds links leading to the web page. Those specific web crawling robot can send out thousands requests in the same time. To do so, Googlebot uses many of computer requesting and fetching web pages. Googlebot makes request to the specific web server more slowly than it can. This behaviour does not overload servers. It has special queue for pages it will visit in the near future. In addition Googlebot must constantly examine and compare pages already existing in the Google’s index. On the other hand, duplicates in the queue list must be removed to prevent scanning the same page few times.

Indexed page are stored in huge Google’s index database. From the Googlebot indexer receives full text of the pages found in searching process. This database is sorted alphabetically by search term. Each index stores a list of web pages that consist desired term. Rapid access can be possible thanks this data structure. To improve search performance Google omits stop words (like the, is, on, why, single letters, etc.). Google’s indexer is also not sensitive to the letter case, multiple spaces and some punctuation.



The Google’s query processor consists of: user interface, the engine that matches queries to the relevant web pages and the result formatter. When a user sends a query via Google’s search box, the web server sends this query to the index servers. Next index server is searching docs with relevant web content. Fragments of web pages are generated to describe each search result, while they are returned to the user in fraction of seconds.

Important part of Google’s search engine is PageRank [177] that provides rank of web pages. It scores importance of the web page using over hundred factors. This value determines web page’s popularity, the position and size of search terms, the proximity of the search terms etc. A page with a higher PageRank is considered as more important and it is in higher position of search results than a page with lower PageRank. PageRank is patented algorithm improved over the years and containing many secret criteria.

4.4.2 Yahoo! Search

Yahoo! is a global Internet company founded March 1st

1995, known mainly as a portal, search engine and many other services, such as Yahoo! Search [129], Yahoo! Directory [130], Yahoo! Mail [131], Yahoo! News [132], etc. It was a successful company in 90ties, founded by two students Yang and Filo originated from Stanford University. Yahoo! has shown rapid and spectacular growth in 90ties but around 2000 it experienced serious difficulties related to situation after dot-com bubble. There was an attempt by Microsoft to acquire Yahoo! in 2008. Starting from 2000 Yahoo! Search uses Google search engine; however in 2004 it started to develop its own engine.

Yahoo! Search is composed of a set of scalable, flexible and highly-available data storage and processing services and implementing them in a cloud model [201]. Yahoo!'s cloud can be used to support externally-facing cloud services and some components of the cloud as possible is open source (such as Hadoop). This will allow to build their own cloud services. Yahoo! has been providing the set of requirements that characterize cloud like multitenancy, elasticity, scalability, load and tenant balancing, availability, security, operability, metering, global and simple APIs.

The main services of Yahoo!’s cloud consist of three tiers: core services, messaging, and edge services. The core services layer is divided into three groups of systems: batch processing (manages CPU cycles), operational storage and provisioning (manages the allocation of servers). The edge services are responsible for reducing latency and increasing supplies to end users. The messaging tier assists to bind different services together.

4.4.3 Bing

Bing [178] is a search engine developed by Microsoft and running in the cloud ([199], [200]). It was launched in 2009 and steadily gains popularity. Currently Bing powers Yahoos search, making it second most popular search engine with nearly 30% combined query share (thus Google, Bing and Yahoo serve over 90% of all web searches) [135].

Marketing angle for Bing was to advertise it as a “decision engine”, that provides targeted results, depending on input gathered from user. Currently targeted results are a norm; however, Bing aims at returning extended information or even answer as opposed to simply listing relevant webpages.



Therefore Bing features include instant answers – display of stock charts, extended travel information, shopping advice, restaurant recommendations etc. Using WolframAlpha [179] engine it can provide results of non-trivial mathematical problems. Since early 2012 Bing is integrated with Facebook.

Bing currently uses four crawlers to build searchable index [134]. Bing’s index contains estimated 13 billion webpages [136]. Query processing is being constantly improved; therefore multiple indexes, partial indexes and training trees are used. They are then evaluated by several metrics, to compare them and find ones that perform better. Those metrics include cost to search, click-through rate (how often users follow link suggested by given query), quick-backs (how often user hits “back” button in browser soon after visiting returned webpage). Those alternative indexes along with various code changes and new features are tested on certain group of users (either random or picked depending on their previous activity), typically without their knowledge. Subsequently analysis of engagement metrics evaluates necessity and effectiveness and impact of given search method or functionality [133].

Figure 4-1: Bing ISN structure [198].

Figure 4-1 presents the structure of Bing search engine [198]. Frequently asked queries are stored in cache. Since the large number of indexed web pages is stored and latency-sensitive search queries is issued, the crawled data is split into multiple and independent Index Serving Nodes (ISNs). Each ISN works in parallel; it identifies and returns most relevant pages with their ranks to the top level machine i.e., the aggregator. The aggregator replies to the requesting client by sending him search results containing a title, URL and context snippets. To achieve better performance ISNs can be replicated [199]. This allows to balance incoming search queries among multiple ISNs that are responsible for the same shard. ISNs communicate only with frontend machines – the aggregators, never with other ISN.


While web searches alone do not create significant traffic themselves, they are fundamental functions allowing efficient usage of Internet resources by Internet users. Two aspects seem to prevail – constant evolution and result targeting.



While initial indexing is a straightforward process, query processing is a complex process. Returned results not only have to match submitted query, but are also ranked via as much as hundred factors and metrics, ranging from accuracy and popularity of page to user preferences and search history. Search engines constantly create new mechanisms and functions to improve their performance, and test them on subsets of their users. Therefore search engines must be designed in a way that allows such on the fly trial and error approach. Since result targeting is a norm it might be reasonable to group users according to their preferences, to optimize and strengthen targeting process.

In terms of energy efficiency aspect, it is important to realize that Google data centres alone consume 0.01% of world’s electricity. Moreover, Google takes pride in those, claiming they’re up to 50% more energy efficient than typical ones (with most energy saved on facilities housing servers) [137]. This makes it an interesting topic for SmartenIT energy-awareness considerations. An example of impact of migration to cloud on energy savings is included above in Section 4.4.2. Moreover, end-users' queries could be crawled to gain insight on their interests and communities. Such information can be employed to complement and support social awareness which is also of high interest to SmartenIT.

4.5 Online Social Networks

In this section, we briefly describe main features, functionalities, implementation issues and performance indices (if available) for fundamental concepts of social networking, which are responsible for creating novel habits and opportunities for Internet users changing way of communication and content exchange. Such insight will be useful towards the design of socially-aware traffic management mechanisms.

4.5.1 Facebook

Facebook is currently the most popular online social network started in 2004 by students from Harvard University. Facebook users are able after their registration to create their personal profile and invite or add other people as friends. Whenever a ‘friend’ changes profile’s content, notifications are sent to his friends. Besides personal profile Facebook allows to create common-interest user groups – by organizations, TV stations, local authorities, etc. Currently Facebook has more than 1 billion registered users [138].

The interface of Facebook enables each user to create a public profile with pictures and other personal information such as date of birth, hometown, contact information, school, employer and lists of personal interests. Moreover, a user can establish a friendship link with other users by sending and accepting a friendship request. A profile of user has a “wall” wherein the user himself and his (or her) friends can post messages, links to external sources (e.g., YouTube videos), photos, etc. A message is visible to everyone who has access to the user’s profile. Uploading messages on the “wall” is the mostly used activity in Facebook [187].

Facebook homepage (as of March 2013) can be split into six independent sections: 1) top bar, 2) navigator area on the left side, 3) news feed – in the central part, 4) advertisement and birthday and event notifications part, 5) Ticer – a real-time information about status, friendship, photos, etc., and 6) Facebook Chat.

In accordance with [185], a single Facebook web page is retrieved from more than 15 different IP addresses corresponding to multiple servers. During homepage downloading, Facebook creates a number of parallel TCP connections to these servers. Each



homepage section is downloaded in fixed order (from 1 to 6). Each object is retrieved with 3-7 parallel streams for data transmissions. Downloaded files may be divided into 24 types according its properties (file type, name pattern, location on the web page, source IP address) [185]. Moreover, during FB page downloading two peak periods were observed [185]. The first, approximately in the middle of downloading, occurs when core files, pictures and scripts of extensions are collected, whereas the second peak appears at the end of page loading.

Several studies of the traffic patterns generated by Facebook have been performed over the last few years, due to Facebook rapidly increasing popularity and traffic volumes generated. Facebook’s applications and traffic are built on a mix of many IP applications (messages, downloads of photos, video streaming, etc.) over HTTP. Below, we summarize major results from measurements ([82], [83]) on Facebook traffic profiles within the last years:

Communication via Facebook is most often between users in local communities;

Facebook relies on large replicated data centres in the USA, but most users outside the USA experience long delays and high loss rates for Facebook requests;

Long paths also bring unnecessary traffic load on expensive transcontinental links;

Facebook posts generate non-negligible traffic volume, although they have a small volume, but often are spread over large friendship communities.

A comparison [85] of the session activity of users in OSNs with users navigating via search engines reveals that OSN visitors are more likely to stay within the origin website compared to search engine visitors. Moreover the variety and number of domains visited from search engines is higher than those visited from OSNs; in particular, search engines lead their visitors to a wider variety of websites including more shopping and reference-related websites, whereas OSNs are more likely to send their visitors to less popular domains in comparison with search engines. The latter may also be related to the fact that through OSNs significant amounts of content is exchanged, e.g., photos, and videos, which is mostly user-generated (UGC), and thus, long-tail content [63].

An in-depth investigation of Facebook’s traffic patterns, which are compared to respective patterns obtained by observation of YouTube, as the application that contributes the highest traffic volumes to global IP traffic, as well as the model used for their traffic characterization are provided in Appendix C.

4.5.2 Twitter

Twitter [180] is an online social networking service created on March 21st 2006 and then

launched in July 2006. The service rapidly gained worldwide popularity. Since its launch, Twitter has become one of the ten most visited websites on the Internet.

Twitter has similar features to Facebook: The interface allows users to post short messages (up to 140 characters) know as tweets that can be read by any other Twitter user. Often, a Twitter user will see another user’s tweet and wish to rebroadcast that message. This is traditionally accomplished with a retweet. A retweet is a re-posting of someone else's tweet on your profile. Moreover, a user can follow other users they find interesting in order to get real-time updates of content those users publish.



Twitter’s privacy policy is a two option that either allows every message a user creates to be publicly available, or allows only a user’s followers to see posted messages. In addition unregistered users can only read tweets.

As official statistics show Twitter achieves over 500 million registered users as of 2012 but only one out of four is active user, generating over 340 million tweets daily. About 43% of Twitter users are male and only 57% are female. The most users of Twitter, almost 57%, are in age between 26 to 45 years.

4.5.3 Google+

Google+ [181] started on June 28th

2011 as closed social network (users can join only with invitation). The platform is open for public from September 20

th 2011 (registration without

invitations) [31].

Google+ has similar features to Facebook or Twitter: users can share their information in the stream (like the wall in Facebook) where all users’ activity is shown. Unlike to Facebook users Google+ users friendship is unidirectional (similar to Twitter), e.g., first user can observe the second (as a friend) public posts without reciprocate of friendship.

To protect the visibility of a user's activity, Google+ allows users to manage their contacts’ access to their content by creating circles. This feature allows users to control all information visibility to labelled groups of friends. There are two types of circles: in- and out-circles [32]. The first one is the list of other users who contain a given user in their circles. The second – list of added users by a given user. List of circle user and circle name are only available for circle creator. People can add other users to their circles without confirmation. Note that default privacy settings in Google+ are public (all information visible to anyone). In addition there are also four other options: extend circles – profile open to users that are in circles and their friends, your circles – information visible in user circles, only you and custom – custom list of circles that can see user activity.

Figure 4-2: Google+ User Statistics (Dec. 2012) [33].

Google+ uses encrypted channel (i.e., SSL) throughout the connection. Like in other Google applications, the user can enable a two-way-authentication; this functionality propagates also to other Google applications (e.g., Gmail, Google Calendar, Google Docs, etc. This feature improves account security by requiring a combination of two elements for the confirmation of identity – a user name and password and the special random code (delivered as SMS, via phone call or supplied with special mobile application, e.g., “Google Authenticator” for Android smartphones). Additionally, each Google+ user has individual ID [32], which contains a 21-digit integer, where the first digit is always 1. This



protects users to identify valid user IDs by generating random number (the ID space is very large, i.e., 10

20). Figure 4-2 shows that Google+ achieved by Dec. 2012 500 million

total users with over 135 million active users [33].

4.5.4 LinkedIn

LinkedIn [182] is a social networking site focused on business relationships. It is currently ranked 13th in Alexa Ranking [24] with over 175 million registered users. Professional offers can be posted there and users can look for jobs and career opportunities. Users can build their contact network with direct and indirect contacts. Each can construct their resume showcasing their background, skills and experiences.

LinkedIn is focused on connections and job offers; however some content can be published, ranging from business-related articles, market analysis, through group discussions to user pictures, which aid identification. Other features include individual message system, groups and “LinkedIn Answers” where one can ask questions or seek answers and opinions in chosen category.

There is no charge for the basic functions of LinkedIn. Bulk of the company’s revenue comes from advertising, but in 2008 LinkedIn launched sponsored advertising as a new functionality for users. Nevertheless, there are three tiers of service in LinkedIn. Data tier, maintaining persistent state including user data. The service tier, implementing an API layer, is responsible for capturing the logical operations on the website. The display tier translates API into user interfaces (web pages and widgets on external sites). This separation, along with fact that service and display tier are stateless allows for easy balancing of the load, it also restricts harder problem of scaling the systems with state to smaller number of devices.

LinkedIn employs two data storage systems, one for the rich search capabilities and one for the simple and fast key-value access. It also uses “Databus” system, to capture data changes and provide a common pipeline for transporting those changes from primary databases to various applications. This solution is easy to scale and extend to add support for other data sources.


Social networks produce currently considerable traffic volumes (estimated between 20% - 25%). Their characteristics are quite consistent – mostly made up of small posts that are spread over very large communities of users. Since interacting users are in most cases geographically closely located and communication relies on distant data centres, there is clear space for improvement and management with traffic related aspects being addressed by SmartenIT. Additionally, due to the multitude of websites integrated with many social networks, solutions involved in this process (traffic management) may be of interest to SmartenIT. Some level of predictability of OSN traffic and user behaviour may be useful to build social-aware mechanisms in WP2.

4.6 Online Gaming

We focus, in this section, on cloud gaming which is a sub-category of online gaming. Cloud gaming is a service offered to gamers (consumers) and involves the streaming of games to any personal device using a thin client, while storage and processing are



performed in the service provider’s server clusters. This service has the following advantages compared to traditional online gaming:

Consumers are given the opportunity to stream and play a particular game on any of their personal devices,

Consumers are not required to have powerful and expensive machines, in order to play recently-released high-end games, but a decent Internet connection.

Below, we provide descriptions of OnLive, Gaikai and Zynga. OnLive and Gaikai are two major competitors in the cloud gaming market, offering high-quality game titles from well-known game publishers, whereas Zynga represents a social-network-oriented aspect of cloud gaming that deploys cloud services to serve millions of gamers daily.

4.6.1 Onlive

OnLive is a San Francisco-based company, founded in 2009, offering cloud gaming and desktop services. Its cloud gaming platform is available in the US and has recently entered the European market, in the UK and Belgium. Its partnership with over 50 game publishers, such as Sega, Warner Bros, etc., as well as its 300 game titles, makes OnLive one of the leading platforms in the cloud gaming industry.

Its gaming platform is available through either the OnLive client (OnLive Game System) for certain consumer devices, e.g. PCs, Macintosh computers, iOS and Android smartphones and tablets or the “MicroConsole TV Adapter”, which can be connected to any digital TV or monitor. In addition, consumers are able to watch the service’s demo videos through any web browser. OnLive’s games consist of 2 streams, one for gameplay, and another for social and demo purposes, and are available in 720p format; hence an internet connection of 5Mbit/s is recommended.

OnLive’s network consists of 5 data centres across the US, covering a 1000-mile radius of users, while contracts have been made with major US tier-1 network providers to reserve bandwidth explicitly for the service [34] and resolve the Best-Effort problem of the Internet, aiming to achieve high Quality of Service (QoS) and QoE for the consumers’ gaming experience. OnLive has also designed and developed their own video compression algorithm, claiming to reduce latency to 1ms at servers’ side. A few studies have been carried out in the recent years, measuring the service’s characteristics. More specifically, total latency was measured to 135-240 ms. [35], [36], in a variety of games (from basic to high-end), with high downstream (5 Mb/s) and low upstream (100kb/s) bitrates and frame rates ranging from 25 to 60 fps depending on the provided capacity [37].

4.6.2 Gaikai

Gaikai is a company offering cloud-based gaming services, recently acquired by Sony Computer Entertainment for 380 million USD. It is based in Orange County, California and was founded in 2008, aiming to provide cloud gaming services worldwide. It serves more than 50 million consumers per month, offering more than 200 game titles and has partnered with major industries of the electronics and gaming sector, such as Samsung, LG, Warner Bros, Electronic Arts, etc.

Consumers may access Gaikai’s gaming platform from any of their personal devices, e.g. personal computer, smartphone, tablet and digital TV, by using their web browser (client is embedded to websites or Facebook). Hence, on the consumers’ side, no extra “heavy”



software is required, but a Java or Flash plugin and a reasonable internet connection (Gaikai recommends an internet connection of 5Mbit/s). As also presented during the Google I/O 2012, the service was also ported to Google Native Client, making it instantly available to Chrome browser and Chromebooks.

As a cloud gaming platform, all the game processing, storage and video encoding is performed on Gaikai’s server clusters. Gaikai has partnered with NVIDIA and has adopted the NVIDIA GeForce GRID (PaaS) [38] cloud gaming platform, operating a worldwide streaming network of 24 data centres, and aiming to achieve low latency/lag, high server density, high-performance video encoding and energy efficiency. Digital Foundry [39] has performed a research, comparing Gaikai to OnLive for a variety of games and measured latency of 133-216ms (while OnLive’s latency ranged from 183 to 300 ms.) [40].

4.6.3 Zynga

Zynga is social gaming creator and provider, based in San Francisco, founded in 2007. In the recent years, its games have become very popular due to its close collaboration with major social networks, such as Facebook. It features more than 30 game titles and serves 300 million active users monthly in 166 countries.

Zynga’s games are accessible on most consumer devices, through either a web browser in personal computers (via Zynga’s website or social networking websites, e.g. Facebook, Google+) or native applications in iOS and Android smartphones and tablets. The minimum requirement is an internet connection.

The massive growth of Zynga’s popularity, has forced the company to investigate ways for managing the increased traffic. Initially, Amazon Web Services (Amazon EC2 platform - Section 3.5.1) were employed (12000 EC2 nodes in 2010), but Zynga has recently moved parts of its service to its private cloud, called “zCloud”, which uses load balancing and scaling techniques similar to EC2, and is based on Apache CloudStack. The cloud management platform RightScale is used for management of both cloud infrastructures.


Internet gaming traffic has witnessed a steady growth over the past years, and according to Cisco, it will continue growing by 52% annually, estimated to reach 630 PBs per month in 2016 [20]. While most gaming services move their infrastructure and services to the cloud, and millions of users access such services from their mobile devices or/and personal computers, it could be important for the SmartenIT project to address the expected traffic growth in the gaming sector and provide efficient solutions.

Generally, cloud gaming generates traffic that travels through the public Internet, while it aims to achieve as lower latency possible to ensure the best QoE to the end users. Although cloud game providers place their PoPs in strategically-selected locations globally, and sign SLAs with major network providers, there are still studies that reveal potential pitfalls of the services, such as high latency due to network delay and bandwidth, reduced QoE in consumers’ side, and unreachable markets due to lack of PoPs. Hence, it is clear that its performance relies heavily on end users’ distance from the closest data centre, and the best effort nature of the Internet. SmartenIT could enforce its traffic management mechanisms to optimize the delivery paths between game servers and end users and reduce latency, by assigning optimal alternative routes, instead of best effort ones, and/or allocating more resources to downstream traffic. As a result, cloud gaming



services’ footprint could be widened to currently unreachable locations and markets, and end users’ QoS and gameplay experience could be significantly improved. It is possible that SmartenIT could cooperate with network providers to intervene in and manage cloud games’ generated traffic over the public Internet, but the proprietary nature of all cloud gaming services would impose significant obstacles.

4.7 Cloud Content Delivery Networks

As defined in Section 3.1, CDNs are distributed systems of strategically deployed servers over the Internet, aiming to store replicas of the content to be disseminated close to end-users and redirect users’ requests to the closest copy. Hence, end-users can access the requested content, with less delay and timeouts, while traffic is reduced as content is not delivered from the origin content server.

Nowadays, there are multiple traditional CDN providers, with many points of presence (PoPs) worldwide, offering content distribution and delivery services to content providers. Akamai [43] is the leader in the CDN market, with more than 100,000 servers in more than 2000 locations in 80 countries, partnering with several popular content providers, e.g., Youtube, Facebook, CNN [41], ESPN [42], etc. Akamai’s major competitors are Limelight [44] with 6000 servers at 75 PoPs in Europe, US and Asia, Level3 [45] with 215 PoPs globally and ChinaCache [46]. Such major CDN providers deploy and operate an extensive and expensive network of physical and virtual edge servers worldwide; hence, it’s rather unlikely that new CDN providers would afford joining the CDN market and competing with them.

Cloud-based CDNs provide the same services as the traditional ones, but take advantage of cloud attributes such as virtualization in order to improve their reliability and performance, as well as reduce operational costs. Cloud-based CDNs can be classified in two main categories.

The first one includes CDNs which i) partly use cloud resources to optimize their services, e.g., cloud storage for content caching, and cloud computation for automatic content replication or service management, ii) are deployed by cloud operators and are provided in an integrated manner with other cloud services. Prominent examples of this category have been described in Section 3.5, e.g., Amazon's CloudFront, Windows Azure CDN, and Google's Cloud Storage.

On the other hand, the second category includes CDN-as-a-Service platforms, i.e., CDNs that are fully deployed on the cloud and operate on a virtualized environment of servers that cache, distribute and deliver content in a CDN-like fashion. The only example of this category, at least to our knowledge is MetaCDN.

4.7.1 MetaCDN

MetaCDN, as aforementioned, is a fully cloud-based CDN company, which has been founded in 2011 as a result of research that took in the University of Melbourne. The problem that MetaCDN faces is the global presence of traditional CDNs such as Akamai, who have invested billions in operating an extensive network of physical servers in multiple locations around the world, making very difficult for others to enter this market. On the other hand, existing cloud storage providers while offering less expensive services, they only provide virtual storage space, missing all essential CDN functionalities, such as load balancing and redirection, and automatic content replication.



Figure 4-3: MetaCDN architecture [47].

MetaCDN’s aim is to integrate major cloud storage services, such as Amazon S3 and Nirvanix SDN into one low-cost, fully-functional overlay CDN. As depicted in Figure 4-3, MetaCDN implements connectors for all supported cloud storage services, in order to provide an abstraction and hide their requirements and complexity from its users. It offers a website and exposes Restful web services, with which content creators and providers are able to distribute their content to one of the supported cloud storage providers, while end-users experience high-performance content delivery due to load balancing and redirection mechanisms.

To achieve this, MetaCDN consists of several components, handling specific tasks; QoS monitor measures storage services’ performance, Allocator selects the optimal provider based on multiple criteria, Database and Manager components store information and perform internal tasks, respectively, while Load Redirector handles request redirection to the most appropriate replica. A unified namespace is also adopted to simplify content management and routing, and make the mechanisms transparent to end-users. The service’s features also include website acceleration, cloud-based video encoding and streaming and high-performance content delivery.

Table 4-4: MetaCDN performance and Content Providers’ benefits [47].

MetaCDN consists of around 80 PoPs in 15 countries worldwide. A few studies were performed regarding MetaCDN’s performance [47], [48]; they focused on response time compared to direct replica access time and the overhead was from insignificant to



noticeable, depending on the client’s and closest replica’s location. More specifically, a few tests were conducted in an experimental testbed, to measure response times and throughput from multiple client locations worldwide, using 3 redirection policies, random, geographical and utility-based; average response times varied just over 1 second due to the proximity and distribution of Amazon and Nirvanix nodes globally, with the exception of Beijing clients and a few timeouts when random redirection was applied (Table 4-4).

As MetaCDN’s network expands and further integration of other popular storage services is adopted, performance increases due to higher availability of more local replicas.


MetaCDN integrates multiple cloud storage services into one virtual and efficient CDN, achieving lower, but comparable, and cost-efficient performance to traditional CDNs and cloud storage services. Its internal architecture and mechanisms could be a basis for SmartenIT architecture design which aims to support the federation and inter-connection of multiple data centres and clouds. In addition, SmartenIT’s traffic management mechanisms could be used to improve MetaCDN’s response times towards end users, especially in cases that cloud storage PoPs are not in close geographical proximity, while energy efficiency and social awareness criteria could be enhanced into service’s redirection policies, improving replica distribution and selection and overall service performance.

Though due to its proprietary nature, potential for intervention in the MetaCDN overlay might be quite low. Moreover, the impact of a mechanism addressing MetaCDN's traffic cannot be predicted, as currently no information on traffic characteristics of the MetaCDN was found.

4.8 Mobile P2P File-Sharing

Over the last few years, P2P file sharing traffic share over global IP traffic has declined from 19.2% in 2010, to 12.0% in 2012 [21]. However, regardless of the share decline, P2P file sharing still is responsible for huge amounts of data carried over the network, i.e., according to [23], P2P file sharing traffic (excluding P2P video streaming traffic such as PPStream and PPLive) will increase by 2016 with 17% CAGR.

Moreover, BitTorrent is expected to experience a drop in overall ranking [21]; still though its place will remain high as it is predicted to fall from being the second largest application on the network, to being the fourth, after Netflix, YouTube, and HTTP web browsing.

In recent years, P2P file sharing applications have been developed to operate also on mobile devices, e.g., smartphones and tablets. Nevertheless, mobile P2P file sharing has not been very popular due to the fact that mobile devices participating in content sharing communities face at least two challenges, both of which are related to mobile devices limitations in terms of storage and energy capacity.

First, content shared over P2P networks such as BitTorrent [5] comprises mainly of video files, whose size varies from few hundreds of MBs (e.g., standard definition (SD) video clips, TV shows episodes, etc.) to few tens of GBs (e.g., high definition (HD) 3D movies). Taking into account that the storage capacity of mobile devices currently is up to few tens of GBs, it can be easily inferred that storage constitutes an important constraint to mobile P2P file sharing. An also important constraint and more serious obstacle is the energy consumption of the mobile device. The progress on battery technology is steady but slow



so spending as little energy as possible on different operations is likely to remain an important design constraint for mobile solutions.

Hence, in the sections below, we overview applications for mobile P2P file sharing, briefly describe their operation and address their relevance w.r.t. SmartenIT's objectives. Moreover, an overview of P2P file sharing applications mainly addressing desktop devices such as PCs and laptops, and not mobile devices such as smartphones and tablets, is provided in [183].

4.8.1 BitTorrent

In this section, we briefly present most notable P2P torrent-based file sharing applications for mobile phones.

Generally, P2P clients for mobile devices address certain platforms and OSs. For instance, SymTorrent [86] is an open source BitTorrent client for the 2nd and 3rd generation of the Symbian OS, which supports downloading of multiple files simultaneously and resuming after restarting the application. Transmission [89] is a torrent client directed at Mac OS X, Linux, FreeBSD and Solaris, which is claimed to can also be ised on a mobile phone by the use of a web user interface named "Clutch". MobTorrent [90] is the first Bittorrent client for Java Micro-Edition (ME) based mobile phones, e.g., Nokia S40, Nokia 6500, and supports all Java ME capable phones with the proper JSRs. Moreover, wmTorrent [93] is available for Microsoft's Windows Mobile, respectively.

Particularly for the Android OS, there are plenty of BitTorrent clients such as tTorrent [91], aTorrent [92], BitTorrent for Android [94], and uTorrent for Android [95], with plenty of features, some of which include: download of multiple torrents, queuing search for torrents, use of both Wifi and WiMAX mode, ability to limit upload/download speed, web browser integration, multiple parallel downloading, trackerless torrent (DHT) support, IP filtering support, proxy support, encryption, option to pause downloads when external power supply is not connected, etc.

Apart from BitTorrent clients that download the content file to the mobile device, there are also several clients such as uTorrent Remote [97] and BitTorrent remote [96] that practically provide remote access to a PC or laptop running the actual BitTorrent application so as to monitor downloading, add, remove, stop and resume torrents.

Last but not least, another BitTorrent client, highly interesting w.r.t. to SmartenIT's targets, is CloudTorrent, which is an extension to SymTorrent. CloudTorrent [88] is an alternative mobile P2P file sharing architecture addressing energy efficiency. In particular, the CloudTorrent architecture consists of two main parts: a phone application communicating with the cloud and a cloud server (an Amazon EC2 instance with 10 Mbps upload capacity was used in the evaluations) hosting the remote (regular) BitTorrent client. Evaluation results have shown that CloudTorrent is able to reach much higher transfer speed, i.e., 88% faster than SymTorrent. Additionally, CloudTorrent consumes less energy compared to SymTorrent. According [88], CloudTorrent can be extended to incorporate also users' PCs as servers; practically, that would lead to a hybrid solution which would achieve higher decentralization of traffic, and lower energy consumption in the cloud.


Cisco [23] predicts that global mobile P2P traffic volume will increase from 46 PBs (i.e., 10

6 TBs) in 2011 to 194 PBs by 2016, which implies a CAGR of 33%. Though, taking into

http://clutchbt.com/



account the fact that global mobile traffic is expected to reach up to 10.8 EBs, this implies that mobile P2P traffic will constitute only 1% of total mobile IP traffic. Note that the aforementioned traffic volume does not include P2P video streaming traffic, as all kinds of video streaming traffic is considered to be a consolidated category.

The development and adoption of cloud-based solutions such as CloudTorrent may affect this trend, increasing the popularity of mobile P2P applications, and thus the overall traffic volume generated by such applications.

From SmartenIT's point-of-view, mobile P2P file sharing applications are of interest as they seem to gain popularity and therefore are expected to generate higher traffic volumes on the network in the near future. Additionally, cloud-based mobile P2P applications such as CloudTorrent might be interesting, as the decentralized approach incorporating also users' PCs as servers, e.g., in order to run the actual BitTorrent client, could be extended to provide also other high-volume, or bandwidth-intensive mobile applications, e.g., video streaming or gaming, in an energy efficient manner. Moreover, the cloud-based P2P file sharing applications can constitute a nice example of service mobility as content chunks exchanged over the P2P network can become simultaneously accessible through different devices of the same end-user.

4.9 Mobile Instant Messaging and Voice-over-IP

With the increasing usage of mobile devices and in particular smartphones, a family of special applications has been published that targets the communication needs of users in mobile environment. These applications are not cloud-based; though they might be of interest to SmartenIT as they might constitute a source for obtaining social meta-information on mobile users. Well known examples are WhatsApp [49], Viper [50], HeyTell [51], and Skype [52]. Their motivation is to provide a rich platform for communication, going beyond the traditional services of mobile providers. They are available for all big players in the mobile operating system market, such as Android, iOS, and Windows Phone. In contrast to traditional mobile services, such as telephony and SMS, the apps provide their services Over-The-Top (OTT) using the mobile data connection, without any explicit cooperation with mobile service providers. Thereby, they largely interfere with the traditional service providers’ business models [53].

According to Sandvine [53], the provider’s revenue is already drastically impacted by the users’ switch to OTT messaging services such as WhatsApp. While for SMS services, the network provider’s revenue is estimated to be as high as 30,000 USD per delivered GB text data, for OTT delivery, e.g., using WhatsApp, it is only about 10 USD for the same amount of data. Sandvine therefore expects that SMS delivery as a very important segment for the provider’s revenue becomes less important, while the operation costs remain the same and conclude that the missing revenue has to be replaced, most probably by increasing prices for all other services (i.e., voice and data services). This shows that, from a business model point of view, these special applications and their adoption for mobile devices are highly relevant for mobile providers.

From the technical point of view, it is important to understand the mobile data traffic that is generated by such communication services. In 2011, WhatsApp announced that, globally, they deliver one billion messages per day [53]. In comparison to traditional SMS, it is characteristic for WhatsApp messages that they include a high percentage of media objects generated by the user itself. This includes pictures, videos, and sound recordings that users share with single contacts or a whole group. WhatsApp is most prominently



seen in the Asia-Pacific region, where, during peak hours, it occurs in the top ten services that are contributing to the upstream traffic in mobile networks. According to another report by Sandvine [21], it accounts for up to 2% of the overall uploaded data during these hours. In an example mobile network with 1 million subscribers, about 7% of them were using WhatsApp, sending in average 12 messages each hour during an average day.

Another kind of applications, such as Viper and Skype, targets real-time voice communication. In comparison to WhatsApp, they produce longer lasting flows of data with rather small bitrates. For voice traffic, quality of service aspects such as low delays and jitter are very relevant, whereas for sharing media objects, e.g. via WhatsApp, bursty traffic with no tight timing requirements is the reality.

4.9.1 Viber

Viber [50] is a VoIP and Instant Messaging (IM) service for mobile users. The introduction of the service took place on December 2010. The user needs to install the Viber client in his mobile device and register in to the Viber network. After the registration process the user can invite other users, and/or search which of his contacts are already registered in to the Viber network. After the registration completeness the user has access to a variety of services like IM, group IM, IM delivery report, full-duplex (FDX) voice calls, images exchange, and geo-location information sharing.

The registration process demands from the user a valid Mobile Subscriber Integrated Services Digital Network-Number (MSISDN). The user provides to the network his MSISDN through the client. Then, the user receives an SMS with a verification code. After entering the verification code the user is granted access to the network. Only one client at a time is allowed to be paired with an MSISDN. However, the client can run in a different device than the Subscriber Identification Module (SIM) card is installed.

Currently the Viber client is available for the majority of the Operating Systems (OSs) used by the majority of the mobile devices vendors (i.e., iOS, Android, Windows Phone, Blackberry, Nokia, and Bada). The client is available in ten languages and the services provided to the Viber network users are free. To our knowledge, there is no significant research, neither a technical traffic evaluation concerning the Viber protocol, nor information concerning the call-accounting schema if any.

4.9.2 HeyTell

HeyTell [51] is an instant voice messaging service for mobile users. HeyTell is a Half-DupleX (HDX) communication application. The user need to install the client, which is currently available for iOS, Android and Windows Mobile devices, register to the network and start using services like instant voice messaging, group instant voice messaging, geo-location sharing, and voice transformation effects. The client as well as the service is provided for free. However, some add-on services like group instant messaging capability and voice transformation effects are charged. The user can register to the network either by his MSISDN, or his e-mail address. Then the user needs to invite all the users that he wish to establish a communication with. After both users accept the invitation they can exchange instant voice messages. The client can equally transmit messages over any available Internet connection. Since the communication is HDX the quality of the sound is not affected from by quality of the network (e.g., bandwidth, latency). The only relation to the network quality is the maximum time needed for a message to be transmitted. Similarly to Viber, on our knowledge there is no significant research, or traffic analysis of



the protocols used in HeyTell, neither any service related accounting information, nor any Mobile Network Operator (MNO) that blocks this specific service.

4.9.3 AbaCUS

AbaCUS is an application that integrates VoIP solutions in order to support the Auction-based Charging and User-centric System described at [54], while it can also act as a VoIP client. The subscribers of every MNO might act as a caller, as well as a callee.

Concerning the caller role, the MNO’s subscribers will define their QoS demand per call, and their maximum cost tolerance for the requested service. Furthermore, the application will consult third party providers, such as social networks, in order to retrieve all the necessary information for the AbaCUS system, like the callee’s IP address, his home MNO, his current location, the caller preferences for potential multimedia, or advertisement content while the service is being initiated, etc. Finally, the initiation of the call in the auction-based environment will be handled by the application. However, the respective steps defined by the AbaCUS protocol will not be transparent for the caller, since that could negatively affect the overall call procedure experience.

The MNO’s subscribers, concerning the callee role will have to set, and update when is needed, their home MNO and provide access to the application to switch between different MNOs. Thus, the AbaCUS application will handle the temporary MNO switching for the callee. Last but not least, the AbaCUS application both on the caller and the callee side, acts (1) as a QoS measuring mechanism, and (2) as a reporting system when a QoS agreement is violated. The information collected during the QoS measuring process will be available to the stakeholders.


The presented special applications for mobile devices are important to be considered in the analysis of today’s content delivery and communication applications. As presented, WhatsApp, Viber and HeyTell have a large user basis and contribute a small but, nevertheless, relevant part to the overall upload traffic share in mobile networks. AbaCUS is using charging as a provider selection solution for the mobile voice services, thus relating also to management of this traffic. The importance of this type of traffic management is mainly due to the fact of the significantly higher cost, compared to the mobile data traffic, for the end users. Furthermore, the aforementioned applications can be valuable source for the analysis of social interdependencies between users as they have detailed knowledge on the communication pattern and the social graph of active users. It should be evaluated if and how such information could be accessed to enhance content delivery mechanisms.

Besides their use as source for social information, the management of traffic generated by mobile instant messaging and VoIP applications is out of the scope of SmartenIT due to the fact mainly that these applications are not cloud-based. Other limitations include the fact that they generate extremely low amounts of traffic, whereas in comparison to, e.g., online social networks where often the content distributed is meant to be radically spread among very large populations, i.e., viral effects, for content that is sent via applications like WhatsApp in most cases this is not the case.



4.10 Summary of Overlay Applications Characteristics

In Chapter 4, we have overview 9 categories of overlay applications. The large majority of these categories, e.g., video streaming, P2P video streaming and file sharing, online storage, online social networks, cloud CDNs and online gaming refer to cloud-based applications which generate significant traffic volumes. Moreover, applications belonging to the aforementioned categories are responsible for increased energy consumption either in data centres, e.g., video streaming, online storage and search engines, or in end-users' mobile devices, e.g., mobile video streaming, mobile P2P video streaming and mobile P2P file sharing.

On the other hand, one category, namely mobile instant messaging and VoIP, may not be generating high traffic volumes or energy consumption, though they might be of some interest to our investigations, as they may reveal social relationships between users; thus, information extracted by them can potential serve as input to socially aware traffic management mechanisms to be designed, implemented and evaluated by SmartenIT.

Moreover, apart from well-established applications, we have also addressed some emerging ones. The reason for addressing also some less known applications is the fact that they can constitute nice examples for evaluating traffic management mechanisms where intervention potential is higher than other well established proprietary applications where intervention may not be applicable.

Summarizing the investigation conducted in this chapter, we have provided description of the aforementioned applications from multiple viewpoints, in terms of their operation, traffic characteristics, energy consumption, and end-user's device. The identified characteristics will play fundamental role and will provide input to the classification of cloud services, which will be performed by Task T1.1 and will be described in D1.2, where also final decisions on the selection of applications to be addressed by SmartenIT are to be made.

Finally, we highlighted relevant to SmartenIT characteristics of new overlay applications, i.e., related to the main concepts and scenarios of applications, as well as we have discussed potential for intervention and optimization by traffic mechanisms to be designed by SmartenIT in WP2.

Table 4-5 summarizes the characteristics of the overlay applications categories described in this chapter taking into account the traffic characteristics and energy consumption considerations for cloud services discussed in Section 3.6 and Section 3.7, respectively.



Table 4-5: Summary of cloud-based applications' characteristics.

Type of Application

Video streaming Mobile P2P streaming

Online Storage

Search Engines

Online Social Networks

Online Gaming

Cloud CDNs Mobile P2P File Sharing

Traffic generated

Very high High; expected

to increase

Medium; expected to

increase

Low to medium (due to the

magnitude of queries)

Medium to high High Medium to high Low to medium;

expected to increase

Energy consumption

Very high; both in data centres and mobile devices

High; mainly for mobile devices

High; mainly in data centres

High; mainly in data centres

Medium to high; mainly in data

centres

No data; expected to be high mainly in data centres

Expected to be high mainly in

data centres as for traditional

CDNs

High; in mobile devices

QoE concerns

Long startup times and

stalling; longer startup could

reduce stalling (critical to QoE)

Long startup times and

stalling; longer startup could

reduce stalling

High latency Accuracy of results, high

response times Increased latency

Increased latency; critical

to QoE

Higher response times than traditional

CDNs

High download times

Potential for intervention

Low to medium; e.g., alternative streaming client; incentives need to be addressed

Low to medium; e.g., alternative

P2P client; incentives need

to be addressed

Medium; incentives need to be

addressed

Low; proprietary solutions

integrated with other services by the same

operator

Medium to high; depending on

available social information;

incentives need to be addressed

Medium (or low to medium);

incentives need to be

addressed

Medium (or low to medium);

incentives need to be

addressed

High; multitude of solutions for

P2P file sharing can be

extended

Potential for optimization

High; unburden data servers in terms of energy consumption,

traffic localization,

improved QoE

High; reduce energy

consumption

High; improve QoE for end-users, reduce

energy consumption

Medium to low; reduce energy consumption in

data centres

Medium to high; traffic

localization, improve QoE, reduce energy consumption

High; traffic localization,

improved QoE, reduction of

energy consumption


improved QoE; reduce energy consumption


improved QoE



5 Stakeholders Characterization

In this chapter, we identify major stakeholders involved in cloud services provisioning scenarios, as well as to investigate their (possibly conflicting) interests, and analyse their interactions and potential for collaboration among them, their technical dependencies and

business relationships. By the term stakeholder designates an entity which supervises or makes decisions that affect how the Internet ecosystem operates and evolves. A same entity often plays multiple roles, for example ISPs offer connectivity services but at the same time can provide entertainment content services.

As mentioned above, stakeholders characterization depends on specific scenarios. In this chapter, we indeed describe four scenarios of interest: (i) inter data centre / inter cloud communications, (ii) collaboration between data centres for energy efficiency, (iii) global service mobility, and (iv) exploitation of social awareness for content delivery. Three of these scenarios, in particular, (i), (iii) and (iv), were also discussed in SmartenIT's DoW [204], whereas the fourth one, i.e., (ii) was introduced in the SmartenIT kick-off meeting. Note that we chose to present the latest identified scenario right after the inter data centre / inter cloud communication, as they are more closely related.

All four scenarios reflect the main objectives and challenges faced by the project, i.e., economic considerations and incentives for collaboration, energy efficiency, social and QoE awareness. Nevertheless, the description of scenarios in this chapter is not final and will be further evolved by Task T1.4, while the final scenarios description is to be provided in D1.2. The stakeholders characterization, as well as the technical dependencies and business relationships between them, will be taken into account both by WP1, especially in the decision making for the selection of applications to be addressed, and WP2, in the definition of relevant use-cases by Task T2.2, and mainly the specification of traffic management mechanisms by Task T2.3. Specifically, the traffic management mechanisms will be designed so as to address the involved stakeholders' incentives and lead the system to a stable outcome [105]. Moreover, the business relationships and technical dependencies will also assist designers, i.e., WP3, to define the interfaces in the inter-cloud, or ISP-to-cloud, etc. communications.

5.1 Basic Definitions

In this section, we define significant terms for the stakeholder characterization to be performed in the next sections.

First of all, in order to perform stakeholder characterization, we need to define specific scenarios and then identify involved stakeholders and analyse their relationships within it.

A scenario is an overall outline of real world actions and events occurring during an interaction with the underlying SmartenIT system. It describes the general behaviour of all system mechanisms and enables the identification of challenges and problems to be solved in order to achieve desired functionality. Stakeholders are entities, individuals or organizations, supervising or making decisions that affect how the Internet ecosystem operates and evolves. The stakeholder analysis comprises of the identification and assessment of the major stakeholders related to an activity based on their influence and

importance, while the stakeholder relationship analysis identifies the types of inter-dependence between stakeholders and how these dependencies can be affected towards some specific outcome.



Stakeholders are considered to be rational, self-interested players; following these interests they perform certain actions towards the satisfaction of their interests. In complex, multi-stakeholder environments, it can be expected that different stakeholders involved in a specific activity may result having conflicting interests. To describe the competitive behaviour due to conflicting interests of stakeholders in the Internet, the term

tussle was introduced by Clark et al. [105]. A tussle is a process in which stakeholders inter-actions usually lead to contention. Reasons for tussles to arise in our scenarios of interest are manifold, e.g., overlay / cloud traffic management and routing decisions between autonomous systems constitute a typical example for tussle space. Thus, with the ongoing success of the Internet and with the assumption of a future Internet being a competitive marketplace with a growing number of stakeholders such as users, service providers, network providers, and cloud providers, tussle analysis becomes an important approach to assess the impact of stakeholder behaviour.

To address an up-rising tussle among Internet stakeholders, incentives, as an economic mechanism to motivate an entity to perform certain actions or to pursue certain goals, can be the tool to motivate stakeholders towards an incentive-compatible behaviour,. In SmartenIT, we aim to design mechanisms that address incentives of the various involved stakeholders, so as to avoid potential tussles among them. We focus on two types of

incentives: monetary incentives, e.g., revenues for providing a specific service, and

performance incentives, e.g., enjoying high(er) QoE or experiencing low(er) congestion.

In the next sections, we describe tools and methodologies proposed in literature to perform stakeholder relationship analysis, and finally, we combine these tools to analyse the interests and types of relationships among identified stakeholders.

5.2 Tools and Methodologies for Stakeholders Characterization

In order to identify significant stakeholders involved in scenarios of interest, we use tools such as Value Network Configurations (VNCs), and employ methodologies proposed and used before in literature such as the Business Model Reference Framework (BMRF) presented in [108].

Moreover, in this section, we briefly introduce the Tussle Analysis Framework (TAF) [106] proposed by SESERV [107], despite the fact that to this end it only partly employed in our investigations. In particular, the stakeholders identification which is the first step of our investigation is based on TAF's stakeholders identification map.

Besides that, another reason for introducing TAF is that it is a useful tool for analysing stakeholders' relationships and interactions, and qualitatively assessing the adoption and evolution of specific network mechanisms or architectures in a long-term time scale. Thus, TAF is aimed to be employed by Task T2.4 to specific scenarios and use-cases, after initial versions of traffic management mechanisms are considered by Task T2.3. Then, the outcome of the theoretical investigation by means of TAF will provide feedback to Task T2.3 on the specification of the mechanisms.

5.2.1 Value Network Configuration

A Value Network (VN) [113] is a business analysis perspective that describes social and technical resources within and between businesses. The nodes in a value network represent stakeholders or actors. The nodes are connected by edges which represent interactions, and tangible / intangible deliverables. These deliverables take the form of



knowledge or other intangibles and/or financial value. Value networks exhibit interdependence.

The VNC method has been proposed in [112] and constitutes a particularly useful tool for stakeholder relationship analysis as it separates the stakeholders (or actors) from the functional roles the actors can take, thus allowing to analyse multiple role combinations instead of limiting to a single value network. Additionally, the VNC method emphasizes technologies’ role in defining the possible value networks and identifies the technical components and technical interfaces between them. Thus, the method improves our understanding of the relationship between the technical architecture, i.e., a set of technical components linked to each other with technical interfaces, such as protocols, and the value network configuration, i.e., role division and related business interfaces among actors.

In Figure 5-1, the basic notation of the VNC method is illustrated. In particular, the larger rectangle with rounded acnes represents a stakeholder (or actor), while the smaller one inside it a role played by that stakeholder; note that multiple roles can be played by the same stakeholder, e.g., a cloud operator may provide both IaaS and PaaS services. The bold arrow is single-directional and represents a product or service delivered from a stakeholder to another one, where the one receiving the service / product is located on the side that the arrow is directed to. On the other hand, the medium and light arrows are bi-directional and represent a business and a technical interface between two stakeholders, respectively.

Stakeholder/Actor

Technical interface

Business interfaceProduct/service

deliveryRole

Figure 5-1: VNC notation.

The VNC method can be nicely combined with two methodologies for business relationship analysis, i.e., BMRF, which partially incorporates VNs, as well as TAF [113].

5.2.2 Business Model Reference Framework

BMRF is an adaptation/customization of the framework introduced by Ghezzi in [109] and refined in [108]. It is based on the conceptual framework originally developed by [114], who claims that the business model components broadly refers to the underlying concepts of 'control', i.e., the inter-firm or VN relationships that the firm is involved in and controls over, 'value', i.e., the way a firm creates actual benefits to its customers and to itself through its value proposition and financial configuration.

The proposed reference framework attempts to shed light on the core Business Model (BM) design parameters for stakeholders in the SmartenIT scenarios of interest. Taking also into account the related literature, according with a value proposition / revenue Business Model, a set of three core building blocks / design themes was identified: Value Proposition (VP), Value Network (VN) (interconnection) and Financial Configuration (FC).

This means that the BM starts with the VP, describing the customer‘s benefits and explaining how they are achieved by the architecture of the value chain and how they are



monetized by the revenue model, i.e., FC. BM provides the required understanding of the way the business entities transform their input to monetizable output. This is necessarily linked to the value chain structure, which provides the means of performing an external business ecosystem analysis (rather than focusing to just one actor) from both a business and technology perspective. The VN model depicts the network of relationships and inter-dependencies among firms that can be structured on several levels or layers. To this end, a set of key layers and variables are identified and then employed to perform the value network analysis of scenarios of interest, thus identifying open issues, stumbling blocks and both business and technical opportunities. The definition of the key variables and roles’ characteristics is complemented with a description of stakeholders’ (seller-buyer) relationships, the direction of services’ provisioning and money apportionment along the value chain. The latter must inevitably take into account (even implicitly) the actors’ incurred costs, so it is important to be able to identify them per actor in scenarios where multiple actors compose services.

To this end, the proposed framework’s three design themes have initially been divided into 13 parameters. The value range of each of the parameters has been adapted (where needed) with respect to the initial proposal of [108] so as to capture the SmartenIT needs and focus, while one of the 13 parameters (i.e., Key Players) has been omitted (but mentioned below for completeness reasons) since it results in information repetition as this is included in the SmartenIT scenario template. Therefore, the SmartenIT business model framework consists of three design themes, each pertaining to four key parameters. The value range illustrates the main feasible strategic decisions of each of the value chain actors under a specific scenario. The reference framework is depicted in Table 5-1.

Table 5-1: Business Model Reference Framework.

Business Model

Parameter Value Range Value Network

Value Proposition Parameters

Product/Service Delivered

Basic connectivity SmartenIT Service Content

Basic Connectivity and Content Provisioning are already provisioned in the current VN. However, the presence of the coordination function performed by CDN Providers together with the establishment of a SmartenIT Intermediation with roles of matching between interconnection demand and offer, brokering and advertising of interconnection agreements, QoS and QoE Management, SLA definition, monitoring and enforcing.

Target Customer Content Provider End user Network Provider Data Centre OTT Provider

The VN provides a clear picture of how different types of actors are interconnected with each other, and particularly the interconnection resulting from a “producer-consumer” relation. Moreover, through the interconnection designed, the possibility for Content Provider and End User to be target by different stakeholders is disclosed.

Customer Value Basic connectivity Packaged Service Content

With regard to the service/product delivered, as described above, the VN is conceived to disclose how different players could actually exploit their customer value.

Resources & Competencies

Technology-oriented Content-oriented

Both resource and competence are enclosed in the VN picture, according with the activities and functionalities carried out by each stakeholder.

Value Network Parameters



Vertical Integration Infrastructure Layer coverage Internet Service Layer coverage Complete Integration

According to the choice made by each actor within the value chain, whether focus on one or more service layer provisioning, the specific stakeholder will cover different activities encompassed in the VN structure. Moving from the bottom side up, the activities belonging to the infrastructure layer shift to the service layer’s one.

Customer Ownership and

relationship

Intermediated Direct

Though the “direct option”, is the preferred one, and is highly considered in the VN picture, the introduction of an Intermediation (SmartenIT / Marketplace) with roles of matching between demand and supply could reduce the dependence on actors directly managing the offer and revenue flow from/to the customer.

Interconnection Modality

Transit prevalence Peering prevalence Marketplace Abstract/Cloud

The network interconnection modality configuration. The VN proposed, comprises this opportunity for new network products with also the already existing transit and peering interconnections, which are relevant in the current marketplace at wholesale level.

Key partners

CP, OTT, Data Centre Marketplace / Intermediary Vendors

The VN embraces all these possible alternatives, indeed the activities performed by Content Application Provider, Content Distribution Network and Network Service Provider, are connected one another in order to represent all these opportunities.

Content-Data delivery model

Client-server Cloud P2P Content Delivery Network (CDN)

To the traditional Client-Server model, the VN with the integration of the distributed configuration adding the technical solutions regarding the Cloud Computing and the Content Delivery Network present all the possible delivery model and all its combinations increasing the interconnection the service management system’s optimization, scalability and flexibility.

Financial Configuration

Revenue Model Usage-based fee Subscription Free Bundled

According to this parameter, the VN does not imply the adoption of a specific model, i.e. the revenue model will vary according to the strategic decision of the specific stakeholder rather than the position on the VN position.

Revenue Sharing Model

Present Absent

As mentioned above, the introduction of the Intermediation Marketplace Community with roles of matching between interconnection demand and offer, brokering and advertising of interconnection agreements, QoS and QoE Management, Service Level Agreement definition, monitoring and enforcing, open up the opportunity of implement new forms of revenue model, such as the revenue sharing model.

Traffic Charging scheme

Receiving Party Pays Sender Party Pays Congestion Charging Absent

VN network structure does not provide an indication on how stakeholders should choose between different Traffic Charging Scheme, however the model proposed seems to be more suitable to the Sender Party Pays and Congestion Charging schemes.

Cost Model Concentrated Investment Joint Investment

As for Revenue Model, the introduction of the Intermediation Marketplace with the roles of matching between demand and supply, brokering and advertising of SmartenIT agreements, QoS and QoE Management, Service Level Agreement definition, monitoring and enforcing, could incentivise the choice of joint investments.

A main feature of this framework is that entails an attempt to cross Business Model Parameters with Value Network Characteristics, thus providing a complete picture of the actors and the nature of their relation under a service scenario. Key actors can utilize this framework while mapping their service-specific VN and BMs of the key stakeholders involved. The association of the different combinations of BM parameters, with each relative variable value, and the currently models adopted, should determine the refinement process through which, the different scenario will undergo, shaping the business models



of the key stakeholders, mapping the activities and the functionalities carried out within the provision of a specific service, taking into account the relationships and interconnections described in the VN framework.

5.2.3 Tussle Analysis Framework

In [115], a framework is described that extends the methodology proposed in [106] for identifying and assessing socio-economic tussles in large, highly dynamic, multi-player environments such as scenarios of cloud services provision. The approach proposed in [115] is a top-down one starting from generic functionalities and studying related tussles, while the approach in [106] was more of a bottom-up: starting from tussles and then consolidating them.

Figure 5-2: Tussle analysis methodology - The SESERV project [107].

The tussle analysis methodology as described in [115] is composed of three steps (see Figure 5-2) and can be executed recursively; the three steps are as follows:

Identify main Internet and cloud computing functionalities and potential bottlenecks, i.e., critical control points and/or scarce resources (step 0),

Identify major stakeholders for each networking functionality and cloud computing functionality and their interests (step 1),

Identify tussles (step 2) and provide scenarios of their evolution by examining whether technologies designed by those research projects meet major stakeholders’ interests (step 3).

Concerning SmartenIT's investigation in this chapter, we focus here on the first part of step 1, namely the stakeholder identification. In [115], an indicative taxonomy of Internet stakeholder roles has been formed, by studying a number of European research projects and identifying missing roles from previous attempts and taking into account interactions with the FISE community. Figure 5-3 depicts the full stakeholders’ map where seven major categories can be seen; these seven categories are broken down to multiple types of stakeholders. Next, the second part of step 1, namely the stakeholders’ interest identification is a procedure that depends highly on the specific scenario which the tussles analysis methodology is applied to.



Figure 5-3: Stakeholders map - The SESERV project [115].

5.3 Inter Data Centre / Inter-Cloud communication

In this section we focus on the inter data centre/inter-cloud communication scenario. In the analysis to follow, we deliberately focus on the inter data centre communication scenario, as the most concrete one between the two. This choice is motivated by the need to focus on a concrete scenario, the existing market needs (also identified in Section 3.6 of this deliverable) and the related research works.

5.3.1 Scenario

A concrete instance of the latter is the NetStitcher system [116] that employs a network of commodity storage nodes that can be used by data centre operators to stitch together their unutilized bandwidth, whenever and wherever it is available. Such storage nodes can be co-located with the data centres or available at other parts of the network. It is clear that this research work is one concrete instance of the SmartenIT inter data centre communication scenario (and potentially service). For this service, some store-and-forward algorithm is envisioned for the chunks of bulk data that need to be transmitted from source to sink data centre(s), as depicted in Figure 5-4. Scheduling should take into account the availability of leftover bandwidth at different parts of the network as well as storage and bottleneck links constraints. This way, an optimal store-and-forward schedule is constructed and the system maximized the utilization of leftover resources.

Therefore, this scenario depicts the need to efficiently and in-time carry traffic among data centres that are located in different parts of the network so as to comply with certain market-driven requirements and deadlines. Below we provide two concrete use cases



envisioned under the inter data centre communication scenario. Note that energy efficiency is not included here since it is handled in Section 5.4.

Fault tolerance services for data centres: Data of a data centre are periodically (e.g. every 24 hours) replicated to a backup facility located at a remote location of the network, thus increasing redundancy and security.

Customer experience enhancement: Replication of data across the network is crucial for large businesses (e.g. CDNs, social networks such as Facebook) in order to push the content or service closer to the end user (corporate or residential) who consumes them. To this end, a SmartenIT inter data centre communication service could serve as a backend service for supporting the data replication.

Figure 5-4: A source data centre uses available bandwidth to perform bulk backup to sink data centre [116].

In order to better illustrate the ideas and apply the relevant tools, we now provide a concrete example for the Fault Tolerance services use case, taken from [116], which is also in the scope of SmartenIT: Assume that a data centre in New York wants to back up its data to Palo Alto. Due to the different load and thus storage availability in the two data centres, direct transfer of the data between them may not be feasible in a common time window. However, taking advantage of the availability of intermediate storage nodes and the available capacity of the backbone links over different time zones, this can be feasible if the intermediate data centres/storage nodes of Boston, Chicago, Phoenix, Denver and Seattle are used. Note that these data centres may be in general served by multiple ISPs; their interconnection is possible due to the Transit ISPs that sell global connectivity network service. This transit service is seamless to both the data centres and the end customers who are purchasing Internet access from their Edge ISPs.

Note that the above scenario is an indicative case of inter data centre communication. Simpler scenarios such as the communication of two data centres for the exchange of data needed to perform computing operations over massive datasets (e.g. using a medical data database in the cloud) are also possible. However, this example comprises a simple yet illustrative use case of this type of scenarios that is representative enough to perform



stakeholder analysis and gain insight on the market needs and solutions SmartenIT is suitable for.

The big picture capturing both the aforementioned use cases (depending on the placement of data centres/caches is depicted in Figure 5-5. Note that it is possible that Source and Destination ISP are direct neighbours (there is a direct link between the two AS Border Routers) or interconnected via one or more Transit Connectivity Providers. This difference will only impact the number of network connectivity providers that must intervene in the service and will be present in the respective value network configurations and the respective business models and money flows.

Figure 5-5: The scenario use cases and network placement of the data centres.

5.3.2 Stakeholders, Roles, Interests

We employ the stakeholders taxonomy described in Section 0 to identify all involved stakeholders and to describe their role. Table 5-2 includes the identified stakeholders, their role description and a preliminary identification of their interests.

Table 5-2: Stakeholder identification for inter data centre / inter-cloud communication.

Stakeholder Role(s) Interests

Connectivity providers

Edge ISP Provides connectivity to end-users Reduce / keep low inter-connection costs; avoid network congestion

Transit ISP

Provides inter-connectivity between the various service providers or edge ISPs; provides inter-connectivity between the PoPs of one service provider

Sell transit to Edge ISPs or large business customers (CDNs, Internet giants), peer with other Transit ISPs; avoid or limit links congestion, apply hot potato routing when possible

Information / service providers

Data Centre Provides the data centre services, i.e., storage of large amounts of data and possibly replication to multiple locations.

Provide high QoE to end-users; reduce inter-connection costs; reduce storage overhead and costs; reduce energy consumption across his own infrastructure

Application Provider

Provides the high-level service of inter data centre communication management (e.g. the NetStitcher) service, i.e. the schedule and

Provide high QoE to end-users for a competitive price; reduce network costs; utilize network resources



traffic management, so that customer objectives are met; this service can be provisioned under the SaaS model; may own and control data centres or storage nodes; decides on the data replication schedule, thus operating on top of the network and data centre infrastructure. (This stakeholder may be integrated with the data centre role.)

efficiently and achieve multiplexing gains for his customers

Infrastructure providers

Cloud operator

Provides IaaS (servers, links) to Application providers and Data centres (as well as end users) so that E2E data transfer from source to sink data centre is made feasible; owns and controls several (intermediate) storage servers across the globe; owns and controls links (even multiple ASes) among his servers, or uses Transit ISP for interconnection.

Provide high QoE to service providers; reduce / keep low network costs; improve performance and scheduling of customers tasks; improve / limit energy consumption by his servers

Users

End-user

The customer of the data centre, typically a business user (e.g. a multinational company forwarding its data all over the globe to data centres close to its PoPs)

Security, redundancy and fast access to data; Enjoy high QoE in terms of low latency, and high availability, fault tolerance, low network costs


We now provide the value network configuration for this scenario. Depending on the level of (vertical or horizontal) integration in the market, there can be a variety of possible value network configurations. This is due to the fact that the roles envisioned in the market and their separation or integration by stakeholders may vary.

In order to highlight this, we begin the presentation with the simplest possible configuration where the (SmartenIT) inter data centre communication service is a new service offered by and integrated in the data centre stakeholder’s business. This could be a valid scenario since such an extension of the data centre functionality is likely due to the foreseen benefits in its business and thus the potential for enhancing its competitiveness in the business, as well as attracting additional revenue to the richer services provided. The resulting VNC is provided as Figure 5-6. Note that under this VNC there is no separation between the Application Provider and the Data Centre stakeholders identified in Table 5-2.

Similarly, it is typical in the European market for certain large providers to undertake both the Edge and the Transit roles. This pertains to providers who both serve their national markets but also have strong presence in other countries (including America and Asia). This allows these providers to sell transit (or partial transit) connectivity services. Thus, such provider integrate both those roles, and though they typically maintain different AS numbers for access and transit, those services are provided by (different divisions of) the same carrier.

Thus, the VNC of Figure 5-6 is the simplest one in terms of number of stakeholders due to the fact that certain stakeholders integrate in their business new roles, both a) in the network layer, where the Connectivity Provider is both Transit and Edge ISP and b) in the application layer, where both the SmartenIT inter data communication service and the data centre services are undertaken by the Data Centre stakeholder.



Figure 5-6: Basic VNC model.

A more generic configuration, as opposed to that of Figure 5-6, where there is a clear separation between the stakeholders that provide Data centre and inter data centre communication services is depicted in Figure 5-7. Under this VNC, there is a clear separate Application Provider who is offering the inter data centre communication service. Both the network layer stakeholders and the data centres focus solely on their domain of expertise are separate entities that conduct business with this new Application Provider. Hence, under this configuration the SmartenIT inter data centre communication service justifies the introduction of an OTT service provider, namely the SmartenIT Application Provider, who under the SaaS model provisions this new service in a similar fashion that CDNs like Akamai nowadays push content close to the networks end users.

Next, we provide as Figure 5-8 the most generic VNC where each role in both the network and the application layer is undertaken by a separate stakeholder. Thus, there is no integration, vertical or horizontal, in the market.

Figure 5-7: Modified generic VNC model.



Note that there are several other alternatives, such as having integration of the Data centre and SmartenIT Application Provider roles without having integration of the Edge and Transit roles by the same Connectivity Provider stakeholder, or even have full vertical integration in the market by an actor who undertakes the Data centre, SmartenIT and ISP roles. Though these scenarios are valid, we omit to provide the respective VNC by means of graphics since these variations add little or no additional insight regarding the SmartenIT scope and stakeholders collaboration compared to the three aforementioned VNCs. However, for completeness reasons these configurations are also briefly discussed in the Business Modeling subsection of this analysis, which follows.

Figure 5-8: Generic VNC model.

5.3.4 Business Modeling

We now apply the business modeling reference framework to this scenario. In particular, we select the appropriate values for the key business model parameters described in the template so as to provide insight to the way business is conducted.

We begin our work with the first design theme, the Value Proposition, and we apply its basic parameters to this market.

Product/Service Delivered: The SmartenIT inter data centre communication service is the end-to-end product delivered. This comprises also custom managed network services products (e.g. standard or premium transit agreements) that support the transfer of the inter data centre communication traffic from the source data centre AS to the destinations. The data centre that originates the data transfer compensates the connectivity provider for the traffic it injects by purchasing this managed network service. Similar interconnection products need to be purchased from the involved networks in cases of inter-domain data centre communication. At the retail level, we can also envision micropayments coming from the customer layer e.g. via (business) customer subscription to the data centres for accounts that allow them to host and replicate a certain amount of data.



Target Customer: The end-to-end service target customer is the data centre who is providing data replication/backup/mirroring/caching services to end customers (via some portal where users subscribe). This could mostly include business customers, including content providers.

Customer Value: The Packaged Service model prevails: Guarantees on the efficient traffic delivery for the data centres, predictability and management of heavy traffic for the connectivity (network) providers.

Resources and Competencies: The stakeholders utilize their resources and competencies in their core business, i.e. is technology-oriented.

Next, the four parameters of the Value Network design theme are provided.

Vertical Integration: It is likely that large data centres that already own or are affiliated with some connectivity provider will perform vertical integration by aggregating the roles of Application Provider, Data centre and Connectivity Provider so as to offer advanced data replication services to its customers.

Customer Ownership and Relationship: There is mediation of the Application Provider with the data centres and networks involved for the efficient provisioning of the service. Thus, there is an intermediated relationship and indirect revenue flows across the Infrastructure and Internet Service Layer, with the exception of the scenario where vertical integration is present in the market. In the latter case, there is direct customer ownership, since the integrated Data Centre / Application provider has ownership and direct interaction with the end customer.

Interconnection Modality-Business Agreements: For the agreements among the Data Centres (and the Application Provider, if present) and the connectivity providers involved in the service provisioning, wholesale transit or custom transit-like agreements are envisioned. Thus, we have transit prevalence.

Content-Data Delivery Model: The CDN model is the model closest to this scenario. In particular, the content delivery of this scenario can be seen as the evolution of existing Akamai-like solution, with some more advanced features regarding the customer requirements for the service.

Last, the parameters of the Financial Configuration theme are investigated.

Revenue model and revenue sharing: The services are purchased from data centres and subsequently by business users who subscribe to those services e.g. via a Dropbox-like portal. Thus, there are two distinct charging layers with separate/co-existing money flows: the aggregate level traffic charging at the network layer (inter-ISP charging) and the per-session based charging at the application layer (end user payments). In the wholesale layer, revenue sharing schemes could be present in the cases of vertical integration and or business alliances among data centres and connectivity providers in the market.

Traffic Charging Scheme: For the management of the aggregate traffic flows, it is expected that the purchaser of the involved traffic contracts will have to compensate the involved networks (if any) for carrying this traffic on top of their network, according to the Sending Party Network Pays principle. This means that the connectivity provider is compensated by the source data centre and then might need to compensate additional connectivity networks that may be involved for the delivery of the traffic at the sink(s), proportionally to the quality attained and the volume of traffic handled.



Cost Model: Concentrated Investment since the stakeholders must invest to compete for and manage the data centre-generated traffic, whereas no Joint Investment scenario is likely.

5.3.5 Potential for Collaboration and Relevance for SmartenIT

Inter data centre / inter-cloud communication is clearly an interesting scenario for SmartenIT. Its scope is in the core of the project’s interests. Also, a traffic management solution for inter data centre communications is considered as an attractive solution for the market, serving an actual market need. Moreover, the inter data centre communication scenario is less vague than the inter-cloud communication scenario and allows us to assume that the positioning of the data centres is known. This fact can also help the project to promote its added value more effectively by working on this more concrete scenario. This way, SmartenIT can make a realistic solution/value proposal to the market, increasing its impact and visibility.

Furthermore, this scenario applies to an advanced service that has to meet different stakeholders’ requirements at various levels. The network and the application layer requirements and interests are coupled and should both be considered for providing a realistic solution. To this end, it is clearly depicted that collaboration is needed among the stakeholders involved in the end-to-end service. This way, the different stakeholders can thus express their interests and constraints and find some incentive-compatible TM solution to be applied.

Note also that this inevitable collaboration can be a major driving force towards integration of roles and the respective product offerings and value propositions in the market. The inter data centre communication service can be seen as an attractive niche market solution that is also a “generalization” of the services already provided in this market. Connectivity Providers, Data Centre owners, or even Application Providers, either existing or emerging, could try to perform bundling of this new service with existing ones so as to dominate the market and strengthen their position in the markets where they are active. Clearly the additional offering of such a sophisticated service would be of high value to the Data Centre's customers; thus the Data Centre owner is the stakeholder who is identified as the most likely one to integrate this role in his business. This in turn could create mixed incentives for collaboration and competition with the other data centres, and could also have adverse implications on the agreements with the network layer providers, i.e., ISPs. Thus, there is an interesting tussle and different incentives regarding who should undertake the provisioning of this service and under what terms and with what scope. Analysis of this issue is beyond the scope of this deliverable and is left for future work.

This scenario analysis also highlights the potential nature of the SmartenIT solution as a whole and the placement of the SmartenIT functionality required to support this solution in the inter data communication service scenario and also beyond. In particular, this SmartenIT solution could be seen as a set of architecture/functional elements placed within the aforementioned stakeholders so as to coordinate, exchange information and enforce the schedule for the inter data centres communication. As both the aforementioned analysis and the NetStitcher example [116] indicate, the control, market offering and orchestration of the end-to-end inter data centres communication service could be performed either by the Data Centre owners, which would integrate this SmartenIT functionality in its own product offerings, or alternatively by a new Over-The-Top provider, i.e., an Aggregator. The latter can be seen as an extension of the Akamai model that is also differentiated with respect to that of Akamai in order to cater for the



emerging Overlay Traffic needs and the more complex requirements of advanced services (beyond caching) such as the one of this scenario: Thus, the SmartenIT OTT stakeholder can be envisioned, serving as the meeting point/clearing-house for networks and data centres who use its “smart” scheduling service in order to ensure that their interests are considered in the service provisioning and the way traffic is managed. In both options however, the alignment of a critical mass of the key stakeholders’ incentives regarding the scheduling and management of traffic is required for the adoption of this service by the market. This critical set of stakeholders could promote this new business role and by taking advantage of their existing customer base and business agreements with key market players enforce the desired solution as a de facto standard in the market. Other players in the market would either buy this service if it matches their business needs or try to offer similar solutions in order to be able to remain competitive in the market. This would further strengthen the added value and sustainability of the envisioned SmartenIT Aggregator service for inter data centre communication.

5.4 Collaboration for Energy Efficiency

In this section, we perform stakeholder relationship analysis for two scenarios where energy efficiency is pursued through collaboration among stakeholders: in the first case, collaboration is sought between data centres, i.e., cloud operators, while in the second one, collaboration is considered between the data centre and the ISP, and consequently the end-users. Note that apart from fundamental stakeholders identified in cloud service scenarios (as in Section 5.3) such as Cloud Operator, ISP, Application provider, we also consider here a new entity/stakeholder, namely that of the Energy Provider, and address his role(s) and interests in Section 5.3.2.

5.4.1 Scenarios

Below, we describe two scenarios where energy efficiency is pursued through collaboration among different stakeholders. Both scenarios are considered to be of high interest to SmartenIT; the first one is an extension of the scenario described in Section 5.3.1, while the second one is inspired by [192].

5.4.1.1 Collaboration between Data Centres

Federation of data centres and live migration of virtual machines offer new effective ways to manage and optimize energy consumption. This allows establishing new energy saving policies and procedures dealing with energy demand patterns of data centres and collaboration with energy producers. Apart from energy saving, also the exploitation of renewable energy sources may have high priority.

Federated data centres create a decentralized energy ecosystem where the workload can be exchanged to address energy shaping requests from the energy providers, economical demands (lower costs of energy in certain locations) or prioritizing access to stable renewable energy sources.

Advanced adaptive power consumption requires close collaboration between data centres and energy providers [117]. Normally the data centres are treated as common business customers expecting high level of energy but to achieve valuable benefits this relation has to be enriched. There are already efforts introducing interfaces between those two sides to share the information and dynamically adjust power consumption.



The information about energy consumption and related implications can be included in SLAs between service users and service providers. Possible and measurable limitations could be compensated with discounts or other benefits in order to achieve high optimization and thus lower energy consumption. On the other hand users could require from service providers to use substantial level of green energy (or steady increase of it). Also relevant OLAs including energy consumption aspects could be established between data centres for workload migration and access to distributed energy sources.

Figure 5-9 presents an example with two federations where virtual machines can be located in a federated data centre according to the optimized decisions.

Figure 5-9: An example of two data centre federations.

5.4.1.2 Collaboration between Data Centre and ISP

In [192], the issue of energy efficiency in data centres is addressed in an alternative way by a distributed service platform, called Nano Data Centres or NaDa, based on tiny (i.e., nano) managed “servers” located at the edges of the network, i.e., users' premises. In NaDa, both the nano servers and access bandwidth to those servers are controlled and managed by a single entity, typically an ISP who is also providing connectivity to end-users. The role of the nano servers can be played by ISP-owned devices like Triple-Play gateways and DSL/cable modems that sit behind standard broadband accesses. According to [192], such gateways can potentially form the core of the NaDa platform and, in theory, can host many of the Internet services hosted in today's data centres, including the increasing popularity video streaming services.

The proposed approach follows a P2P philosophy, but is coordinated and managed by an ISP that installs and runs the gateways that act as nano servers, as well as the network that interconnects them. Additionally, ISPs can easily implement nano servers by providing new customers with slightly over dimensioned gateways, whose extra storage and bandwidth resources would be used to implement services like video hosting, all of that will be totally isolated from the end-user via virtualization technologies.

The architecture proposed in [192] is depicted in Figure 5-10 and consists of: Gateways which provide storage, and bandwidth resources, whereas in the case of VoD service, they store full or partial replicas of video content objects, and provide uplink bandwidth to deliver these objects to other gateways. Additionally, a Tracker coordinates all VoD-related activities, i.e., it monitors the availability and content possession of gateways, and



answers requests for content by lists of gateways holding the desired content. Then, it is up to the requesting gateway to download movie parts from other gateways. Finally, Content Servers provide the content from legacy data centres or caches. They can belong to the ISP, or to content providers, whereas their primary role is to pre-load gateways in offline mode with content that they can subsequently re-distribute. Content servers can also serve content requests online if no gateway can treat the request.

Figure 5-10: The NaDa architecture [192].


According to the description of the scenarios for collaboration between data centres or data centre and ISP so as to achieve energy efficiency, we employ the stakeholders taxonomy described in Section 0 to identify all involved stakeholders and to describe their role. Table 5-3 integrates the identified stakeholders, their role description and a preliminary identification of their interests.

Table 5-3: Integrated stakeholder identification for collaboration for energy efficiency.


Connectivity provider

ISP

Provides connectivity and inter-connectivity between the cloud operators and the application providers, provides connectivity to end-users.

Increase revenues from inter-connection of cloud providers; low network cost; optimized network bandwidth utilization (expected traffic patterns)

Cloud Operators

IaaS Provider

Provides IaaS (servers, links) to users (i.e., PaaS provider, application provider or end-user); owns and controls servers and links inter-connecting his servers, or his servers with other PoPs; establishes OLA with other IaaS or PaaS providers, and SLAs with energy providers

Provide high QoE to users; reduce OPEX (i.e., inter-connection costs, energy costs); improve his servers utilization; avoid / limit congestion on his links; limit energy consumption by his servers

PaaS Provider

Provides PaaS to users (i.e., application provider or end-users); establishes and manages OLAs with IaaS and other PaaS providers, and SLAs with users and energy providers

Provide high QoE to users; reduce OPEX (i.e., IaaS costs); avoid / limit overload on his platform



Energy Provider

Energy Provider

Provides the cloud operators with energy (power); may own and control power plant (producer), or be a re-seller of energy (transmitter)

Meet energy demand, eliminate outages; reduce OPEX including transmission costs; increase revenue

Users

Cloud User Consumer of IaaS or PaaS offered by the cloud operators, e.g., application provider, content provider or end-user

Enjoy high QoE, availability, etc. as dictated by the established SLA; reduce OPEX (i.e., cost for IaaS/PaaS service)

End-User Consumer of cloud and connectivity services, buys and owns equipment for Internet access

Enjoy high QoE as dictated by the established SLA; enjoy high bandwidth and reduce energy cost


In the following sections, we construct certain VNCs which depict possible interactions between major stakeholders of the scenarios under study. Initially, we consider collaboration between federated data centres / cloud operators so as to achieve reduction of the overall energy consumption among data centres. Practically, this means that in order for the cloud operators to cooperate towards overall energy efficiency, they must be provided incentives such as their individual energy and cost reduction. The ISP/Connectivity Provider stakeholder is in purpose omitted in the VNCs for the first scenario as its role is not primary. However, in the second scenario, the ISPs role becomes central as his collaboration is fundamental to achieve energy consumption reduction in the data centres.

We provide first the value network configuration for the collaboration between data centres depicting the technical and business dependencies, as well as the products and services provided by one stakeholder to another. Depending on the level of (vertical or horizontal) integration in the market or the insertion of new entities, there can be a variety of possible value network configurations. This is due to the fact that the roles envisioned in the market and their separation or integration by stakeholders may vary.

In order to highlight this, we begin the presentation with a more generic configuration where there is a clear separation between the stakeholders who provide cloud services focusing solely on their domain of expertise without undertaking additional roles. The resulting VNC is provided as Figure 5-11. Note that under this configuration, IaaS and PaaS Providers are considered to be separate entities, thus no vertical integration applies. There applies though some horizontal integration, in the sense that multiple roles may be integrated under one stakeholder.

Note that the generic VNC model of Figure 5-11 assumes no horizontal or vertical integration in the market. Globally, this specific setup is met quite often, e.g., IRT offers IaaS services to PaaS providers. However, another setup that often appears is the scenario where the IaaS and PaaS are provided by one stakeholder in an integrated fashion, e.g., Amazon or Google. Figure 5-12 depicts such an integrated setup. The main difference to the previous VNC is that OLAs are now required between different cloud operators, i.e., different instances of IaaS/PaaS provider. Still, a hybrid scenario may also occur, e.g., integrated IaaS/PaaS operators may collaborate with IaaS or PaaS only providers so as to achieve energy efficiency.



SLA Mgmt Energy

Transmission

Energy

Production

Power Plant

Ownership

SLA Mgmt

SLA Mgmt OLA Mgmt Platform

providion

Infrastructure

Ownership

Infrastructure

Provision

OLA Mgmt

Cloud

Service

Consumption

Energy Provider

IaaS Provider

PaaS Provider Cloud Service User

SLA Mgmt

SLA

OLA

SLA

Figure 5-11: Generic VNC for separation of IaaS - PaaS Providers.

SLA Mgmt Energy

Transmission

Energy

Production

Power Plant

Ownership

OLA Mgmt

Infrastructure

Ownership

Infrastructure

Provision

SLA Mgmt Cloud

Service

Consumption

Energy Provider

IaaS/PaaS Provider Cloud Service User

Platform

Provision

SLA Mgmt

SLA

SLA

Figure 5-12: Updated VNC for integration of IaaS - PaaS providers.

Next, we consider an alternative VNC where though a new entity is inserted called Energy Broker (see Figure 5-13). The Energy Broker is responsible for orchestrating the collaboration between cloud providers so as to achieve energy efficiency. Thus, a new role is introduced here, as so far collaboration was achieved so far in a distributed manner between the federated data centres. The Energy Broker can be controlled by either a member of the federation, or by a trusted third-party.



SLA Mgmt Energy

Transmission

Energy

Production

Power Plant

Ownership

Energy

Orchestration

Infrastructure

Provision

Cloud

Service

Consumption

Energy Provider

IaaS/PaaS Provider Cloud Service User

Platform

Provision

SLA Mgmt

Energy Broker

OLA Mgmt

SLA Mgmt

OLA

SLA

SLA

Figure 5-13: Extended VNC to depict the insertion of the Energy Broker.

Finally, in Figure 5-14, the VNC for the collaboration between the data centre and the ISP is provided. It can be seen that part of the cloud service is migrated to the End-Users. Additionally, in this VNC, we also illustrate the delivery of energy (power) to End-User by the Energy provider, as the energy consumed is significantly affected due to the cloud service migration at the last mile.

SLA Mgmt Energy

Transmission

Energy

Production

Power Plant

Ownership

Cloud

Service

Consumption

Energy Provider

End-user

SLA Mgmt

SLA

Application

provider / VoD

service

Inter-

Connectivity/

Connectivity

OLA Mgmt

ISP

Cloud/ISP

service

consumption

Infrastructure

Ownership /

Sharing

Cloud operator

Infrastructure

Ownership

Infrastructure

Provision

Platform

provision

SLA Mgmt

SLA

SLA

OLA Mgmt

OLA

Figure 5-14: Extended VNC to depict the distributed approach (NaDa).


We apply the BMRF presented in Section 5.2.2, to build the business model for the collaboration between data centres for energy efficiency scenario. In particular, we select the appropriate values for the key business model parameters described in the template



so as to provide insight to the way business is conducted. We begin our investigation with the first design theme, the Value Proposition, and we apply its basic parameters to this market.

Product/Service Delivered: The energy (i.e., power) is the main product delivered from the Energy provider to the Cloud Providers, the ISP and the Users. By using this energy, the IaaS Provider caters its servers and data centres, and provides storage, system backups, etc. Additionally, the PaaS Provider and OTT/Application Provider use energy to power their platform, and provide computation, VMs, etc. to its customers, e.g., End-Users. Moreover, the End-Users are delivered in a retail level energy by the Energy provider. All of the stakeholders provisioned with energy compensate the Energy provider for the amount of energy consumed.

Secondary products in this scenario are the cloud services provided by the Cloud operators to their Users.

Target Customer: The Energy provider addresses mainly the IaaS and PaaS Providers, i.e., the Cloud Providers; additionally, he targets the Connectivity provider, i.e., the ISP, the Application Provider, e.g., VoD platform, and the End-Users.

Customer Value: The customer value is guaranteed by means of OLAs between Cloud Operators, and SLAs between the Cloud Operator and the User of the cloud service (e.g., OTT provider or end-user), or between the Cloud Operator and the Energy Provider that caters the first.

Resources and Competencies: All three types of stakeholders, i.e., Energy Provider, Cloud operators, Connectivity provider and OTT / Application provider utilize their resources, and competencies in their core business, thus the business model is technology-oriented.


Vertical Integration: It is likely that large IaaS providers that already own or are affiliated with some PaaS provider will perform vertical integration by aggregating the roles of IaaS/PaaS provider.

Customer Ownership and Relationship: There can be an inter-mediated relationship between the Energy provider and the various federated Cloud providers. In such a case, the mediating entity is the Energy Broker and the value flows across the stakeholders through him. Of course, a non-mediated relationship is also possible, where the Cloud operator directly communicates with other Cloud providers and ultimately, with the Energy provider.

Interconnection Modality-Business Agreements: There are several agreements established in the considered scenario; the most important of which include the OLAs among the Cloud Providers, and between the Cloud Providers and the Connectivity Providers, and SLAs between the Cloud Operators and the Energy Provider, where the Energy provider is penalized if violation of the SLA occurs, whereas SLAs between the Cloud operators and the Application provider are also established, where though the Cloud operators are the ones responsible for meeting the SLA terms. Finally, an SLA is necessary to be established between the Cloud operator and the ISP, by which the ISP is committed to offer a certain level of QoS in his connectivity services towards the Cloud provider.

Content-Data Delivery Model: The model closest to this scenario is the peering ISPs model, i.e., ISPs who establish peering agreements so as to achieve reduction of their



inter-connection traffic, i.e., transit costs. The case of the Energy Broker is similar to the P2P case when the tracker coordinates the content delivery among peers, e..g, the federated data centres.

Last but not least, we address the parameters of the Financial Configuration theme.

Revenue model, revenue sharing & money flows: The energy provided by the Energy provider to its customers can be charged on e.g., Time-of-Use (ToU) pricing, where customers pay different prices at different times of the day, while on-peak prices are higher and off-peak prices are lower than a standard rate; another pricing scheme for energy consumption which is typically combined with ToU is Critical Peak Pricing (CPP), where very high critical peak prices are assessed for certain hours on event days (10-15 yearly). Specifically for large customers, such as Cloud operators, a CPP scheme may be followed, namely the Cloud operator will pay a certain, relatively low price, until a threshold is reached. Then, the Cloud operator will be charged with a much higher price, e.g., penalty, for over-utilization and for exceeding the pre-defined threshold. Finally, in the case depicted in Figure 5-13, if the Energy Broker inserted is owned by the federation members, no money flows are foreseen. However, if the Energy Broker belongs to a third-party, then some compensation should be provided to him for the orchestration of the federated data centres activity.

As secondary products, cloud services will then be charged based on either on a pay-per-use basis, or on flat rate. Finally, the Application Provider may charge the end-user for the use of a premium version of the application with a subscription fee, and do not charge him at all for the basic version, e.g., like YouTube, Dropbox and Google Drive do. Alternatively, the Application provider may not charge the end-user, but follow the ad-driven road.

Traffic Charging Scheme: Not applicable in this scenario.

Cost Model: The energy cost can be mitigated by the establishment of a federation among cloud operators through a long-term agreement. Additionally, Cloud Operators can make an investment so as to develop a joint platform and interfaces between their clouds / data centres, e.g., the Energy Broker, in order to efficiently and fairly allocate energy to servers aiming to simultaneously decrease energy consumption by all operators in the federation.

Regarding the scenario for collaboration between the data centre and the ISP, which is based on the scenario described in [192], the business model is similar to the one provided for the collaboration between data centres. The main differences w.r.t. the Value Network Design are highlighted below:

Vertical Integration: Vertical integration is possible between an IaaS and a Connectivity Provider so as to distribute efficiently the cost of energy to the entire network, and specifically, the gateways in End-Users' premises.

Customer Ownership and Relationship: Here, a mediated relationship is necessary. In particular, as the cloud resources, i.e., capacity in gateways, located in End-Users' premises are organized in a P2P manner, some orchestration is required. Thus, a Tracker coordinates all application-related activities, i.e., it monitors the availability and data possession of gateways, and answers requests for content by lists of gateways holding the desired content.

Interconnection Modality-Business Agreements: The SLA between the Cloud operator and the ISP needs to be extended to include not only ISP's commitments to offer a certain



level of QoS in his connectivity services towards the Cloud provider, but also the amount of data/traffic offloaded by the Cloud operator to the ISP's (and consequently his End-Users') premises, the patterns of the traffic, and the related compensation for the ISP. Moreover, a business agreement needs to be established between the ISP and the End-Users; practically, an extension of the End-User's contract, to describe the amount of storage and bandwidth shared from the User's gateway and the compensation for him, e.g., monetary benefit like a reduced price for similar QoE Internet access.

Content-Data Delivery Model: The model closest to this scenario is clearly a P2P-like model, e.g., BitTorrent.

Regarding the Financial Configuration, the following changes apply.

Revenue model, revenue sharing & money flows: Similar revenue model and sharing, and money flows to the previous case. Though, the End-User must be compensated for sharing his bandwidth, storage and computational power, and increasing his energy consumption, due to the higher power consumption by the enhanced gateways. Compensation could be either monetary, e.g., reduction of price charged for connectivity services, or performance-related, e.g., upgrade of the User's access speed/package, or service-related, e.g., provision of an extra service such as free access to the IPTV/VoD platform.

Traffic Charging Scheme: A volume-based or congestion-based scheme can be employed by the ISP to charge the Cloud Operator for the extra traffic flowing over his network; which has also direct impact to the energy consumption by the ISP's networking equipment.

Cost Model: The energy cost will be mitigated for the Cloud operators by the establishment of the collaboration with the ISP. Though, the energy cost will increase for End-Users participating in the P2P architecture by obtained an enhanced gateway. Additionally, the ISP will deal increased energy cost due to the extra traffic trespassing his networking equipment. An investment can be made so as to develop a joint platform and interfaces between the End-Users, the ISP and the Cloud provider, e.g., the Tracker, so as to efficiently and fairly allocate energy to data servers and gateways.


Pursuing collaboration between stakeholders to achieve energy efficiency by employing incentives is one of SmartenIT's key goals. The considered VNCs for the scenario of collaboration for energy efficiency may constitute the basis to develop attractive solutions, while their analysis provides some insight on what expectations each stakeholder will likely express, and also gives some directions on how to address / meet these expectations.

Specifically, we have considered four cases. In the first one, the IaaS and PaaS stakeholders are separate entities, e.g., a cloud provider as IRT and one of its customers, and one where they are assumed to be an integrated one, i.e., vertical integration, e.g., Google. As the various blocks representing stakeholders in the VNC of Section 5.4.3 represent multiple instances of a specific stakeholder, we need to clarify that there is also the possibility of interaction between an integrated IaaS/PaaS Operator to other non-integrated ones IaaS and PaaS Providers. Thus, in second case, a server - customer relationship is considered between the IaaS and PaaS providers. In the aforementioned setups, collaboration to achieve energy efficiency takes place in a distributed manner.



In the third case, the introduction of a new entity called Energy Broker is assumed. This entity is responsible for orchestrating the collaboration between cloud providers so as to achieve energy efficiency, and can be controlled by either a member of the federation, or by a trusted third-party. However, in order for the Energy Broker to be controlled by one member of the federation, then it would be necessary to achieve a win situation not only for the entire federation, but also for each member of it in the long term; otherwise negatively affected members of the federation will desert. Clearly, in such a setup a centralized management of the federated data centres / Cloud Operators takes place to achieve energy efficiency. Then, we consider a fourth case where the Cloud Operators are not burdened, i.e., energy consumption decreases, as part of the storage/computational service migrates to the End-Users' premises. In this setup, the Energy Broker stakeholder's role may be played by either the ISP (Connectivity Provider) as in [117], or a Cloud Operator, or a third-party.

Depending on the considered setup, various inter-connection agreements need to be established between the various Cloud Operators, Connectivity Providers, the Energy Broker and the End-Users, whereas different money flows will be generated.

A SmartenIT mechanism could be seen as a set of architecture/functional elements embodied within the aforementioned stakeholders, or placed as an extra entity among them, so as to coordinate and enforce the collaboration for the efficient energy consumption behaviour among all players. In the first case, the SmartenIT point-of-intervention would be a module or "box" within the entity controlling the data centre, which would be responsible for coordinating with other similar "boxes" within other data centres, and ultimately, communicating with the Energy provider. Alternatively, in the Energy Broker case or the P2P-like one, the SmartenIT point-of-intervention is exactly this broker or tracker orchestrating the energy consumption by the involved parties, e.g., data centres and gateways. Of course, even in such a case, specific "boxes"/modules are necessary within the data centres, which will constitute the interfaces for the collaboration with the broker/tracker. In all cases, alignment of the participating stakeholders' incentives regarding the energy consumption as described in Section 5.4.4 is required for the adoption of such a solution by the market, at least in certain cease where all of them can benefit.

Last but not least, a tussle analysis is needed to take place in order to identify possible conflicts between stakeholders and to investigate the adoption of potential traffic management mechanisms in this scenario. The application of tussle analysis however presumes the (initial) specification of traffic management mechanisms; thus, it is left as future work to be performed within WP2.

5.5 Global Service Mobility

Global Service Mobility relates to the services’ ability to follow user’s location. That could refer to the ability of providing services, or delivering content as close to the user as possible. In more detail, the Global Service Mobility goal is to serve content to possibly moving end-users with high availability and high performance levels, by the means of, e.g., the use of coordinated CDNs, different Cloud Providers, or distributed private data centres.

In order to enable a dynamic service and content reallocation, services should detect when users move to a different location. The word “location” means a different Autonomous System (AS), which may, or may not, represent a different geographic



location. There are well-known techniques to detect user’s mobility. The most used and trivial one is to associate the user’s IP address to the corresponding AS, and check if the user is accessing the service in an AS different than usual. However, nowadays, ASs can present Virtual links to different other ASs, which makes such technique not fully realistic to detect users location. Therefore, other methods can be combined to reach a desired level of accuracy. One possible example would be to combine service’s QoE, response time of most used services actions, traceroute-based measurement results, and AS discovery. The need to combine several techniques always depends on the desired level of accuracy. Moreover, accuracy also depends on how the service is architected and what are the desired levels of performance to be reached.

There are two approaches considering the Global Service Mobility topic: (1) one-time data move, and (2) gradual data move. Even though both approaches are different strategies, they can co-exist as different options to enable service mobility.

In the one-time data move, the service should fully move its resources to the new location at one action. Important aspects like how to technically migrate, where to migrate (e.g., other Cloud Providers), and when to migrate, should be considered.

In the gradual data move, the service should not fully adapt to the user new location straight after user’s move gets detected. Instead, the service should observe (considering a time-frame) if the user remains in the new location. E.g., when the user is temporarily out of its usual location, gradually moves the data closer to the new location considering most frequently accessed content. As the time passes, and considering user interaction with the service, more data should be moved to the new user location. However, trade-offs between long-term and short-term move should be investigated. The gradual data move can be considered a set of smaller one-time data moves but depending on policies (e.g., based on most accessed data, most triggered actions in the service, etc.).

Figure 5-15: Global Service Mobility high-level approaches for moving data resources.




Table 5-4 presents the stakeholders and their interests identification for the Global Service Mobility scenario.

Table 5-4: Stakeholder identification for global service mobility.



Edge ISP Provides connectivity to end-users Reduce / keep low inter-connection costs; avoid network congestion

Transit ISP

Provides inter-connectivity between the various service providers or edge ISPs; provides inter-connectivity between the PoPs of one service provider

Increase revenues from inter-connection of service providers due to content delivery provision; avoid or limit links congestion


Cloud Provider

Provides IaaS (servers, links), PaaS, or SaaS; owns and controls several servers across the globe; inter-connects his servers with other PoPs

Provide high QoE and QoS to end-users; reduce / keep low inter-connection costs; improve his servers utilization; avoid / limit congestion on his links; improve / limit energy consumption by his servers

Co-location Provider

Provides co-location services (servers, storage, links, etc.) with a certain commitment, differently from Cloud Providers; owns and controls several data centres to place co-located servers, disk arrays, and internet links

Provide a continuous service to its customers, respecting an SLA; Protect co-located servers, links, storage disks from any hazard; Co-location Providers’ customers can be either Cloud Providers or Application Providers

Users


Contracts services from Cloud Providers or Co-location Providers to enable its businesses. E.g., Cloud Customer can be a game developing company which contracts IaaS from Cloud Providers in order to provide a game service to end-users

Provide high QoE to end-users; high availability and low service response times; interested to motivate end-users to consume more services

End-user Participates in the online social network; generates / consumes content, i.e., uploads / downloads videos, files, etc.

Enjoy high QoE in terms of low latency and availability, independently of its location; seamlessness


In this section, we describe the relationship between the identified stakeholders in Section 5.5.1. Three use cases under the Global Service Mobility scenario are presented: the first exploring Global Service Mobility without the presence of a Cloud Provider, the second considering a Cloud Provider, and the third assuming the Cloud Providers use the nano data centers concept (within Edge ISPs) instead of Co-location Providers.

Nevertheless, the use cases description for the Global Service mobility scenario are preliminary to be refined and finalized at a later stage of the project, i.e., scenario definition in Task T1.4, use cases definition in Task T2.2 and complete business analysis in Task T2.4. The first use case description is used to show the relations of cloud providers in comparison to the second use case. The second and third use cases are



intended to study QoS, QoE, social awareness, and especially for the third use case: energy efficiency.

5.5.2.1 Application Provider not using Cloud to enable Global Service Mobility

Global service mobility can be achieved without the use of Cloud Providers. However, to achieve the mobility property, the Application Provider should manage users’ data on its own, contracting a Co-location Provider to place its own servers and storage disks in a third-party data centre facility. Without the use of a Cloud Provider, the Application Provider should detect where end-users are located, detect users moving from/to different locations, and therefore move their data (or resources) from/to another data centre – possibly in a different Co-location Provider.

Connectivity

Provision

DataCenter

Location 1

DataCenter

Location n

Inter-

connectivity

Provision 1

Content

Provision

Content

Consumption

End-User

Datacenter (Co-location provider)

Transit ISP

Edge ISP


Inter-

connectivity

Provision n

Figure 5-16: Application Provider without using Cloud to enable Global Service Mobility.

Figure 5-16 depicts an Application Provider which should move end-users’ data from data centre 1 to data centre 2, in order to enable Global Service Mobility and, therefore, provide a high QoS and QoE. Note that data centres are contracted in a co-location fashion (within a Co-location Provider), where the Application Provider has physical servers in a data centre, having the responsibility to not only manage end-users data but also to manage the whole IT infrastructure solution – from the hardware to the application. In this case, the Co-location provider just has an SLA agreed with the Application Provider, to ensure the access to space, power, cooling, and physical security for the Application Provider servers, storage, and networking equipment.



5.5.2.2 Application Providers using Cloud Providers to enable Global Service Mobility (UZH)

In this case, Application Providers use Cloud Providers to enable Global Service Mobility. With the use of Cloud Providers, Application Providers do not need to manage the whole IT infrastructure solution in a Co-location Provider and, in order to move end-users data/resources back and forth, benefit from the elasticity and on-demand properties inherent to cloud computing. Therefore, it is possible to manage end-users data/resources focusing just on the application layer and moving policies (e.g., when to move – based on most triggered actions, most accessed data, etc.).

Figure 5-17 depicts the scenario, showing the relations between the Application Provider and the Cloud Provider. It is important to highlight that, with the use of Cloud Providers and optimizations made within cloud environments, the Global Service Mobility can be enabled also considering energy savings – since the management part of clouds can be optimized, e.g., to share physical nodes to mirror data around the globe, or to process data resources faster in a pool of servers.

Connectivity

Provision

IaaS

Provisioning 1

IaaS

Provisioning n

Inter-

connectivity

Provision 1

Content

Provision

Content

Consumption

End-User

Cloud Provider

Transit ISP

Edge ISP


Inter-

connectivity

Provision n

Content

Cache

Figure 5-17: Application Provider using Cloud Providers to enable Global Service Mobility.

5.5.2.3 Application Providers using NaDa to enable Global Service Mobility

Application Providers can use nano data centres to enable Global Service Mobility. Nano data centres have the goal to create a distributed service platform on tiny servers locate at the edges of the network (Edge ISPs). In NaDa, the nano servers and the link access to these servers are managed by the Edge ISPs. Other use cases are possible where end-users manage nano servers that would enable Application Providers, which are aware of nano servers to form a Private Cloud.



In this case, we focus on the NaDas managed by ISPs. Therefore, we assume the following scenario described in Figure 5-18. Edge ISPs has the responsibility to either provide connectivity to end-users as well as to provide the infrastructure to Cloud Providers. In this manner, Edge ISPs act like Co-location Providers, but providing servers as close to end-users as possible.

When Cloud Providers detect end-users moving to another location, their resources will be moved as close to where they are located. Basically, this is done contacting the Edge ISP and therefore getting access to the nano servers in order to move users’ resources.

Connectivity

Provision

IaaS

Provisioning 1

IaaS

Provisioning n

Inter-

connectivity

Provision 1

Content

Provision

Content

Consumption

End-User

Cloud Provider

Transit ISP

Edge ISP


Inter-

connectivity

Provision n

Content

Cache

Figure 5-18: Application Provider using Cloud with NaDas to enable Global Service Mobility.

5.5.3 Business Model Analysis

We apply the BMRF to the second case described in Section 5.5.2 and is depicted in Figure 5-17, in order to build the business model for the collaboration between Application Providers and a Cloud Provider. We focus on these two stakeholders, as they are relevant for SmartenIT and while Co-location Providers in the first case described in Section 5.5.2 and Private Cloud Providers in third one share functionalities with Cloud Providers as well, they do not provide a single unified access pattern for Application Providers. Cloud Providers will enable the Global Service Mobility scenario. In particular, we select the appropriate values for the key business model parameters described in the template, in order to provide insight to the way business is conducted. We begin our investigation with the first design theme, the Value Proposition, and we apply its basic parameters to this market.

Product/Service Delivered: The capability to move resources close to where end-users are located, without disrupting the service delivered, is the main product from Cloud Providers to Application Providers. Cloud Providers should be aware where to move the



resources and, therefore, should have bilateral agreements with Transit ISPs and possibly with other Cloud Providers. The Cloud Provider that wants to provide global service mobility should have agreements with Transit ISPs due to the amount of traffic generated, and also due to traffic prioritization. Agreements with other Cloud Providers should be made to enable a dynamic and on-demand utilization of other data centres which are located closer to where the end-user moved.

Target Customer: The Global Mobility Service, which is offered by Cloud Providers, mainly targets Application Providers. The Application Provider is mainly interested to contract such service to give a better QoS and QoE to its end-users.

Customer Value: End-users' customer value, i.e., improved response time for their service requests, is guaranteed by means of SLAs between the Application provider and the Cloud Providers.

Resources and Competencies: All the stakeholders, i.e., Application Provider, Cloud Providers, Transit ISPs, Co-location Providers, and Edge ISPs, use their resources and competencies in their core business. Therefore, the business model is technology-oriented.


Vertical Integration: Cloud Providers will integrate several levels of technology. Most important, Cloud Providers will integrate the roles of Co-location Providers and data transfers between Transit ISPs to enable the Global Service Mobility.

Customer Ownership and Relationship: The relationship between Application Providers and Cloud Providers is direct. Cloud Providers offer a service (in this case, the Global Service Mobility) and Application Providers contracts it considering some agreements. Moreover, Cloud Providers should have direct agreements with other Cloud Providers and Transit ISPs to establish a viable way to move resources close to end-users.

Interconnection Modality-Business Agreements: There are several agreements established in the considered scenario; one of the most important is between Application Providers and Cloud Providers. This contract is fundamental and should present parameters related to QoE and QoS. Another very important agreement is between Cloud Providers: in this case, Cloud Providers should agree on prices to have resources on-demand, in specific data centres. Contracts between Cloud Providers and Transit ISPs should consider the amount of traffic generated to other Cloud Providers and, most important, traffic prioritization, since QoE and QoS are the key factors to offer in Global Service Mobility.

Content-Data Delivery Model: The model closest to this scenario is the Cloud model. Application Providers contract Global Service Mobility with Cloud Providers related to their application which is already in the Cloud. Therefore, the Cloud Provider has the responsibility to interface to the application to know where the end-users are located, if end-users moved, and then to move resources close to them. All this work is done in a transparent fashion and, on demand (just if necessary).


Revenue model, revenue sharing & money flows: Cloud Providers have several options to monetize the Global Service Mobility. First, the Global Service Mobility can be done as a subscription. The Application Provider pays a fixed amount to the Cloud Provider to enable such service, world-wide. The Cloud Provider can calculate the price based on,



for example, number of end-users using the application, or amount of data usually generated out of the Cloud Provider. A refined calculation should be made combining several factors of the application. Second, pay-per-use should also be an option. Cloud Providers will just charge Application Providers depending on how many end-users moved and, therefore, based on how many resources had to be re-allocated to different locations. In both options, the Cloud Provider should perform a refined estimative of what exactly to charge, also assessing the risk of not providing Global Service Mobility like agreed beforehand – therefore, the contracts between other Cloud Providers and Transit ISPs should reflect such decision.

Traffic Charging Scheme: It is expected that Cloud Providers and Transit ISPs charge based on bandwidth and QoS parameters.

Cost Model: The cost of employing Global Service Mobility can be mitigated by the establishment of a federation among Cloud Providers and Transit ISPs through a long-term agreement.


Global Service Mobility is an interesting scenario for SmartenIT. The project's interests in respect to the traffic management partially overlap with the global service mobility VNCs presented in this section. In more detail in the first case, when the Application Provider is not using cloud in order to enable Global Service Mobility, the Application Provider should manage the traffic generated by his users with own resources. Furthermore, the Application Provider on its own should handle the end user location detection. Thus, an SLA agreement with a Co-location provider should be established in order to provide the necessary computational resources.

In the scope of SmartenIT, which addressed cloud-based applications/service, clearly the two latter cases presented in Section 5.5.2 are of high interest. In particular, we identified that when the Application Provider is using a Cloud Provider in order to move users data/resources closer to the user, the former stakeholder may focus on the application layer, and the data moving policies. Last but not least, Global Service Mobility could be enabled with the use of NaDas. NaDas might be managed either by the Edge ISPs or by the end-users enabling the Application Providers to form a private cloud. Furthermore, the energy efficiency aspect of the NaDas is of high relevance to the SmartenIT project.

Nevertheless, the definition of the Global Service Mobility scenario and related use-cases is an ongoing process and it is yet to be investigated within Task T1.4 and Task T2.2. Thus, SmartenIT's deeper investigation into these use-cases will be elaborated within D1.2 and D2.2 respectively, once the on-going discussion in the project will be finalized and solid use-cases will be decided.

5.6 Social-Awareness for Content Delivery

In this section, we perform stakeholder relationship analysis for two scenarios where social awareness is employed to achieve efficient content delivery. In both scenarios, similar social information is sought and exploited; though, their main difference relies on the architecture deployed for the content delivery itself, i.e., centralized vs. distributed, provider-driven vs. peer-assisted content delivery.



5.6.1 Scenarios

Below, we describe two scenarios where social information is employed to achieve efficiency in content delivery, in terms of either content placement or pre-fetching. Both scenarios are considered to be of high interest to SmartenIT; the first one is inspired by the evaluation scenario described in [63], while the second one is on scenarios considered in [149] and [150].

5.6.1.1 Exploitation of Social Information by a Centralized Content Delivery Platform

We consider an online social network (e.g., Facebook) having users around the globe who upload user-generated content (e.g., home-made videos) to an online video streaming platform (e.g., YouTube). This content can be viewed by their online friends, their friends' friends, etc. through the Friend-of-Friend (FoF) relationship.

In order to cater the users of the video streaming platform, the video platform is operated on a geo-diverse system comprising multiple points-of-presence (PoPs) distributed globally. These PoPs are connected to each other by links, which can either be owned by the entity that also owns the POPs, or be leased from network providers.

Each user is assigned and served out of their nearest (geographically) PoP, for all of his requests. Placing data close to the users is an approach followed by most CDNs. Therefore all content uploaded by a user A is first uploaded to the nearest respective PoPA. When content is requested by another user B, the nearest PoP to B, i.e., PoPB, is contacted and if the content is available there, the request is served. The content can be already present at PoPB, if content was first uploaded there or was brought there by an earlier request. If the content is not available in PoPB, then a request is made to PoPA and the content is brought to the PoPB.

Figure 5-19: Content dissemination of UGC over an OSN.

The scenario considered in this section is illustrated in Figure 5-20, and as aforementioned, is inspired by the evaluation scenario described in [63]. In such as setup, social relationships between the users of the OSN can be taken into account to to predict where a piece of content will be requested next, i.e., by which PoP. For instance, it is expected that due to their social relationship, users of the online social network will request for a piece of content that a friend of them, i.e., user A, has uploaded to the video platform with higher probability than users that have no social relationship with A. The so-



called social awareness involves the exploitation of such social information (e.g., social graph, social relationships among users), and meta-information (e.g., behaviour patterns, login time, time spent in the OSN) in order to predict where and by whom a video uploaded by a user will be consumed. Such predictions can be employed, also in the context of SmartenIT, to develop social-aware mechanisms such as TailGate proposed in [63] that will enable pre-fetching of the uploaded video in various locations (e.g., POPs).

5.6.1.2 Exploitation of Social Information by a Distributed Content Delivery Platform

We consider again an OSN such as Facebook, whose users scattered around the globe and upload videos, i.e., UGC to an online video streaming platform like YouTube. This content similarly to the scenario described in Section 5.6.1.1 can be viewed by their friends, their friends' friends, etc.

End-users, also called peers, download parts of the file, e.g., chunks, and are considered to be able to re-upload them to another peer. Additionally, a proxy server is considered to participate in the content dissemination, to which a video can be uploaded by a peer, and as well as video originated from the content provider, e.g., a music video clip. Consequently, a proxy server is connected to a content provider, e.g., a TV channel. Moreover, multiple proxy servers are considered to be also distributed globally and each one of them to have a specific domain of influence, e.g., an ISP's domain, an Autonomous System (AS). The initial peer uploading a video to the proxy server (in the case of UGC), the proxy server itself and the peers participating in the dissemination of that particular video are considered a swarm. Furthermore, the video parts exchange among peers is performed based on some specific peer and chunk selection policy.

As aforementioned, placing video chunks close to the end-users is an approach followed by most CDNs as it leads to lower latency and stall time, and thus high QoE for end-users. Therefore, social information is extracted from OSN by the video platform owner, so as to predict by whom a video uploaded to the proxy server will be viewed. According to [149] and [150], direct friends (1-hop friends) and friends of friends (2-hops friends) of a user C have high probability, i.e., more than 80%, to watch a video uploaded or posted by C. Additionally, social meta-information such as user's interests, e.g., sports, music, prove to be also important, as users having a FoF relationship with C and also the same interests are highly possible to view the video uploaded by C. Figure 5-20 illustrates the case, where content is disseminated in a P2P fashion among OSN's users, taken from [149].

Figure 5-20: Structure of a socially aware P2P scheme for content delivery [149].




According to the description of the social awareness scenario, we employ the stakeholders taxonomy described in Section 0 to identify all involved stakeholders and to describe their role. Table 5-5 includes the identified stakeholders, their role description and a preliminary identification of their interests.

Table 5-5: Stakeholder identification for social-awareness for content delivery.



Edge ISP

Provides connectivity to end-users; may own a proxy server assisting video dissemination among his customers, i.e., End-Users

Reduce / keep low inter-connection costs; avoid network congestion

Transit ISP

Provides inter-connectivity between the PoPs/proxy servers of a service provider through leased lines; sells transit to Edge ISPs

Increase revenues from inter-connection of service providers or selling transit to Edge ISPs

Information / service providers

Content Distribution Network / Video Streaming Platform

Owns and controls the video delivery platform; may own PoPs/proxy servers or rent them from a Cloud Provider; buys inter-connectivity between PoPs/proxy servers through leased lines by the transit ISP; establishes SLAs with transit ISPs; decides on the content replication among PoPs/proxy servers; may charge the End-User for access to content, i.e., performs Authentication, Authorization, Accounting (AAA)

Provide high QoE to End-Users; reduce inter-connection costs; reduce management overhead and costs; reduce energy consumption across his own infrastructure

Application Provider / Online Social Network

Provides the online social network service; may own and control proxy servers across the globe or rent them from a Cloud Provider

Provide high QoE to End-Users; reduce inter-connection costs; reduce energy consumption across his own infrastructure; increase revenues from advertisements on the OSN webpage

Content Provider Provides popular content to the CDN; may charge the End-User for access to content, i.e., performs AAA

Provide high QoE to End-Users; increase revenues from content delivery


Cloud Provider / Cloud Operator / IaaS Provider

Provides IaaS (i.e., storage, computation) to service providers, i.e., CDNs and OSNs; may own links interconnecting his data centres or buy inter-connection from Transit ISP

Provide high QoE to service providers; meet SLA requirements; reduce / keep low inter-connection costs; improve his infrastructure utilization; avoid / limit congestion on his links or data centres; improve / limit energy consumption by his servers

Users

End-User

Participates actively in the OSN, e.g., establish friendships, post and comment videos uploaded by friends or friends of friends , related to his interests; generates and uploads content to the CDN, i.e., downloads and consumes

Enjoy high QoE in terms of low latency and stall time; additionally, high service and content availability, access bandwidth and download speed along with low connectivity cost



videos from the CDN


In the following sections, we construct certain VNCs which depict possible interactions between major stakeholders of the scenarios under study. Nevertheless, there can be a multitude of possible value network configurations, though in this section, we reside to three highly relevant to SmartenIT’s scope. Note also that some of the roles which have been identified in Table 5-5, e.g., AAA, are not considered in the following analysis intentionally. The reason for this is the fact that the focus is here on roles directly related to the extraction and use of the social information, thus the aforementioned roles considered to be of secondary importance to our investigation below.

We begin to build the value network configuration for the exploitation of social awareness to provide efficient content delivery through a centralized architecture where the OSN Provider consents to offer social information to the CDN. In particular, we consider a CDN Provider that controls the video delivery platform and offers both popular content by the Content Provider, and UGC by End-Users. Moreover, the CDN makes decision on where the content, i.e., videos, is being stored based on social information and meta-information. Additionally, a Content Provider is considered who provides the popular, non-UGC content to the CDN. Additionally, we consider an OSN Provider that owns and controls the OSN platform and delivers OSN information and meta-information to the CDN. Both the CDN and the OSN Providers are buying IaaS from a consolidated Cloud Operator.

Moreover, we assume an ISP providing access connectivity to End-Users, and a Transit ISP who inter-connects the data centres of the Cloud Provider. Apart from his basic role as connectivity provider, the (edge) ISP is also assumed to have installed and control a local cache/proxy server, which is made available to the CDN provider to reduce distance to the End-Users and improve content delivery performance. Finally, the End-User is practically a service consumer; he consumes content, i.e., watches videos, and he is also using the OSN platform connecting to friends, posting content, sending messages, etc. The produced VNC is illustrated in Figure 5-21.

Connectivity

Provision

DC

Ownership

IaaS

Provision

Content

Cache

Ownership

Inter-

connectivity

Provision

Content

Provision

Content

Placement

OSN Service

Provision

OSN Client/

Activity

Content

Consumption

End-User

Video

Delivery

OSN providerCloud Operator/IaaS provider

Transit ISP

ISP

CDN/Video Platform Service Provider

OSN (Meta)-

Information

Mgmt

Content Provider

Figure 5-21: Basic VNC for Exploitation of Social Awareness for Content Delivery – Centralized; Collaborative.



An alternative setup is one, where the OSN is considered not to consent to provide social information to the CND to perform content placement. In such a setup, the CDN, or the ISP can deploy an OSN crawler extracting as much as possible social information by the social graph and End-Users’ interests, as expressed through their posts, public messages, etc. If the CDN employs the crawler, the information extracted can be directly be used by the Content Placement module to perform its role. Such a setup is depicted in Figure 5-22; the differences to the previous VNC are mainly found in the upper right side of the VNC figure.

Connectivity

Provision

DC

Ownership

IaaS

Provision

Content

Cache

Ownership

Inter-

connectivity

Provision

Content

Provision

Content

Placement

OSN Service

Provision

OSN Client/

Activity

Content

Consumption

End-User

Video

Delivery

OSN provider

Cloud Operator/IaaS provider

Transit ISP

ISP


Content Provider

OSN (Meta)-

Information

Crawler

Figure 5-22: Updated VNC for Exploitation of Social Awareness for Content Delivery – Centralized; Non-Collaborative; CDN-driven.

On the other hand, if the ISP employs the crawler, he can combine the social information with underlay information derived by his network, e.g., network condition (such as level of congestion), physical location of End-Users, latency, delay, etc. and offer an integrated information service to the CDN. The respective VNC is illustrated in Figure 5-23.

Integrated

Information

Provision

Connectivity

Provision

DC

Ownership

IaaS

Provision

Content

Cache

Ownership

Inter-

connectivity

Provision

Content

Provision

Content

Placement

OSN Service

Provision

OSN Client/

Activity

Content

Consumption

End-User

Video

Delivery

OSN provider


Transit ISP

ISP


Content Provider

OSN (Meta)-

Information

Crawler

Underlay

Information

Monitoring

Figure 5-23: Updated VNC for Exploitation of Social Awareness for Content Delivery – Centralized; Non-Collaborative; ISP-driven.



So far, the three VNCs which have been described are referring to a centralized architecture for content delivery, based on the scenario described in Section 5.6.1.1. Next, we extend the setup depicted in Figure 5-22 to address the scenario discussed in Section 5.6.1.2. Therefore, in this setup, we consider a more distributed architecture employing peer-assisted content delivery. Specifically, we assume that storage space in End-Users’ premises is used and orchestrated by the CND to improve performance of content delivery (see Figure 5-24). Nevertheless, the exploitation of End-Users’ capacity, both in terms of storage and bandwidth, is not pre-assumed, as incentives must be provided to them to coincide. Ultimately, the hybrid content delivery architecture can also be combined with the local cache/proxy server in ISPs’ premises considered in all three previous VNCs.

Content Mini-

Cache

Ownership

DC

Ownership

IaaS

Provision

Connectivity

Provision

Inter-

connectivity

Provision

Content

Provision

Content

Placement

OSN Service

Provision

OSN Client/

Activity

Content

Consumption

End-User

Video

Delivery

OSN provider


Transit ISP

ISP


Content Provider

OSN (Meta)-

Information

Crawler

Figure 5-24: Extended VNC for Exploitation of Social Awareness for Content Delivery – Distributed; Non-Collaborative.


We apply also for this scenario the BMRF presented in Section 5.2.2, to build the business model for the exploitation of social information for content delivery. In particular, we select the appropriate values for the key business model parameters described in the template so as to provide insight to the way business is conducted. We begin our investigation with the first design theme, the Value Proposition, and we apply its basic parameters to this market.

Product/Service Delivered: The main product delivered from the Content Provider or the End-Users (depending of the content's nature) to (other) End-Users over the CDN, i.e., video delivery platform is video. Additionally, another important product in the scenarios described in Section 5.6.1 is the OSN service through which the End-Users make online friends, interact with them and their friends' friends, comment, post, upload and download a video.

Target Customer: The CDN Provider addresses the Content Provider and the End-Users; i.e., the CDN distributes through his network the Content Provider’s data, while he also delivers/distributes content requested/uploaded by the End-Users. The Cloud Operator addressed the CND and the OSN Providers offering them IaaS; the ISP and OSN Provider address the End-Users providing them connectivity and OSN service,



respectively; finally, the Transit ISP addresses the (edge) ISP and the Cloud Operator inter-connecting their AS and data centres, respectively.

Customer Value: The customer value is guaranteed by means QoS requirements for content delivery and OSN service (e.g., low latency). Additionally, the customer value between the Cloud Operator and his customers, i.e., CDN and OSN Providers, is guaranteed by means of SLAs.

Resources and Competencies: The Cloud Operator, CDN Provider, OSN Provider, and ISPs utilize their resources, and competencies in their core business, thus the business model is technology-oriented.


Vertical Integration: It is likely that large CDN Providers deploy their own data centres and thus, integrate also the role of the Cloud Operator. Additionally, Cloud Operators usually also own links inter-connecting their data centres, thus aggregating also the role of Transit ISP.

Customer Ownership and Relationship: There can be an inter-mediated relationship between the CDN Provider and the End-Users. In such a case, the mediating entity can be a cache or proxy server located in ISP's premises.

Interconnection Modality-Business Agreements: There are various agreements established in the considered scenario; the most important of which include the SLAs among different Cloud Providers, and between the Cloud Providers and the CDN Providers, or the Cloud Providers and the OSN Providers. If a Cloud Operator does not meet his SLA performance requirements towards the CDN, OSN Provider or another Cloud Operator, he is penalized. Additionally, an SLA is possible to be established between the CDN Provider and the ISP, by which the content stored in ISP's cache/proxy is controlled by the CDN provider. Moreover, an SLA can be established between the Transit ISP and the Cloud Operator; in such a case, the Transit ISP is committed to provide connectivity of certain QoS to inter-connect the data centres of the Cloud Provider. Nevertheless, an SLA agreement is also necessary between the CDN and the Content Provider, whose content is distributed through the CDN's infrastructure.

Content-Data Delivery Model: The CDN model is the dominant one.


Revenue model, revenue sharing & money flows: Usually, the CDN Provider is paid by the Content Provider to replicate the latter's content in multiple servers close to the End-Users. The CDN Provider, then, can pay a Cloud Operator to buy storage capacity in the latter's data centres, while he can also pay an ISP to employ the ISP's cache/proxy server to diminish the distance to his End-Users. Similarly to the CDN case, the OSN provider also pays a Cloud Operator to buy both storage and computation. Moreover, a Content Provider can be paid by an End-User; specifically, the End-User may login using his unique credentials in the Content Provider's portal and rent movies from it paying with a credit card, e.g., the Netflix and Hulu cases. In a socially aware context, the CDN may pay the OSN Provider, i.e., provide him monetary incentives, so as to share with the former one, social information and meta-information about his End-Users.

Traffic Charging Scheme: Not applicable in this scenario.



Cost Model: Inter-connection costs may be imposed by the Transit ISP to the Cloud Operator for connecting his data centres. Additional costs may also be incurred for the CDN Provider due to the provision of local cache/proxy server by the ISP, and the social information potentially made available by the OSN provider. In the peer-assisted case, an extra cost may be inferred for the CDN as he may be obliged to compensate the End-Users for providing their resources for the delivery of non-UG content.

To address the scenario for exploitation of social awareness for peer-assisted content delivery, which is based on the scenario described in [149], we extend the business model described to include also the P2P characteristics of the hybrid, centralized and distributed, overlay for content delivery.

Vertical Integration: Vertical integration is also possible between the CDN provider and the End-Users, in the sense that the End-Users may install and use an application client in their premises which provides them the capability to interact with the CDNs servers (rented by the Cloud Operator) and mainly to be orchestrated by the Content Placement module of the CDN.

Customer Ownership and Relationship: In the peer-assisted case, the mediating entities can also be mini-caches in other End-Users' premises, e.g., [149].

Interconnection Modality-Business Agreements: An agreement needs to be established between the End-Users and the CDN; practically, the agreement will dictate how the End-Users will be compensated for the resources they offer to assist content delivery; the compensation may be monetary, e.g., lower subscription fee, or performance-related, e.g., better QoS.

Content-Data Delivery Model: The (hybrid) peer-assisted CDN model is the dominant one.

Regarding the Financial Configuration, the following changes apply.

Revenue model, revenue sharing & money flows: Similar revenue model and sharing, and money flows to the previous case. Nevertheless, the End-User must be compensated for sharing his bandwidth, storage and computational power. Compensation could be either monetary, e.g., reduction of price charged for connectivity services, or performance-related, e.g., upgrade of the User's access speed/package, or service-related, e.g., provision of an extra service such as free access to the IPTV/VoD platform.

Traffic Charging Scheme: A volume-based or congestion-based scheme can be employed by the ISP to charge the CDN Provider for the extra traffic in his network due to the peer-assisted content delivery.

Cost Model: The transit costs will be mitigated both for the CDN Provider and the edge ISP by the peer-assisted content delivery. Though, OPEX will increase for the ISP due to the extra traffic in his network.


The exploitation of social awareness to employ efficient content placement and therefore content delivery is an upcoming trend which has proven to achieve significant benefit for some of involved stakeholders, e.g., the CDN, the ISP and the End-Users, whereas, on the other hand, a Transit ISP may see his revenue from selling transit reduced. Moreover, the OSN Provider should be given incentives to provide the required information, otherwise the entity employing a socially aware traffic management mechanism should be



in place to perform extensive crawling of the OSN. Nevertheless, the latter choice may not reveal as much social information as a collaboration with the OSN would, but it is a viable solution that still brings significant performance improvements according to work in literature.

A major question related to the social awareness scenario is who will apply the mechanism, e.g., the CDN or the ISP? In Section 5.6.3, both cases have been described; however, different constraints must be taken into account and interactions take place in each one of the setup, which need to be considered when designing a socially aware traffic management mechanism.

Exploiting social awareness to perform traffic management is one of the main objectives of SmartenIT, whereas collaborative approaches employing incentives to persuade the various stakeholders to contend are also highly relevant to the project's scope. The considered VNCs for the scenario of social awareness for content delivery may constitute attractive solutions, while their analysis provides some insight on what expectations each stakeholder will likely express, and also gives some directions on how to address/meet these expectations.

A SmartenIT mechanism could be seen so far as a set of architecture/functional elements embodied within the aforementioned stakeholders. Alternatively, such a module could be employed by a new entity, e.g., an OTT Provider, and placed among the existing stakeholders, so as to collect social information and provide it to the CDN or the ISP, or the End-Users themselves in the peer-assisted setup. Thus, the SmartenIT point-of-intervention can be either a module or "box" within the entity employing the mechanism, which would be responsible for coordinating with other similar "boxes" within other entities, and ultimately, communicating with the Content Provider. Alternatively, in the OTT Provider case or the peer-assisted one, the SmartenIT point-of-intervention is the role played by the OTT or a tracker orchestrating the P2P overlay. Of course, even in such a case, specific "boxes"/modules are necessary within the CDN and ISP, which will constitute the interfaces for the orchestration with the OTT/tracker. In all cases, alignment of the stakeholders' incentives regarding the exploitation of social awareness for content delivery as described in Section 5.6.1 is required for the adoption of such a solution by the market.

As in the previous 4 scenarios, a tussle analysis is needed to take place in order to investigate the adoption of potential traffic management mechanisms in the long term and to identify possible conflicts between stakeholders. The application of tussle analysis however presumes the (initial) specification of traffic management mechanisms; thus, it is left as future work to be performed within WP2.



6 Summary and Conclusions

The purpose of this deliverable was two-fold. The two main target were stated in Section 2.1, and are repeated below:

Target 1 was to identify major characteristics of cloud-based overlay applications and services such as traffic volume generated, traffic patterns, energy consumption, QoE aspects, and end-user's devices implications. Additionally, the potential for intervention and optimization w.r.t. traffic, energy and QoE need to be addressed as the ultimate target of this investigation is to provide input to Deliverable D1.2 and the selection process of the overlay applications to be addressed by the SmartenIT project.

Target 2 was to describe basic scenarios of cloud-service provisioning in which SmartenIT can add value with innovative effective traffic management mechanisms to be developed, identify the involved stakeholders, i.e., network provider, cloud provider, end-users, etc., in these scenarios, as well as their interests in terms of cost, traffic, energy and QoE optimization. Finally, Target 2 was to analyze the identified stakeholders' relationships, so as to gain insight on the technical and business dependencies among them, and identify potential for collaboration among them to achieve a mutually beneficial situation.

The next sections summarize key outcomes and lessons learnt from our investigations towards the two targets, and outline the next steps and open issues to be addressed in the next phases of SmartenIT.

6.1 Key Outcomes & Lessons Learnt

Our study initially presents the definition of crucial terms related to SmartenIT, which is provided in Section 3.1, that gives our understanding of: what cloud computing stands for practically, the differences between cloud computing and grid computing, cloud and CDN, or cloud and data centre. Deployment models, and economic and QoS aspects of cloud computing are briefly addressed in Sections 3.2, 3.3, and 3.4, to develop a common background on the impact of cloud computing on the various Internet stakeholders.

Regarding our Target 1, Section 3.5 describes a variety of cloud services offered by well-known major cloud operators, as the so-called Internet giants, such as Amazon and Google. Most cloud-based applications are built on Internet giant cloud services, e.g., Dropbox employs both Amazon's EC2 and S3 to provide its personal online storage service. This description allows us to understand the architecture and operation of the cloud-based overlay applications described in Chapter 4, to define realistic scenarios of cloud service provision, and last to identify possible stakeholders corresponding to real operators described in Chapter 5.

6.1.1 Traffic Characteristics

An overview of various sources of traffic generated by data centres and cloud is provided in Section 3.6, both as a share of global IP traffic, and in terms of absolute traffic volumes, popularity and future trends. The main conclusions and future projections are as follows: Global data centre IP traffic will nearly quadruple, while it will grow by 31% CAGR over the next 5 years. On the other hand, global cloud IP traffic will increase 6 times and grow by 44% CAGR over the next 5 years, while global cloud IP traffic will account for



nearly two-thirds of total data centre traffic by 2016. Moreover, the rapidly growing number of mobile devices (e.g., smartphones and tablets) used to access new overlay applications constitutes a key driver of the cloud traffic increase. Specifically, it has an important influence on how application ecosystems are implemented nowadays, namely, applications 'run' in the cloud (i.e., computation, data storage), while only a 'shell' providing an interface to access the application remains at the mobile device.

Concerning the efficient management of cloud traffic, a main challenge lies in an optimized distribution of data over traditional and cloud data centres with regard to short transport paths from data centre to the end-user and between data centres themselves so as to achieve energy efficiency. Additionally, energy efficiency is highly important for

mobile devices due to their limited power resources. Characteristics of energy consumption by data centres, cloud, ISPs' networks and end-users' mobile devices have been derived from a survey described in Section 3.7.

Our major conclusions for data centres are as follows: Although the IP traffic was expected to grow by 46% per year in the last 5 years, the energy consumption for data centres did not increase as predicted. In particular, it has been estimated to have increased by up to 36% in the US, due to the use of virtualization that leads to more efficient hardware utilization. Moreover, currently, only, 30% of total energy is delivered to IT equipment for critical operations such as hard disk drive operations, networking, memory, CPU, motherboards, fans, etc., implying PUE equal to 3.3. On the other hand, the remaining 70% of total data centre energy consumption splits to 45% for cooling infrastructure and 25% for power delivery units. Although virtualization is a key driver for energy efficiency in the cloud, it does not provide for energy efficiency in the network domain. SmartenIT will design mechanisms, which will manage cloud traffic by live VMs migration from data centre to data centre, migration that will be strictly tied to data centre resources orchestration and energy efficiency achievements.

Concerning ISPs' networks, the energy consumption has increased with traffic volume and popularity of Internet applications and has become a critical issue for further traffic and bandwidth growth. Fibre optic links and equipment consume more energy than older equipment for switching and routing, while caching infrastructure within ISPs, which is increasingly adopted to shorten traffic paths in content delivery, leads to installation and operation of small data centres and server resources in ISP's premises, which again consume energy. Moreover, energy saving by switching off parts of the network equipment in phases of lower load and reactivating them on demand is often not feasible currently due to long set up periods of networking equipment.

On the other hand, considering energy implications on users' mobile devices of all user/smartphone interactions have been recorded as being related to communication, a) email, SMS, instant messaging and voice calls, nearly 50%, b) browsing, i.e., 12%, and c) media consumption, i.e., 9%. Proposed approaches that aim to optimize the consumed energy utilize the trade-off of transmission range and energy consumption. For instance, cellular technologies like HSDPA or LTE offer wide range communication at a high price in terms of energy, whereas low range technologies like WiFi or Bluetooth have a relatively low energy-footprint and can provide higher data rates.

The investigation in Section 3.6 allows us to foresee that applications such as video and audio on demand/streaming are strong factors in cloud traffic growth, while emerging services such as online storage e.g., Dropbox, are also gaining in popularity. This investigation was continued in Chapter 4, and will provide input to Task T1.1 and specifically Deliverable D1.2, where cloud service classification is to be completed.



Additionally, the investigation of energy implications by cloud traffic in Section 3.7 showed that although cloud improved energy consumption w.r.t. traffic growth; yet significant inefficiencies still exist either in data centres and ISPs' networks, or in end users' mobile devices. Such inefficiencies provide an opportunity for SmartenIT for intervention and optimization.

6.1.2 Cloud-based Applications Characteristics

An extensive overview of a wide selection of cloud based overlay applications is provided in Chapter 4. In particular, we have provided a description of such applications from multiple viewpoints, namely in terms of their operation, traffic characteristics, energy consumption, and end-user's device. The identified characteristics will play important role and provide input to the classification of cloud services, as well as in the selection of applications to be addressed by SmartenIT. Additionally, we highlighted characteristics of new overlay applications that are relevant to SmartenIT, and discussed particularly the potential for intervention and optimization by traffic mechanisms to be developed within SmartenIT.

A comprehensive summary of the characteristics of the considered overlay application categories is provided in Table 4-5 taking also into account the traffic characteristics and energy consumption considerations for cloud services discussed in Section 3.6 and Section 3.7. The main conclusions of this investigation are as follows:

Video streaming, P2P video streaming and file sharing, online storage, cloud CDNs and online gaming generate high or very high traffic volumes, while online social networks, search engines and mobile instant messaging and VoIP generate medium to low volumes,

All categories of applications, except the mobile ones, cause a significant increase of energy consumption in data centres, whereas mobile applications increase the energy consumption in mobile devices, which is higher for video streaming,

Considering QoE aspects, all of these applications require low latency, which may not be always achieved, as their performance is highly dependent on network conditions such as congestion.

Regardless of the type of application, the potential for intervention by a SmartenIT traffic management mechanism is expected to be higher for non-proprietary solutions. The investigation of stakeholders' incentives (as performed in Chapter 5) can help identify intervention potential in specific setups, and enable collaboration between any entities employing (either individually or collaboratively) a traffic management mechanism, e.g., ISPs, cloud IaaS operators, etc., and all other involved stakeholders. Moreover, potential for optimization is considered to be directly dependent both on generated traffic volumes, energy implications and QoE requirements. Thus, potential for optimization is considered to be higher for video-related applications, e.g., video streaming and online gaming, where there is higher potential for a more “visible” impact by SmartenIT. Potential for intervention and optimization will be severely considered as selection criteria by SmartenIT, when the final selection of applications, whose traffic is to be addressed by SmartenIT's mechanisms designed in WP2, will take place in WP1 and specifically Deliverable D1.2.



6.1.3 Stakeholders Characterization

Target 2 was to identify stakeholders and characterize their relationships in four basic scenarios of cloud-service provisioning in which SmartenIT can add value. First, a survey of selected methodologies and tools, i.e., tussle analysis framework, value network configuration and business modeling framework, to perform the stakeholder relationship analysis is provided in Chapter 5.

Tussle analysis was not fully applied in this deliverable; only its first step, i.e., the stakeholders identification based on the stakeholders map generated by the SESERV project was employed to identify involved stakeholders and their interests in the four scenarios of interest. A complete tussle analysis can be the subject of theoretical investigations in WP2 to qualitatively assess the impact of the designed mechanisms in the scenarios of interest, potential conflicts and consequences in the long term. As a next step in this deliverable, we used the business agreement framework to analyse relationships and dependencies among the identified stakeholders, while as part of the business analysis we employed value network configuration to illustrate these relationships and inter-dependencies. Below, we summarize the main outcomes of the stakeholder characterization performed for each of the four scenarios:

The inter data centre communication scenario depicts the need to efficiently and timely carry traffic among data centres that are located in different parts of the network so as to comply with certain market-driven requirements and deadlines. ISPs, Data Centre Providers, or even Application Providers, either existing or emerging, may try to perform bundling of the inter data centre communication service as provided to either other (possibly smaller) Data Centre, or Application Providers and End-Users with existing services, e.g., storage, computation, so as to dominate the market and strengthen their position in the markets where they are active. Clearly the additional offering of such a sophisticated service would be of high value to the data centre customers, whereas this in turn could create mixed incentives for collaboration and competition with the other data centres and also have adverse implications on the agreements with the ISPs. Moreover, the control, market offer and orchestration of the end-to-end inter data centre communication service could be performed either by the Data Centre Provider, which would integrate this SmartenIT functionality in its own product offers, or alternatively by a SmartenIT Application Provider, i.e., a new OTT provider.

Collaboration between cloud operators to achieve energy efficiency can be performed either in a distributed or centralized manner. In the first case, a SmartenIT mechanism could be seen as a set of architecture/functional elements embodied within the aforementioned stakeholders, while in the second one, the SmartenIT mechanism could be developed or placed as an extra entity among them, so as to coordinate and enforce the collaboration for the efficient energy consumption behaviour among all players. Such an entity would be responsible for orchestrating the collaboration between cloud providers, and can be controlled by either a member of the federation, or by a trusted third-party. However, in order for the Energy Broker to be controlled by one member of the federation, then it would be necessary to achieve a win situation not only for the entire federation, but also for each member of it (due to the attained energy efficiency), at least in the long term; otherwise, negatively affected members of the federation will either leave from it, or even decide not to join at all. Nevertheless, depending on the considered setup, various inter-connection agreements need to be established between the various Cloud Operators, Connectivity Providers, the Energy Broker and the End-Users, whereas different money flows will be generated then.



Global Service Mobility involves the capability to move resources between data centres or clouds, close to where the end-users are located, without disrupting the service delivered. In order to achieve Global Service Mobility, the user's location, i.e., AS, needs to be detected; then, service-related data need to be moved to the new user location. For the latter, two major approaches were identified, i.e., one time data move, and gradual data move. The two approaches serve different purposes and applications, while they can also be combined. The stakeholders’ characterization conducted in Section 5.5 revealed that when the Application Provider is using a Cloud Provider in order to move users’ data/resources closer to the user, then the former stakeholder may focus on the application layer and on the data-moving policies, which are subsequently implemented by the Cloud Provider based on the latter's optimization criteria and constraints, e.g., one-time data move, or gradual data move. On the other hand, Global Service Mobility could be achieved by a distributed approach involving also ISPs and end-users; e.g., the NaDa solution could be extended to address service mobility apart from content mobility. The stakeholders and business analysis for this scenario is to be further investigated within WP2following the complete definition of concrete use-cases within Task T2.2.

Finally, the exploitation of social awareness to perform efficient content delivery can be employed both in a centralized manner and in a more decentralized one, i.e., forming a P2P overlay/swarm where also end-users participate by offering the storage and computation resources. A major question related to the social awareness scenario is who will apply the mechanism, e.g., the CDN or the ISP; in any case, though, the social awareness scenario within SmartenIT is considered to be a provider-driven solution, i.e., initiated or deployed by an ISP, cloud operator, etc.. Moreover, a SmartenIT mechanism could be seen either as a set of functional elements embodied within the involved stakeholders, e.g., CDN provider, ISP, end-users' clients, or alternatively as a module that could be employed by a new entity, e.g., an OTT Provider, placed among the existing stakeholders, so as to collect social information and provide it to the CDN, or the ISP. Alternatively, in the peer-assisted setup, the SmartenIT point-of-intervention could be: either a functional module within the ISP, or a new OTT entity introduced, or the enhancement of the overlay tracker orchestrating the P2P overlay to gather and utilize social information.

Generally, collaborative approaches employing incentives to persuade the various stakeholders to collaborate in each one of the four aforementioned scenarios are highly relevant to the project's scope and will drive the design and specification of the traffic management mechanisms within WP2. The analysis in this deliverable has revealed that there is indeed considerable potential for collaboration in a variety of cases.

6.2 Next steps

This deliverable, D1.1, provides a categorization of cloud-based applications based on the services offered to the end-user in order to address the issues related to the main characteristics of the categories they belong to, e.g., traffic volumes generated, or QoS requirements. Though, a complete classification of the cloud-based applications is planned in D1.2, where also Task T1.1 "Cloud Services Classification" is to be concluded.

Additionally, in D1.1, describes the four major scenarios of high interest to SmartenIT and employs stakeholders characterization so as to gain insight on their interests, relationships under different setups and potential for collaboration among them. Nevertheless, D1.1 described four scenarios of interest to SmartenIT based on which the stakeholders



characterization has been performed. However, the complete and final definition of the SmartenIT's scenarios will be provided in the upcoming deliverable D1.2.

The outcome of D1.1 will also serve as input to several other activities within the project. The traffic and energy consumption characteristics of the various cloud-based applications and inter data centre communications will be used by WP1 to identify the potential for optimization in terms of traffic management and/or energy efficiency, as well as to decide whether SmartenIT should rather address a few, high impact applications (and which ones) with tailor-made solutions, or a larger set of applications by designing more general solutions with broader applicability. Moreover, the cloud applications categories described in Chapter 4 will be used by T1.1 and D1.2 to employ cloud services classification, while the four scenarios described and analysed in Chapter 5 will be extended by T1.4 and D1.2, as well as the stakeholders characterization is to be finalized within D1.2.

Furthermore, the identification of stakeholders and their interests, as well as the analysis and characterization of their relationship in four interesting cloud service scenarios will allow SmartenIT, and particularly WP2, to design traffic management mechanisms that will successfully address these stakeholders' incentives, promoting collaboration among them to the extent possible, towards a mutually beneficial situation. In particular, the identification of involved stakeholders' and their interests are already used by T2.1 in Deliverable D2.1 (which has the same delivery date as D1.1) to define requirements of the various stakeholders in cloud service scenarios in order to qualitatively evaluate traffic management approaches in the literature, based on whether they address these requirements or not. Moreover, stakeholders' relationships characterization and potential for collaboration will serve as input to T2.3 and D2.3 to design economic-aware traffic management mechanisms addressing the incentives of the various stakeholders, as identified in D1.1. Ultimately, the business analysis performed in Chapter 5 will be extended within T2.4 in order to investigate theoretical issues related to pricing schemes and incentive mechanisms.



7 SMART objectives addressed

Through this document, two overall (O1 and O2, see Table 7-1) and four specific (O1.4, O2.3, O3.3 and O3.4, see Table 7-2) SmartenIT SMART objectives defined in Section B1.1.2.4 of the SmartenIT DoW [204] have been partially addressed.

The overall Objective 1 is defined in DoW as following:

Objective 1 SmartenIT will lay the basis for its traffic management mechanisms for traffic characterization by the definition of key stakeholders, which require mechanisms with incentive compatibility, QoE.

The specific research area covered by this deliverable is: the extensive investigation of traffic, energy and QoE implications of cloud services and applications (Chapter 3 and Chapter 4). Moreover, we have addressed the identification of key stakeholders and analysed their interests and relationships in four interesting scenarios of cloud service provisioning (Chapter 5). Note that the scenarios are to be finalized in the next deliverable of WP1, i.e., D1.2 (end of M12).

The work performed in Chapters 3 and 4 will provide input to the next deliverable of WP1, i.e., Deliverable D1.2 (end of M12). On the other hand, the analysis of Chapter 5 will be the basis for further analysis of stakeholders relationships, business analysis and identification of possible tussles within WP2 and specifically Task T2.4 "Theoretical Investigations", while it will also provide input to Deliverables D2.2 (end of M12) and D2.4 (end of M24).

Table 7-1: Overall SMART objectives addressed.

Ob-

jective

No.

Specific

Measurable

Achievable Relevant

Timely

Deliverable Number Mile Stone Number

O1

Traffic characterization D1.1 Study, design Basic MS1.1, MS1.2

Stakeholders D1.1 Study, design Basic MS1.1, MS1.2

QoE and social awareness

D1.1 Study, design Advanced MS1.1, MS1.2

O2 Theory design for traffic management mechanisms

D1.1, D2.4, D4.3 Design,

simulation Advanced

MS1.1, MS1.2, MS2.4, MS4.3, MS4.5

Table 7-2: Specific SMART objectives addressed.

ID Specific

Measurable

Achievable Relevant

Timely

Metric Project Month

O1.4

Are decentralized traffic management schemes superior to traditional schemes; if yes, why? If not, what is the associated efficiency loss?

Number of identified and tested scenarios for the

comparison of the different traffic management schemes

Design, simulation T1.2, T2.1,

T2.5

Highly relevant output of

relevance for providers

M12 (design),

M24 (simulatio

n)

O2.3 Which of the designed mechanisms should be selected for implementation

Number of metrics identified for the selection process,

number of possible

Design, simulation

Extremely relevant output of relevance for

M12



and field tests within the project?

mechanisms to choose from T1.2, T2.3 providers and users

O3.3

Which feedback loop and what kind of communication protocol should be used to retrieve and evaluate user’s perceived QoE?

Number of metrics identified for the selection process,

number of possible mechanisms to choose from

Design, simulation, prototyping

T1.3

Extremely relevant output of relevance for providers and

users

M12

O3.4

How to monitor energy efficiency and take appropriate coordinated actions?

Number of options identified to monitor energy

consumption on networking elements and end users

mobile devices, investigation on which options perform

best (yes/no)

Design, simulation, prototyping T1.3, T2.3, T4.1, T4.2,

T4.4

Highly relevant output of

relevance for users

M36

This deliverable contributes to answering the following four specific questions:

1. Objective O1.4: Are decentralized traffic management schemes superior to traditional schemes; if yes, why? If not, what is the associated efficiency loss??

Stakeholders relationship analysis has been applied in cases where either centralized or distributed approaches, e.g., P2P approaches as NaDa, were employed (Chapter 5). The outcome of this analysis is that indeed it may be harder to address stakeholders' incentives in distributed schemes; however, if their incentives are successfully addressed then these schemes are much more stable. Moreover, sometimes distributed schemes even bring higher improvements compared to centralized ones, though at an extra cost, i.e., the overhead to orchestrate the multiple roles/stakeholders. Nevertheless, the identification of end users incentives so as to convince them to provide their local capacity (storage, computation, bandwidth) to others is yet to be performed; it will be the subject of further studies within WP2 and specifically, Task T2.4; therefore, we can claim that O1.4 has been partly addressed by D1.1.

2. Objective O2.3: Which of the designed mechanisms should be selected for implementation and field tests within the project?

As reported in Chapter 4 and concluded also in the lessons learnt section (Section 6.1.2), video and audio streaming, e.g., Netflix, constitutes the hot application in the Internet currently, as it is generating the largest share of total IP traffic. Another significant category of applications is also online storage, e.g. Dropbox, which although depicts lower share of total IP traffic, is forecast to become rapidly much more popular within the next 3-5 years. Final decision on applications to be addressed by SmartenIT's mechanisms is to be made in Deliverable D1.2 (M12).

Nevertheless, final decisions on mechanisms to be selected for implementation must handle traffic generated by the selected applications and successfully address stakeholders' incentives which have been identified in Chapter 5 (see discussion for Objective O1.4 above). Thus, Objective O2.3 has been partly addressed by D1.1.

3. Objective O3.3: Which feedback loop and what kind of communication protocol should be used to retrieve and evaluate user’s perceived QoE?

In Chapter 4, a multitude of cloud-based applications have been described and related QoS/QoE metrics for each one of them have been also visited, e.g., video streaming stalling times, online storage latency or slow download bandwidth, etc.



Depending on the decision on which applications to address by D1.2, respective QoE metrics will be considered. The identification of the feedback loop and metrics to be monitored will be concluded within Task T1.3 (end of M12) and will be reported in D1.2; thus, Objective O3.3 has been partly addressed in D1.1.

4. Objective O3.4: How to monitor energy efficiency and take appropriate coordinated actions?

In Chapter 3, we have investigated energy efficiency aspects and overviewed studies performing measurements of energy consumption in data centres, ISPs' networks and end-users' mobile devices, while we also overviewed methodologies for measurement of energy efficiency in mobile devices. Clearly, different energy efficiency metrics need to be monitored for each one of them. This activity will be concluded within Task T1.3 (end of M12) and will be reported in D1.2; thus, Objective O3.4 has been partly addressed in D1.1.

According to the SMART objectives set within SmartenIT's DoW [204], those ones of relevance for Deliverable D1.1 and the respective tasks within WP1, i.e., T1.2, T1.3 and T1.4, the targeted for objectives have been met.



8 References

[1] The NIST Definition of Cloud Computing, National Institute of Standards and Technology, US Department of Commerce, Special Publication 800-145, P. Mell, T. Grance, September 2011

[2] Akamai: http://www.akamai.com/

[3] Limelight: http://www.limelight.com/

[4] Peer-to-Peer networks: http://en.wikipedia.org/wiki/Peer-to-peer

[5] Bram Cohen, Incentives Build Robustness in BitTorrent, In Proceedings of the 1st Workshop on Economics of Peer-to-Peer Systems (2003)

[6] KaZaA: http://www.kazaa.com/

[7] Skype: http://www.skype.com/

[8] Virtual private networks: http://en.wikipedia.org/wiki/Virtual_private_network

[9] Ian Foster, Yong Zhao, Ioan Raicu, Shiyong Lu, Cloud Computing and Grid Computing 360-Degree Compared, Grid Computing Environments Workshop, 2008. GCE'08. IEEE, 2008.

[10] Frost & Sullivan, Comparing CDN Performance: Amazon CloudFront’s Last Mile Testing Results, available online at: http://media.amazonwebservices.com/FS_WP_AWS_CDN_CloudFront.pdf

[11] The OpenStack project. URL: http://www.openstack.org/

[12] The OpenNebula project. URL: http://opennebula.org/

[13] The BonFIRE project funded by the EC FP7. URL: http://www.bonfire-project.eu/

[14] The GEANT BoD (Bandwidth on Demand). URL: http://bod.geant.net/

[15] http://www.markwilson.co.uk/blog/2011/06/microsofts-windows-azure-data centres-some-statistics.htm

[16] http://www.data centreknowledge.com/archives/2011/04/25/microsoft-reveals-its-specialty-servers-racks/

[17] Bundesnetzagentur, Annual reports on regulatory activities (in German), 2012: www.bundesnetzagentur.de

[18] Australian Bureau of Statistics, pages on Internet activity (2012) http://www.abs.gov.au/ausstats/[email protected]/Lookup/8153.0Chapter7Dec%202011

[19] Office of the Telecommunications Authority (OFTA) of the Hong Kong Special Administrative Region, Statistics of Internet Traffic Volume (2012) <tel_archives.ofca.gov.hk/en/tele-lic/operator-licensees/opr-isp/s2.html>

[20] Cisco Systems, Cisco Global Cloud Index: Forecast and Methodology 2011 - 2016, White paper series, 2012

[21] Sandvine, “Rich Communication Services and Revenue Replacement - Spring 2012 Global Internet Phenomena Report,” Tech. Rep., 2012

[22] G. Haßlinger and F. Hartleb: Content Delivery and Caching from a Network Provider’s Perspective, Special Issue on Internet based Content Delivery, Computer Networks 55 (2011) 3991-4006



[23] Cisco Systems, Visual networking index, forecast and methodology, White paper series (2012)

[24] Alexa Ranking: http://www.alexa.com/topsites

[25] Tinypic: http://tinypic.com/

[26] Pixlr: http://pixlr.com/

[27] Dropbox: http://www.dropbox.com/

[28] "How Dropbox sacrifices user privacy for cost savngs", http://paranoia.dubfire.net/2011/04/how-dropbox-sacrifices-user-privacy-for.html

[29] I. Drago, M. Mellia, M. M. Munafo, A. Sperotto, R. Sadre, A. Pras, Inside Dropbox: Understanding Personal Cloud Storage Services, IMC’12, November 14-16, 2012, Boston, Massachusetts, USA

[30] PiCsMu Website: “Platform-Independent Cross Storage for Multi-Usage”. Available at: http://www.pics.mu. Last visited on: January, 2013

[31] Roberto Gonzalez, Ruben Cuevas, Reza Rejaie, Ángel Cuevas: Google+ or Google-?: Examining the Popularity of the new OSN, CoRR, Volume abs/1205.5662, 2012

[32] Shah Mahmood, Yvo Desmedt: Preliminary Analysis of Google+’s Privacy, Poster, Proceedings of the 18th ACM conference on Computer and communications security, October 17 - 21, 2011, Chicago, IL, USA

[33] "Google+ crosses 500 million total users with over 135 million active users", http://google-plus.com/8430/google-crosses-500-million-total-users-with-over-135-million-active-users/, Google+ official statistics, available in Dec. 2012

[34] "The Process of Invention: OnLive Video Game Service", http://tv.seas.columbia.edu/videos/545/60/79

[35] Kuan-Ta Chen, Yu-Chun Chang, Po-Han Tseng, Chun-Ying Huang, and Chin-Laung Lei, Measuring The Latency of Cloud Gaming Systems, MM’11, November 28–December 1, 2011, Scottsdale, Arizona, USA

[36] Domagoj Baričević, Divyashree Hassan RavindraKumar and Manasa Chandrashekar, GameOn: Analysis and Implementation of Cloud Gaming, available online at: http://www.cs.ucsb.edu/~manasa/cs276.pdf

[37] Mark Claypool, David Finkel, Alexander Grant and Michael Solano, Thin to Win? Network Performance Analysis of the OnLive Thin Client Game System, In Proceedings of the 11th ACM Network and System Support for Games (NetGames), Venice, Italy, November 2012

[38] The Gaikai case study: http://www.nvidia.com/content/PDF/NVIDIA-GeForce-Grid-Gaikai-Case-Study-HR.pdf

[39] Digital Foundry: http://www.digitalfoundry.org/

[40] Gaikai vs. Onlive: http://www.eurogamer.net/articles/digitalfoundry-face-off-gaikai-vs-onlive

[41] CNN: http://www.cnn.com/

[42] ESPN: http://espn.go.com/

[43] Akamai: http://www.akamai.com/



[44] Limelight: http://www.limelight.com/

[45] Level3: http://www.level3.com/

[46] ChinaCache: http://en.chinacache.com/

[47] Mukaddim Pathan, James Broberg, and Rajkumar Buyya, Maximizing Utility for Content Delivery Clouds, Proceedings of the 10th International Conference on Web Information Systems Engineering (WISE '09)

[48] James Broberg , Rajkumar Buyya , Zahir Tari, MetaCDN: Harnessing ‘Storage Clouds ’ for high performance content delivery, Journal of Network and Computer Applications archive, Volume 32 Issue 5, September, 2009, pp. 1012-1022

[49] Whatsapp: http://www.whatsapp.com/

[50] Viber: http://viber.com/

[51] HeyTell: http://heytell.com/

[52] Skype: http://www.skype.com/

[53] Sandvine, “What’s Up with WhatsApp? - Fall 2011 Global Internet Phenomena Report,” Tech. Rep., 2011.

[54] C. Tsiaras, B. Stiller, “Challenging the Monopoly of Mobile Termination Charges with an Auction-based Charging and User-centric System (AbaCUS)” NetSys 2013, Stuttgart, Germany

[55] Tobias Hoßfeld, Raimund Schatz, Ernst Biersack, Louis Plissonneau, Internet Video Delivery in YouTube: From Traffic Measurements to Quality of Experience in Data Traffic Monitoring and Analysis: From measurement, classification and anomaly detection to Quality of experience. Editor(s):Ernst Biersack, Christian Callegari, Maja Matijasevic, Springer’s Computer Communications and Networks series , 2012

[56] Ashwin Rao, Arnaud Legout, Yeon-sup Lim, Don Towsley, Chadi Barakat, Walid Dabbous. Network Characteristics of Video Streaming Traffic. Proc. CoNEXT 2011.

[57] Alessandro Finamore, Marco Mellia, Maurizio Munafo, Ruben Torres, and Sanjay Rao. YouTube Everywhere: Impact of Device and Infrastructure Synergies on User Experience. In IMC, 2011

[58] Leonidas Kontothanassis. Content Delivery Considerations for Different Types of Internet Video. Keynote at ACM Multimedia Systems (MMSys) 2012

[59] Plissonneau, Louis; Biersack, Ernst W; Juluri, Parikshit. Analyzing the Impact of YouTube Delivery Policies on User Experience. Proc. International Teletraffic Congress (ITC 24), September 4-7, 2012, Krakow, Poland

[60] Shane Alcock, Richard Nelson. Application Flow Control in YouTube Video Streams. ACM SIGCOMM Computer Communications Review 41(2). 2011

[61] Vijay Kumar Adhikari, Sourabh Jain, Yingying Chen, Zhi-Li Zhang. Vivisecting YouTube: An Active Measurement Study. Proc. Infocom 2012

[62] Ruben Torres, Alessandro Finamore, Jin Ryong Kim, Marco Mellia, Maurizio M. Munafo, Sanjay Rao. Dissecting Video Server Selection Strategies in the YouTube CDN. Proc. 31st International Conference on Distributed Computing Systems (ICDCS). 2011



[63] S. Traverso, K. Huguenin, I. Trestian, V. Erramilli, N. Laoutaris, K. Papagiannaki, TailGate: Handling Long-Tail Content with a Little Help from Friends, WWW'12, April 16-20, 2012, Lyon, France

[64] Monia Ghobadi, Yuchung Cheng, Ankur Jain, Matt Mathis. Trickle: Rate Limiting YouTube Video Streaming. Proc. USENIX Annual Technical Conference, 2012

[65] Tobias Hoßfeld, Raimund Schatz, Michael Seufert, Matthias Hirth, Thomas Zinner, Phuoc Tran-Gia. Quantification of YouTube QoE via Crowdsourcing. IEEE International Workshop on Multimedia Quality of Experience - Modeling, Evaluation, and Directions (MQoE 2011), Dana Point, CA, USA, December 2011.

[66] Tobias Hoßfeld, Sebastian Egger, Raimund Schatz, Markus Fiedler, Kathrin Masuch, Charlott Lorentzen. Initial Delay vs. Interruptions: Between the Devil and the Deep Blue Sea. QoMEX 2012, Yarra Valley, Australia, July 2012

[67] Tobias Hoßfeld, Raimund Schatz, Martin Varela, Christian Timmerer. Challenges of QoE Management for Cloud Applications. IEEE Communications Magazine, April issue, 2012

[68] Sandvine, “Fall 2012 Global Internet Phenomena Report,” Tech. Rep., 2012

[69] PPStream: http://www.pps.tv/

[70] PPLive: http://www.pptv.com/

[71] Susu Xie, Bo Li, Gabriel Y. Keung, and Xinyan Zhang. Coolstreaming: Design, Theory, and Practice. IEEE Transactions on Multimedia, 9, 2007

[72] SopCast: http://www.sopcast.com/

[73] Salvatore Spoto, Rossano Gaeta, Marco Grangetto, and Matteo Sereno. Analysis of PPLive through active and passive measurements. In IEEE International Symposium on Parallel & Distributed Processing, 2009

[74] Susu Xie, Gabriel Y. Keung, and Bo Li. A Measurement of a Large-scale Peer-to-Peer Live Video Streaming System. In Packet Video 2007, 2007.

[75] Salvatore Spoto, Rossano Gaeta, Marco Grangetto, and Matteo Sereno. Analysis of PPLive through active and passive measurements. In IEEE International Symposium on Parallel & Distributed Processing, 2009.

[76] Long Vu, Indranil Gupta, Jin Liang, and Klara Nahrstedt. Mapping the PPLive Network: Studying the Impacts of Media Streaming on P2P Overlays. Technical Report UIUCDCS-R-2006-2758, University of Illinois at Urbana-Champaign IL 61801 USA, 2006

[77] Shahzad Ali, Anket Mathur, and Hui Zhang. Measurement of Commercial Peer-to-Peer Live Video Streaming. In Workshop in Recent Advances in Peer-to-Peer Streaming, 2006

[78] Hei, X., Liang, C., Liang, J., Liu, Y., & Ross, K. W. A Measurement Study of a Large-Scale P2P IPTV System. In IEEE Transactions on Multimedia, Vol. 9, Issue 8, p. 1672–1687, 2007

[79] Liu, Y., Li, F., Guo, L., & Chen, S. A measurement study of resource utilization in internet mobile streaming. Proceedings of the International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV), 2011

[80] http://paranoia.dubfire.net/2011/04/how-dropbox-sacrifices-user-privacy-for.html



[81] A. Finamore, M. Mellia, M. Meo, M. M. Munafo, and D. Rossi. Experiences of Internet Traffic Monitoring with Tstat. IEEE Network, 25(3):8–14, 2011

[82] M.P. Wittie et al., Exploiting Locality of Interest in Online Social Networks, Proc. ACM CoNEXT, Philadelphia, USA (2010)

[83] Y. Zhang , A. Arvidsson, Understanding the Characteristics of Cellular Data Traffic, Proc. ACM CellNet, Helsinki, Finland (2012) 14-19

[84] G. Haßlinger, A. Schwahn and F. Hartleb, 2-state (semi-)Markov processes beyond Gilbert-Elliot: Traffic and channel models based on 2. order statistics, to appear in Proc. IEEE Infocom, Turin, Italy (2013)

[85] C.W. Dunn, M. Gupta, A. Gerber and O. Spatscheck, Navigation Characteristics of Online Social Networks and Search Engines, Proc. WOSN’12, Helsinki, Finnland (2012).

[86] SymTorrent: http://amorg.aut.bme.hu/projects/symtorrent

[87] μTorrent: http://www.utorrent.com/

[88] Kelenyi, I., Nurminen, J.K., “CloudTorrent – Energy-Efficient BitTorrent Content Sharing for Mobile Devices via Cloud Services,” short paper, 7th IEEE Consumer Communications & Networking Conference CCNC’2010, Las Vegas, Nevada, January 2010

[89] Transmission: http://www.transmissionbt.com/

[90] MobTorrent: http://amorg.aut.bme.hu/projects/mobtorrent

[91] tTorrent Lite: https://play.google.com/store/apps/details?id=hu.tagsoft.ttorrent.lite&hl=en

[92] aTorrent: https://play.google.com/store/apps/details?id=com.mobilityflow.torrent&hl=en

[93] WmTorrent: http://www.wmtorrent.com/home_index.html

[94] BitTorrent for Android: http://www.bittorrent.com/bittorrent-android

[95] uTorrent for Android: http://www.utorrent.com/utorrent-android

[96] BitTorrent Remote: https://remote.bittorrent.com/

[97] μTorrent: https://remote.utorrent.com/

[98] Shye, A., Scholbrock, B., and Memik, G.: Into the Wild: Studying Real User Activity Patters to Guide Power Optimizations for Mobile Architectures. IEEE/ACM International Symposium on Microarchitecture (MICRO), 2009.

[99] Zhang, L., Tiwana, B., Qian, Z., Wang, Z., Dick, R. P., Mao, Z. M., and Yang, L.: Accurate Online Power Estimation and Automatic Battery Behavior Based Power Model Generation for Smartphones. IEEE/ACM/IFIP International Conference on Hardware/Software Codesign and System Synthesis (CODES+ISSS), 2010.

[100] Pathak, A., Hu, Y. C., and Zhang, M.: Where is the Energy Spent Inside my App? Fine Grained Energy Accounting on Smartphones with Eprof. ACM European Conference on Computer Systems (EuroSys), 2012.

[101] Pathak, A., Hu, Y. C., Zhang, M., Bahl, P., and Wang, Y.-M.: Fine-Grained Power Modeling for Smartphones Using System Call Tracing. Conference on Computer Systems (EuroSys), 2011.



[102] Wichtlhuber, M., Rückert, J., Stingl, D., Schulz, and M., Hausheer, D.: Energy-Efficient Mobile P2P Video Streaming. International Conference on Peer-to-Peer Computing (IEEE P2P), 2012.

[103] Balasubramanian, N., Balasubramanian, A., and Venkatarami, A.: Energy Consumption in Mobile Phones: A Measurement Study and Implications for Network Applications. ACM Internet measurement Conference (ICM)

[104] Lee, K., Lee, J., Yi, Y., Rhee, I., and Chong, S.: Mobile Data Offloading. ACM Conference on Emerging Networking Experiments and Technologies (CoNEXT), 2010.

[105] D.D. Clark, J. Wroclawski, K.R. Sollins, R. Braden, Tussle in Cyberspace: Defining Tomorrow’s Internet. IEEE/ ACM Trans. Networking 13, 3, pp. 462-475, June 2005.

[106] C. Kalogiros, C. Courcoubetis, G. D. Stamoulis, M. Boniface, E. T. Meyer, M. Wald- burger, D. Field, and B. Stiller, “The future internet,” ch. An approach to investi- gating socio-economic tussles arising from building the Future Internet, pp. 145–159, Berlin, Heidelberg: Springer-Verlag, 2011

[107] EU FP7 ICT SESERV Project: http://www.seserv.org/

[108] The ETICS Project, Deliverable D3.5: “Final Business Models Analysis”, January 2013

[109] Ghezzi, A.: “A proposal of Business Model Design parameters for Future Internet Carriers”. NETWORKING 2012 WORKSHOPS. v. 7291, pp. 72-79. 2012. Available: <http://www.springerlink.com/content/17u5274303204917/fulltext.pdf>

[110] EU FP7 ICT ETICS Project: https://www.ict-etics.eu/

[111] http://en.wikipedia.org/wiki/Value_network

[112] T. Casey, T. Smura, A. Sorri, Value Network Configurations in wireless local area access. 9th Conference of Telecommunication, Media and Internet, 2010.

[113] A. Kostopoulos, I. Papafili, C. Kalogiros, T. Leva, N. Zhang, D. Trossen, A Tussle Analysis for Information-Centric Networking Architectures, in 4th FIABook Future Internet – From Technological Promises to Reality, Editor(s): F. Alvarez, et al., ISO Press Books, May 2012

[114] Ballon, P.: “Business modeling revisited: the configuration of control and value” Info, Vol. 9 No. 5, pp. 6–19, 2007.

[115] The SESERV project, D2.2 Economic Future Internet Coordination Activities, August 2012

[116] Nikolaos Laoutaris, Michael Sirivianos, Xiaoyuan Yang, and Pablo Rodriguez: Inter data centre Bulk Data Transfers with NetStitcher. ACM Conference on Applications, Technologies, Architectures and Protocols for Computer Communication (SIGCOMM) 2011

[117] All4Green - EU funded project. URL http://www.all4green-project.eu/

[118] Microsoft's Cloud Services: http://www.microsoft.com/oem/en/products/other/Pages/cloud_services.aspx#fbid=ZR4sCkErH9Q (last visited 22.03.2013)

[119] Windows Azure: http://www.windowsazure.com/en-us/home/features/what-is-windows-azure/

http://www.springerlink.com/content/17u5274303204917/fulltext.pdf

http://conferences.sigcomm.org/sigcomm/2011/



http://www.windowsazure.com/en-us/home/features/what-is-windows-azure/

http://www.windowsazure.com/en-us/home/features/what-is-windows-azure/



[120] Getting Started with the Windows Azure: http://msdn.microsoft.com/en-us/library/windowsazure/ff919705.aspx

[121] “Introducing Google Drive... yes, really”, official Google Blog, http://googleblog.blogspot.in/2012/04/introducing-google-drive-yes-really.html

[122] “Google: Buy Storage”, https://www.google.com/settings/storage/?hl=en_US

[123] “Get started with Google Drive: System requirements”, http://support.google.com/drive/bin/answer.py?hl=en&answer=2374990

[124] "Google Docs, Sheets, and Slides size limits”, http://support.google.com/drive/bin/answer.py?hl=en&hlrm=en&answer=37603

[125] “About the Google Drive viewer”, http://support.google.com/drive/bin/answer.py?hl=en&answer=2423485&from=1738646&rd=1

[126] "Powering Google Search", http://googleblog.blogspot.com/2009/01/powering-google-search.html

[127] Cloe Albanesius, How Much Electricity Does Google Consume Each Year, 2011, http://www.pcmag.com/article2/0,2817,2392654,00.asp

[128] "Search Engines Online Trends", http://www.experian.com/hitwise/online-trends-search-engine.html (last visited 25 March 2013)

[129] Yahoo! Search: http://search.yahoo.com/

[130] Yahoo! Directory: http://en.wikipedia.org/wiki/Yahoo%21_Directory

[131] Yahoo! Mail: http://en.wikipedia.org/wiki/Yahoo%21_Mail

[132] Yahoo! News: http://en.wikipedia.org/wiki/Yahoo%21_News

[133] Mike Andrews, “Searching the Internet”, IEEE Software, March/April 2012.

[134] Which Crawlers Does Bing Use? http://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0

[135] "Microsoft 's Bing search queries overtake Yahoo! for the first time", http://techcrunch.com/2012/01/11/microsoft-bing-search-queries-overtake-yahoo-for-the-first-time-in-december/

[136] The size of the World Wide Web: http://www.worldwidewebsize.com/ (last visit 19

th March 2013).

[137] Google Green: http://www.google.com/green/bigpicture/

[138] "Facebook Tops Billion-User Mark". The Wall Street Journal. October 4, 2012.

[139] George Koutitas, Prof. L. Tassiulas, Prof. I. Vlahavas “Energy Efficiency Monitoring in Data Centres: Case Study at International Hellenic University”, Gent, February-2012.

[140] Grren Grid: http://www.thegreengrid.org/

[141] Ipoque Internet Studies: www.ipoque.com

[142] G. Haßlinger, A. Schwahn and F. Hartleb, 2-state (semi-)Markov processes beyond Gilbert-Elliot: Traffic and channel models based on 2. order statistics, to appear in Proc. IEEE Infocom, Turin, Italy (2013)

http://www.pcmag.com/article2/0,2817,2392654,00.asp



[143] G. Haßlinger and O. Hohlfeld, The Gilbert-Elliott model for packet loss in real time services on the Internet, 14th MMB Conference, Dortmund,Germany (2008)

[144] W. Lautenschlaeger and F. Feller, Light-weight traffic parameter estimation for on-line bandwidth provisioning, Proc. ITC 24, Cracow, Poland(2012)

[145] W. Leland, M. Taqqu, W. Willinger and D. Wilson, On the self-similar nature of Ethernet traffic, IEEE/ACM Trans. Networking 2 (1994) 1-15

[146] B. Tsybakov and N. Georganas, On self-similar traffic in ATM queues: Definitions, overflow probability bound and cell delay distribution, IEEE/ACM Trans. on Networking 5 (1997) 397-409

[147] IBM Cloud Computing: http://www.ibm.com/cloud-computing/us/en/

[148] Google Developers: https://developers.google.com/

[149] Ze Li, Haiying Shen, Hailang Wang, Guoxin Liu, Jin Li: SocialTube: P2P-assisted video sharing in online social networks, INFOCOM, 2012 Proceedings IEEE, Orlando, Florida, USA, pp. 2886-2890, 2012

[150] Fangfei Zhou; Liang Zhang; E. Franco; A. Mislove; R. Revis, R. Sundaram: WebCloud: Recruiting Social Network Users to Assist in Content Distribution, 11th IEEE International Symposium on Network Computing and Applications, Cambridge, MA, USA, pp.10-19, 23-25 Aug. 2012

[151] U.S. Environmental Protection Agency: “Report to Congress on Server and Data Centre Energy Efficiency”. ENERGY STAR Program, August 2013. Available at: http://www.energystar.gov/ia/partners/prod_development/downloads/EPA_Data centre_Report_Congress_Final1.pdf. Last visited on: March 2013.

[152] Jonathan G. Koomey: “Growth in Data Centre Electricity Use 2005 to 2010”. Analytics Press, August 2011. Available at: http://www.twosides.us/Content/rsPDF_218.pdf . Last visited on: March 2013.

[153] Google Website: “Efficiency: How we do it”. Google Data Centres Website, 2012. Available at: http://www.google.com/about/datacenters/efficiency/internal/. Last visited on: March 2013.

[154] Cisco White Paper: “The Exabyte Era”. Cisco, White Paper, January, 2008. Available at: http://www.hbtf.org/files/cisco_ExabyteEra.pdf. Last visited on: March 2013.

[155] T. Silverston and F. Olivier, “Measuring P2P IPTV Systems,” in Proceedings of the International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV), 2007.

[156] W. Liang, J. Bi, R. Wu, Z. Li, and C. Li, “On Characterizing PPStream: Measurement and Analysis of P2P IPTV under Large-Scale Broadcasting,” in IEEE Global Telecommunications Conference (GLOBECOM), 2009.

[157] J. Jia, C. Li, and C. Chen, “Characterizing PPStream across Internet,” in 2007 IFIP International Conference on Network and Parallel Computing Workshops (NPC 2007), 2007.

[158] Microsoft SQL Azure training: https://www.microsoftvirtualacademy.com/training-courses/introduction-to-microsoft-sql-azure-training

[159] Windows Azure Data Management: http://www.windowsazure.com/en-us/home/features/data-management/



[160] "What's New in Windows Azure SQL Database", http://msdn.microsoft.com/en-us/library/windowsazure/ff602419.aspx

[161] Windows Live: http://en.wikipedia.org/wiki/Windows_Live

[162] "Explore Zune Software", http://www.windowsphone.com/pl-pl/how-to/wp7/start/get-to-know-the-zune-software

[163] H. Kim, kc claffy, M. Fomenkov, D. Barman, M.Faloutsos, KY. Lee, Internet Traffic Classification Demystified: Myths, Caveats, and the best practices, ACM CoNEXT 2008, December 10-12, 2008, Madrid, Spain.

[164] R. Dey, Z. Jelveh, K. Ross, Facebook Users Have Become Much More Private: A Large Scale Study, 4th IEEE International Workshop on Security and Social Networking (SESOC), Lugano, Switzerland, 2012.

[165] Y.A Wang, C. Huang, J. Li, K.W. Ross, Estimating the Performance of Hypothetical Cloud Service Deployments: A Measurement-Based Approach, Infocom 2011, Shanghai, 2011.

[166] T. Karagiannis, A. Broido, M. Faloutsos and kc klaffy, Transport Layer identification of p2p traffic, ACM IMC, October 2004.

[167] T. Karagiannis, K. Papagiannaki and M. Faloutos, Blinc: Multilevel traffic classification in the dark, In ACM SIGCOMM, August 2005.

[168] T. Choi, C. Kim, S. Yoon, J. Park, B. Lee H. Kim and H. Chung, Content-aware Internet application traffic measurement and analysis, IEEE/IFIP NOMS, April 2004.

[169] T.T.Nguyen and G. Armitage, A survey of techniques for Internet traffic classification using machine learning, IEEE Communications Surveys and Tutorials, 2008.

[170] Amazon EC2: http://aws.amazon.com/ec2/

[171] Amazon EBS: http://aws.amazon.com/ebs/

[172] Amazon S3: http://aws.amazon.com/s3/

[173] Amazon Cloudfront: http://aws.amazon.com/cloudfront/

[174] Sandvine, “Spring 2011 Global Internet Phenomena Report,” Tech. Rep., 2011

[175] Google Search: http://en.wikipedia.org/wiki/Google_Search

[176] GoogleBot: http://en.wikipedia.org/wiki/Googlebot

[177] PageRank: http://en.wikipedia.org/wiki/PageRank

[178] Bing: http://en.wikipedia.org/wiki/Bing

[179] Wolfram Alpha: http://en.wikipedia.org/wiki/Wolfram_Alpha

[180] Twitter: https://twitter.com/

[181] Google+: https://plus.google.com/

[182] Linked'n: http://www.linkedin.com/

[183] SmoothIT Deliverable D1.1: Requirements and Application Classes and Traffic Characteristics (Initial Version), March 2008, http://smoothit.org/index.php?eID=tx_nawsecuredl&u=0&file=fileadmin/public_deli



verables/D1.1-v4.0.pdf&t=1365403007&hash=9cc3ea9bcc066862ffa0281ac305e3d7e377f7d4

[184] Compuware: http://www.compuware.com/

[185] Wongyai, W.; Charoenwatana, L., "Examining the network traffic of facebook homepage retrieval: An end user perspective," Computer Science and Software Engineering (JCSSE), 2012 International Joint Conference on , vol., no., pp.77,81, May 30 2012-June 1 2012, doi: 10.1109/JCSSE.2012.6261929.

[186] Facebook features: http://en.wikipedia.org/wiki/Facebook_features

[187] "Facebook and Online Privacy: Attitudes, Behaviors, and Unintended Consequences", http://www.lib.sfu.ca/sites/default/files/10227/facebook10.pdf.

[188] Vijay Kumar Adhikari, Yang Guo, Fang Hao, Matteo Varvello, Volker Hilt, Moritz Steiner, Zhi-Li Zhang, Unreeling Netflix: Understanding and Improving Multi-CDN Movie Delivery, Proceedings IEEE INFOCOM, 2012.

[189] Netflix: http://www.Netflix.com/

[190] Vijay Kumar Adhikari, Yang Guo, Fang Hao, Volker Hilt, Zhi-Li Zhang, A Tale of Three CDNs: An Active Measurement Study of Hulu and Its CDNs, Global Internet Symposium, 2012.

[191] Hulu: http://www.hulu.com.

[192] Valancius, Vytautas, et al. "Greening the internet with nano data centers."Proceedings of the 5th international conference on Emerging networking experiments and technologies. ACM, 2009.

[193] Cloudnomics: http://www.rackspace.com/knowledge_center/whitepaper/cloudonomics-the-economics-of-cloud-computing

[194] G. Haßlinger, G. Nunzi, C. Meirosu, C. Fan and F.-U. Andersen: Traffic engineering supported by inherent network management: Analysis of resource efficiency and cost saving potential, International Journal on Network Management (IJNM), Vol. 21 (2011) 45-64

[195] G. Haßlinger and O. Hohlfeld, Efficiency of caches for content distribution on the Internet, Proc. 22. Internat. Teletraffic Congress, Amsterdam, The Netherlands (2010)

[196] J. Baliga, K. Hinton, and R. S. Tucker. Energy Consumption of the Internet. In Proceedings of COINACOFT (2007)

[197] J. Baliga, R. Ayre, K. Hinton, W.V. Sorin, and R.S. Tucker. Energy Consumption in Optical IP Networks. Journal of Lightwave Technology, 27(13):2391–2403 (2009)

[198] Vijay Janapa Reddi, Benjamin C. Lee, Trishul Chilimbi, and Kushagra Vaid. 2010. Web search using mobile cores: quantifying and mitigating the price of efficiency. In Proceedings of the 37th annual international symposium on Computer architecture (ISCA '10). ACM, New York, NY, USA, 314-325. DOI=10.1145/1815961.1816002.

[199] Michael Ferdman, Almutaz Adileh, Onur Kocberber, Stavros Volos, Mohammad Alisafaee, Djordje Jevdjic, Cansu Kaynak, Adrian Daniel Popescu, Anastasia Ailamaki, and Babak Falsafi. Clearing the clouds: a study of emerging scale-out



workloads on modern hardware.SIGARCH Comput. Archit. News 40, 1 (March

2012), 37-48. DOI=10.1145/2189750.2150982.

[200] http://www.bing.com/blogs/site_blogs/b/bingjobs/archive/2011/05/26/testing-in-the-bing-cloud.aspx.

[201] Brian F. Cooper, Eric Baldeschwieler, Rodrigo Fonseca, James J. Kistler, P.P.S. Narayan, Chuck Neerdaels, Toby Negrin, Raghu Ramakrishnan, Adam Silberstein, Utkarsh Srivastava, and Raymie Stata, Bulletin of the IEEE Computer Society Technical Committee on Data Engineering.

[202] Google Apps: Energy Efficiency in the Cloud, Google White Paper, http://static.googleusercontent.com/external_content/untrusted_dlcp/www.google.com/en/us/green/pdf/google-apps.pdf.

[203] Wydrych, Piotr (editor). "Deliverable D2.1: Overlay Traffic Management Solutions." SmartenIT, April 2013.

[204] SmartenIT. "Grant Agreement for STREP: Annex I – 'Description of Work (DoW)'." 2012.



9 Abbreviations

3GPP 3rd Generation Partnership Project

AAA Authentication, Authorization, Accounting

API Application Programming Interface

CAGR Compound Annual Growth Rate

CAPEX Capital Expenditure

CDF Cumulative Distribution Function

CDN Content Distribution Network

CPP Critical Peak Pricing

CPU Central Processing Unit

CRM Customer Relationship Management

DC Data centre

DFA Deterministic Finite Automaton

DNS Domain Name System

DoW Description of Work

FoF Friend-of-Friend

GPS Global Positioning System

GSM Global System for Mobile Communication

HSDPA High-Speed Downlink Packet Access

HTML Hyper-Text Markup Language

HTTP Hyper-Text Transfer Protocol

IaaS Infrastructure as a Service

ICT Information and Communications Technology

IP Internet Protocol

IRT Interoute S.A.

ISN Index Serving Node

ISP Internet Service Provider

IT Information Technology

LTE Long-Term Evolution

MNO Mobile Network Operator

MOS Mean Opinion Score

MPLS Multi-Protocol Label Switching

NIST National Institute of Standards and Technology

NSP Network Service Provider



OLA Operation Level Agreement

OPEX Operating Expenditure

OSN Online Social Network

OTT Over-The-Top

P2P Peer-to-Peer

PaaS Platform as a Service

PC Personal Computer

RAM Random Access Memory

RTMP Real Time Messaging Protocol

QoE Quality of Experience

QoS Quality of Service

REST Representational State Transfer

SaaS Software as a Service

SLA Service Level Agreement

SnF Store and Forward

SOAP Simple Object Access Protocol

SSL Secure Sockets Layer

STREP Specific Targeted Research Project

T4T Tit-for-Tat

TCP Transmission Control Protocol

TDG Telekom Deutschland GmbH

ToU Time of Use

TV Television

UDP User Datagram Protocol

VC Value Chain

VDC Virtual Data Centre

VM Virtual Machine

VN Value Network

VNC Value Network Configuration

VoIP Voice over IP

VPN Virtual Private Network

XML X-tensible Markup Language

WAN Wide Area Network

WiFi Wireless Fidelity - 802.11 standards family



10 Acknowledgements

This deliverable was made possible due to the large and open help of the WP1 team of the SmartenIT team within this STREP. Many thanks to all of them.



Appendix



A. YouTube Detailed Traffic Characteristics

In this appendix, detailed traffic characteristics of the YouTube video streaming application are provided based on extensive literature overview, specifically works presented in [55] to [67] are overviewed.

A.1 Steps in YouTube video download

Watching a video on YouTube involves a different set of servers. Initially, the embedding Web page is delivered through front end YouTube web servers, whereas the video content is itself delivered by YouTube video cache servers. YouTube currently supports two containers for video streaming, Flash and HTML5 [56]. At the time of writing, the adoption of HTML5 for YouTube playback on PCs is still in an early phase, and almost all browsers use Flash technology as the default to play the YouTube videos [57]. When the container is Flash, a dedicated Shockwave Flash player must be downloaded to control the Flash plug-in in the browser.

Simplified Steps in Accessing a YouTube Video

The process of accessing a YouTube video can be summarized as follows:

1. The user requests a video on the YouTube webpage: http://www.youtube.com/watch?v=videoID and gets to the Web server that delivers the YouTube HTML page;

2. After downloading the embedding HTML web page, the other contents are requested in particular the Shockwave Flash Player (embedded in a HTML object that contains the video parameters);

3. The actual video content is requested from a cache server (lscache server); if this cache is over-loaded, it sends a redirect (HTTP 302) message to the client indicating another cache server;

4. The client sends a request the other cache server (tccache server) for the video, and the FLV file is delivered to the client while being played in the Flash player (Progressive Download).

The details of the redirections depend on the load of the cache servers and are explained in the following. We now focus on the video content delivery, and more specifically on the architecture and the interaction with the cache server infrastructure.

YouTube Cache Server Infrastructure

The users receive the video they request from a cache node. Individual cache nodes are organized in cache clusters, with all the machines of a cache cluster being co-located. The number of machines per cache cluster is highly variable and depends, among others, on the demand for service issued in the region where the cluster is located and also on the available physical space to host the cache nodes. Each cache node as of 2011 has a 10 Gb/sec network access and 78 TBs of disk storage [58].

The various cache clusters are organized in a three tier hierarchy. The global infrastructure of the YouTube caches has been revealed by Adhikari et al. [59] in 2011. They used the distributed infrastructure of the PlanetLab network to request thousands of videos from different vantage points in the world, which allowed to reverse engineer the cache infrastructure and the cache selection policies. The work in [55] complements their



findings with more active measurements [60] undertaken in 2011 and 2012 from France. The analysis in [55] focuses on residential Internet access and reveals the techniques applied by Google to deliver the videos to residential customers. Since our machines where connected to the Internet through different ISPs, we were able to observe differences in treatment of customers coming from different ISPs.

YouTube has a three tier caching infrastructure that comprises of four different logical namespaces as shown in Figure A-1. The machines allocated to the different cache clusters are identified via particular naming conventions. We recall here the main findings of [61] on the YouTube cache clusters. As of 2011 there are:

38 primary cache clusters with about 5000 unique IP addresses corresponding to the lscache namespace;

8 secondary cache clusters corresponding to the tccache namespaces with about 650 IP addresses;

tertiary caches clusters corresponding to the cache and altcache namespaces with about 300 IP addresses.

Figure A-1: Organization of the YouTube Caches; dashed lines indicate possible redirections - Figure taken from [55].

All these cache clusters are located in a total of 47 different locations distributed over four continents; there is no cache cluster in Africa. About ten primary cache clusters are co-located inside ISPs and the IP addresses of these cache nodes in these clusters are not part of the address space managed by Google but part of the ISPs’ address space. Since we have 38 + 8 + 5 = 51 cache clusters but only 47 different locations, some cache clusters belonging to different levels of the hierarchy must be at the same physical location (i.e., some primary and secondary caches are co-located).

For the caches in each cache cluster, a particular logical naming structure is applied.

Each primary cache cluster has a total of 192 logical caches corresponding to the lscache namespace, which looks as follows: city_code.v[1-24].lscache[1-8].c.youtube.com. As city_code the three letter code for the airport closest to that cache cluster is used.

There are also 192 logical caches in each secondary cache cluster, corresponding to the tccache namespace, which are named as follows tc.v[1-24].cache[1-8].c.youtube.com.

Each tertiary cache cluster has 64 logical caches corresponding to cache and altcache namespaces.



Introducing these logical name spaces has the following advantages:

Each video ID can be deterministically mapped via consistent hashing onto a unique logical name in the lscache namespace, which makes it easy to decide for each cache what portion of the videos it is responsible to serve.

There is a one-to-one mapping between the lscache and tccache namespace.

The logical naming is the same for each cache cluster and it is completely independent of the number of real cache nodes in a particular cache cluster.

It is the responsibility of DNS to map logical cache names onto the IP addresses of real cache nodes. In [61], each of the logical names from the lscache namespace is mapped to more than 75 different IP addresses distributed over the 38 primary cache clusters.

A.2 YouTube redirection process

DNS Level Redirections: YouTube uses HTTP redirections to balance the load among its caches. As shown in Figure, the redirections usually direct the video request from a cache layer to the next one. Using traces from a European ISP, Torres et al. [62] observed that as the total number of requests kept increasing, the percentage of requests handled by the closest cache cluster located inside that ISP decreased from 80% to about 30%. In this case, DNS request resolution will direct clients to more distant but less loaded cache clusters.

Impact of Redirections on Performance: Each redirection involves:

1. DNS query to resolve the hostname of the next cache node,

2. Opening of a new TCP connection,

3. Issuing a new video query.

In case of redirections, the final cache node serving the video will most likely not be the closest one in terms of RTT, which has been observed in [62] for the most popular videos of the day. The impact of redirection on the time until the first MB is downloaded (referred to as video initialization time) has also been studied in [61]. The video initialization time is on average 33% higher if the video has been fetched through redirections. The fraction of sessions that have been redirected is evaluated in [57]: between 10% and 30% of all sessions are redirected at least once. The impact of redirections on the startup delay can also be important [57]:

Without redirections, delays are in the order of milliseconds;

With redirections, delay can increase by orders of magnitude, up to 10 seconds.

A.3 Application-layer traffic management

YouTube videos are requested using HTTP over TCP Both error control and congestion control of TCP may result in high delay jitter.

The delivery strategy of YouTube videos has been studied in great detail by Rao et al. [56]. The authors show that the delivery strategy depends on the video container (Flash, Flash High Definition, or HTLM5), the type of client device (PC or mobile devices such as



smart phones or tablets), and the type of browser (Internet Explorer, Chrome, or Firefox). The delivery strategy needs to reconcile a number of potentially conflicting goals such as:

Smooth playout during the entire duration of a viewing session;

Efficient use of the server resources such as disk I/O and timer management;

Avoid to transmit too much data in advance of consumption in order to (i) reduce the amount of buffering at the client, which is particularly relevant in the case of mobile devices and to (ii) reduce the waste of network and server resources by sending data that are never used.

Finamore et al. [57] observed that 60% of the videos requested were watched for less than 20% of their total duration, resulting in an un-necessary transfer of 25–39% of the data. As we shall see in the following section, the impact of playback degradation is a primary factor in the video transfer interruption. As the video transfer is done via HTTP over TCP, there is no guarantee that the data can be delivered to the client at the rate at least as high as the one at which they are consumed. The details of the transfer have been studied in [63], whose findings we summarize in the following: To increase the likelihood of a smooth playback, YouTube performs aggressive buffering when a video is requested. Initially, during a startup phase, the server sends as fast as possible to fill up the initial client playout buffer. This play-out buffer contains about 40 seconds with Flash, and 10–15 MB with HTML5 with Internet Explorer as a browser, which is typically much more than 40 seconds worth of video. Once the initial buffer has been filled, two other strategies are used by the cache server:

Keeps sending as fast as possible, until entire video is transmitted;

Limits the rate of the transfer alternating between on-off cycles with a fixed period. During an on cycle, a fixed size block of video data is transmitted.

The first option is used for HD content, the second one otherwise [63]. We limit our description to the case of streaming a video to a PC with Flash as container, and refer to the original paper [56] for more details. Streaming the video to a PC has been the most extensively studied case [60], [64]. In this case, the server streaming strategy is independent of the browser: When the startup phase is terminated, the cache server sends blocks of 64 KBs at a frequency that allows achieving an average transmission rate of 1.25 times the video encoding rate. As has been first observed by Alcock and Nelson [60], injecting bursts of 64 KBs means sending 45 maximum size TCP segments back-to-back into the network. Such large packet bursts will accumulate in the buffer of the bottleneck link and (i) cause delay spikes that may disrupt other latency sensitive application and (ii) inflict loss on the bursty YouTube flow itself. In response to these problems, Google engineers have recently proposed a modification to the server side sending policy that controls the amount of packets that can be injected back-to-back in order to limit the size of the packet bursts. The main idea is to adjust the sending window of the TCP connection on the server dynamically based on the estimated round trip time of the connection. For details of the new sender algorithm and its impact on packet loss and burst size see [64].

In this section we have provided the technical basis to understand YouTube content delivery over the Internet. Next, we investigate what influences the QoE experienced by the user. In particular, problems in the network may lead to stalling and QoE degradations. Therefore, we have to identify the key factors that influence YouTube QoE by means of



subjective measurements and build an appropriate model, which can be used for QoE monitoring later on.

A.4 QoE of YouTube video streaming – key influence factors

The following section shows the key results on QoE of YouTube taken from [55]. For a detailed description of the measurement campaign, we refer the reader to [55].

When it comes to predicting QoE of YouTube, an essential step is determining those key factors that have the strongest influence on the actual experience. Therefore, we analyse correlation coefficients as well as support vector machine (SVM) weights [65]. The Spearman rank-order correlation coefficient between the subjective user rating and the above mentioned variables is computed. In addition, SVMs are utilized to obtain a model for classification: Every variable gets a weight from the model indicating the importance of the variable. However, SVMs are acting on two-class problems only and cannot differentiate the five different quality categories which were used in these tests on an absolute category rating (ACR) scale: (1) bad; (2) poor; (3) fair; (4) good; (5) excellent. Therefore, the categories 1 to 3 of the ACR scale to the “bad quality” class and the categories 4 to 5 to the “good quality” class.

Figure A-2(a) shows the results from the key influence analysis. On the x-axis, the different influence factors are considered, while the y-axis depicts the correlation coefficient as well as the SVM weights which are normalized to the largest correlation coefficient for the sake of readability. We can clearly observe from both measures that the stalling parameters dominate and are the key influence factors. It is interesting to note that the user ratings are statistically independent from the video parameters (such as resolution, video motion, type of content, etc.), the usage pattern of the user, as well as its access speed. In particular, we used different video contents, but got no differences on the user ratings. A possible explanation may be that a video user does not care about the video quality (i.e. which encoding scheme is used, what is the video bit rate, etc.). If the video stalls, the video experience of the user is disturbed – independent of the actual video characteristics. Thus, for a YouTube QoE model, those video characteristics can be neglected – in contrast to the actual stalling pattern. However from a networking point of view, higher video bitrates lead to more stalling events if there is a bottleneck in the network. Hence, the video bit rate may be considered for QoE monitoring to estimate the number of stalling events.

However, the videos considered in the experiments [65] had a fixed length of 30 s and no initial delays for buffering the video contents were considered. Therefore, a second row of experiments was conducted [66] in which the following parameters were varied: number of stalling events, duration of a single stalling event, video duration, initial delay. Hence in addition to the first row of subjective studies, the video duration and initial delays were considered as influence factors. To check again the influence of user demographics on QoE, the age, sex, and user id were also considered in the analysis.

The results in Figure A-2(b) reveal again that the number of stalling events together with the stalling length are clearly dominating the user perceived quality, while the user demographics have no influence. Furthermore, the impact of initial delays is statistically not significant. We take a closer look at initial delays that may be accepted by the user for filling up the video buffers to avoid stalling. In case of bad network conditions, providers need to select a trade-off between these two impairment types, i.e. stalling or initial delays, which allows QoE management for YouTube video streaming clouds [67].



(a) Stalling parameters, video characteristics, demographics [65].

(b) Stalling parameters, video duration, initial delays [66].

Figure A-2: Identification of key influence factors on YouTube QoE.

Therefore, we ask the question whether initial delays are less harmful to QoE than stalling events for YouTube. However, the results in Figure A-3 show that no statistical differences are observed for video clips of 30 s. and 60 s. regarding the impact of initial delays on the QoE. QoE is thereby quantified in terms of Mean Opinion Scores (MOS) which is the average value of user ratings on the ACR scale for a given test condition, i.e. a certain initial delay in that case. In summary, the QoE is mainly determined by the number and the length of stalling events. In contrast, the initial delay until the video playback starts is of no or only minor importance.

Figure A-3: Initial delays have almost no influence on MOS for videos of duration 60 s and 30 s – compared to influence of stalling length [66].

A.5 Consequences for SmartenIT

To assess the intervention and optimization potential of a solution for video streaming developed in SmartenIT it is important to be aware of the mechanisms that are currently used for video delivery and to identify the factors that have an influence on the quality of the provided service. In this appendix we considered the most prominent video streaming



platform YouTube and showed its service infrastructure, traffic management strategies and key influence factors.

The well-engineered caching infrastructure and locality aware request redirection shows that in the case of YouTube the potential of optimizing content delivery is rather low, but also provides an example how traffic management is implemented in an efficient CDN. The investigations on QoE show, that a SmartenIT solution for video streaming has to consider stalling as the main influence factor on the user perceived quality.



B. Dropbox Detailed Traffic Characteristics

In this section, detailed traffic characteristics of the Dropbox online storage application are provided based on work presented in [29].

B.1 Monitoring setup

The authors of [29] conducted experiments to identify what information is exchanged and what traffic is generated when particular operations take place. These operations include:

adding or removing files on local folders,

downloading new files, and

creating new folders.

During the data collection, Dropbox client version 1.2.52 was used. As Dropbox communications are encrypted with the TLS protocol and no official description of the Dropbox is published, a local test-bed was employed: a Dropbox client run in a Linux PC and used a Squid proxy server under the authors control. On the proxy server, the SSL-bump module [80] was used to save decrypted traffic flows. Additionally, the original Dropbox Inc. certificate was replaced at run-time by a self-signed one signing the proxy server.

Figure B-1 depicts the messages observed while storing a batch of chunks. Initially, the Dropbox client registers with Dropbox control centre via a clientX.dropbox.com server. As soon as new files are added in the local folder, a commit batch command on client-

lb.dropbox.com submits meta-information. This meta-information may trigger store

operations directly with the Amazon servers on dl-clientX.dropbox.com; store

operations are marked with grey background in the figure. Note that each operation is acknowledged by an OK message. Finally, as chunks are successfully submitted, the client exchanges messages with the Dropbox server on client-lb.dropbox.com to

conclude the transactions.

Figure B-1: An example of the Dropbox communication observed [29].

The knowledge obtained from the aforementioned observations is used to identify with high probability what operations are performed by the (encrypted) monitored TCP flows.

The Dropbox client was found to exchange data flows mostly with Dropbox Inc. servers; in particular, three groups of servers were identified:



i. notification servers which provide information upon request to clients on changes performed elsewhere; these do not included changes in the central storage, as the latter are advertised as soon as they are performed,

ii. meta-data administration servers which are responsible for authentication and file meta-data administration; typically, synchronization transaction start with communication with the meta-data administration servers, and

iii. system-log servers which collect run-time information about the clients, including exception back-traces (via Amazon, on dl-debug.dropbox.com), and other

event logs possibly useful for system optimization (on d.dropbox.com).

Concerning the main store and retrieve operations, these are all handled by VMs in Amazon EC2. Clients were observed to communicate with 500 distinct domain names pointing to Amazon servers in rotation for storage operations in order to distribute the workload. Additionally, content stored in Dropbox can be accessed through the Dropbox web interface. For access through the web interface different domain names are used, e.g., dl-web.dropbox.com for content download from private folders, while

dl.dropbox.com provides public direct links to shared files.

B.2 Methodology and data sets

In [29], the authors extended and used Tstat [81], an open source monitoring tool developed at Politecnico di Torino to collect data. Tstat monitors each TCP connection and acquires information about more than 100 different metrics including: client and server IP addresses, amount of exchanged data, retransmitted segments, etc.

First, in order to extract TLS/SSL certificates offered by the server, a classic DPI approach was used. The analysis showed that the string *.dropbox.com is used to sign all

communications with the Dropbox server. Second, the Fully Qualified Domain Name (FQDN) was requested to the DNS server by the clients for server IP addressed in order to reveal the servers location in the DNS tree hierarchy and to identify each specific Dropbox service. Third, Tstat was instructed to expose the list of device identifiers and folder namespaces exchanged with the notification servers.

Table B-1: Datasets for Dropbox evaluation [29].

Name Type IP addresses Traffic volume

(GB)

Campus 1 Wired 400 5,320

Campus 2 Wired / Wireless 2,528 55,054

Home 1 FTTH / ADSL 18,785 509,909

Home 2 ADSL 13,723 301,448

Monitoring by Tstat took place in 4 points in 2 European countries from March 24th, 2012 to May 5th, 2012. Table B-1 summarizes the dataset for each point, reporting the type of access technology present, the number of unique IP addresses and the total amount of data observed in GBs for the entire monitoring period. Note that datasets named Home 1 and Home 2 refer to ADSL and FTTH customers of a nation-wide ISP; these customers were provided with static IP addresses but the connection could be shared at their home through their WiFi router to many devices. Campus 1 and Campus 2 refer to (as inferred



by their name) to users in two academic environments, i.e., Campus 1 refers to research and administrative offices of the Computer Science Department in a European university, while Campus 2 to the peripheral network of routers of another European university including campus-wide WiFi hotspots and student dormitories.

B.3 Popularity of different cloud storage providers

Initially, a comparison of the popularity of different cloud-based storage systems was conducted by the authors in [29]. More specifically, the following services were considered: Dropbox, Google Drive, Apple iCloud, and Microsoft SkyDrive. Moreover, other less popular services, e.g., SugarSync, Box.com, UbuntuOne) were considered too, and reported in [29] as the Others group.

Figure B-2 illustrates the popularity of all considered services in terms of unique IP addresses belonging to the Home 1 dataset (where static IP addresses are assigned among end users). that contacted the respective servers at least once within a given day. Apple iCloud appears to be the most accessed service, i.e., 11.1% of the total number of IP addresses, followed by Dropbox, i.e., 6.9%; other services present much lower percentages, e.g., Microsoft SkyDrive has 1.7% popularity, while Google Drive just appeared just before the end of the monitoring period, thus traces appear only for a smaller time interval.

Figure B-2: Number of unique clients accessing each cloud-based storage service in Home 1 [29].

Figure B-3: Data volume per cloud-based storage service (in bytes) in Home 1 [29].



Next, Figure B-3 depicts the total data volume for each service in Home 1 dataset. For this metric, Dropbox tops all other cloud-based storage services by at least one order of magnitude, i.e., > 10 GB vs. ~1 GB, with more than 20 GB of traffic exchanged daily. Although iCloud is accessed by more users daily, the data volumes generated are significantly lower, due to the fact that users cannot download/upload arbitrary files in their devices. SkyDrive only presents a sudden increase towards the end of the monitoring period, though it returns back to its regular daily traffic volume afterwards.

In Figure B-4, the share of total traffic volume for YouTube and Dropbox is illustrated for Campus 2. As it can be seen, Dropbox is responsible for 100 GB traffic, which is a quite high share of total traffic and up to 1/3 of YouTube's traffic in a daily pattern. In total, more than 3.5 TB were exchanged with Dropbox servers during the monitoring period.

Figure B-4: Share of total traffic volume in Campus 2 [29].

Generally, the aforementioned findings reveal according to the authors also the increasing popularity of cloud-based storage systems; already 6-12% of home users access such services in a daily basis.

B.4 Performance

The amount of traffic handled by the different sets of servers is studied here. Figure B-5 shows the resulting traffic breakdown in terms of traffic volume and number of flows. It can be observed that the Dropbox client application is responsible for more than 80% of the traffic volume in all points, which implies that the application client is highly preferred over the Web interfaces for exchanging data. Respectively, considering the flows number, then control servers for client communications are the dominant contributors.

As discussed in Section B.1, Dropbox distributes the load among its servers both by rotating IP addresses in DNS responses and by providing different lists of DNS names to each client. Figure B-6 shows the number of contacted storage servers per day in the four points. Figure B-6 points out that clients in Campus 1 and Home 2 do not reach all storage servers daily, while, in both Campus 2 and Home 1, more servers are contacted because of the higher number of devices on those points.

Additionally, during the monitoring, it was identified that all Dropbox requests from clients were directed towards the same IP addresses regardless of their geographical location. This practically implies that the same data centres are used by all users around the globe; thus Dropbox is a centralized service in US.



Additionally, notable is the fact that there is significant difference in the RTT between control and storage servers. Most probably this is due to the fact that control servers are located in the U.S. West Coast (e.g., California), while storage servers are in the U.S. East Coast (e.g., Amazon’s Northern Virginia data-centres); the latter was also revealed by taking into account routing information.

Figure B-5: Traffic breakdown between Dropbox servers [29].

Figure B-6: Number of contacted storage servers [29].

In Figure B-7, we observe that up to 40% of flows exchange less than 10 KB, meaning that they are composed mostly by SSL handshake messages and a small amount of user data. Moreover, a very high percentage of flows, i.e., varying from 40% to 80%, consist of less than 100 KB, while most flows were observed to be composed by only a small number of chunks, i.e., storage flows have no more than 10 chunks in more than 80% of the cases in all datasets. This behavior is most probably due to:

i. the synchronization protocol sending and receiving file deltas as soon as they are detected, or

ii. the primary use of Dropbox for synchronization of small files constantly changed, instead of periodic (large) backups.



Comparing the CDFs for the retrieve and storage operations, we can see that retrieve flows are normally larger than the store ones, which was expected as many devices were noticed to be using Dropbox only for downloading.

Figure B-7: Distribution of TCP flow sizes of file storage for the Dropbox client [29].

Finally, the throughput of the Dropbox application is evaluated. In Figure B-8(a) and Figure B-8(b), the throughput for store and retrieve operations is illustrated. The x-axis represents the number of bytes transferred in the flow, already subtracting the typical SSL overheads, while the y-axis shows the throughput calculated as the ratio between transferred bytes and duration of each flow. The duration was accounted as the time between the first TCP SYN packet and the last packet with payload in the flow, ignoring connection termination delays. Flows are represented by different marks according to their number of chunks, whereas θ is the maximum attainable throughput (note that θ is a loose bound for retrieve operation).

Overall, the throughput is low. Additionally, throughput for store is lower than throughput for retrieve as expected. In general, the highest observed throughput is only achieved by flows carrying more than 1 MB. Moreover, flows achieve lower throughput as the number of chunks increases. This can be seen by the concentration of flows with high number of chunks in the bottom part of the plots for any given size.

Figure B-8: Throughput of storage flows in Campus 2 [29].

Generally, the throughput is limited by two major factors, i.e., TCP slow start-up times and



application-layer sequential acknowledgments, while flows with a small amount of data and flows with a large number of chunks are mostly affected. In both cases, high RTT between clients and data centres amplifies the effects. Those problems are further detailed in [29].

B.5 Service workload

Regarding the ratio of total downloaded data to total upload data, it is observed that users tend to download more than they upload. Specifically, this ratio is equal to 2.4 for users in Campus 2 (on average), 1.6 for Campus 1, 1.4 for users in Home 1 and 0.9 only for users in Home 2; in the latter case only few customers were found to massively upload data and hence affected the ratio significantly.

Users' behaviour can be distinguished in four categories:

i. occasional users, who abandon their Dropbox clients without synchronizing any content,

ii. upload-only users, who mainly perform store operations,

iii. download-only users, who mainly perform retrieve operations, and

iv. heavy users, who perform both store and retrieve for large amounts of data.

The occasional group represents about 30% of total number of IP addresses in all four points. The occasional users exchange negligible amounts of data and stay connected for shortest time intervals (compared to other groups). Therefore, the service workload generated by them is marginal.

The upload-only group accounts for 7% of total population of IP addresses monitored, and it is responsible for up to 21% of total traffic uploaded in Home 1 and 11% in Home 2. The upload-only users are interested mostly in back-ups and submission of data to third-parties or geographically dispersed devices. On the other hand, the download only group accounts for 26% in Home 1 and 28% in Home 2, and generates about 25% and 28% of total download traffic volume in Home 1 and Home 2, respectively.

The fourth group, namely the heavy users, accounts for up to 37% of IP addresses in Home 1 and 33% in Home 2. These users are generating the most of the volume transferred by Dropbox clients, own at least 2 devices, were observed to stay on-line more than 60% of the total monitoring period and initiated more than 50% of total Dropbox sessions. The usage of Dropbox for synchronization is a typical scenario for this group. Therefore, this group is the one having the highest impact on both workload and network utilization.

Concerning the number of devices, only 1 device was observed to be used in about 60% of all households using the Dropbox client. The remaining households may have up to 4 devices (up to 40%) and a few of them even more; as expected most of them belong to the heavy group. Figure B-9 depicts the distribution of the number of devices per household for Home 1 and Home 2.

Next, in order to measure the extent of Dropbox's usage for content sharing and collaborative work, the number of namespaces per device was examined. In particular, the number of users with only 1 namespace (the users’ root folder) is small: 13% in Campus 1 and 28% in Home 1, while having 5 or more namespaces is equal to 50% in the former, and 23% in the latter. In general though, users in Campus 1 have more namespaces than



in Home 1. When considering only IP addresses assigned to workstations in Campus 1, each device had on average 3.86 namespaces.

Figure B-9: Distribution of the number of devices per household [29].

Monitoring the session start-ups by distinct devices and the number of active devices per time interval, it was discovered that in Campus 1 and Campus 2 both metrics are related to employee's office hours (i.e., no activity at night), while in Home 1 and Home 2, some activity was observed early in the morning, and most of the activity took place during evening hours.

Analyzing the session duration, it was observed that in both home networks, a significant number of notification flows most of which are originating from a few devices are terminated in less than 1 minute. These seem to be abruptly terminated due to network equipment (e.g., NATs or firewalls). As depicted in Figure B-10, most devices in Home 1, Home 2 and Campus 2 stay connected up to 4 hours in a single session, while in Campus 1, a significantly higher percentage of long-lasting sessions is seen.

Figure B-10: Dropbox session duration [29].

B.6 Consequences for SmartenIT

In this appendix, we overviewed the in depth traffic characterization of Dropbox provided in [29], which highlights some of Dropbox's inefficiencies, and thus to identify potential for intervention and optimization. We focused in Dropbox as it is the most prominent example of online storage applications, which are observed to win an increasing popularity among



Internet users and to contribute significantly to the IP traffic. To assess the intervention and optimization potential of a solution for online storage developed by SmartenIT in WP2, it is important to be aware of the mechanisms that are currently used for in Dropbox and to identify the factors that have an influence on the quality of the provided service.



C. Facebook and YouTube Traffic Statistics

In this section, measurements for Facebook and YouTube traffic profiles are provided, as well as evaluations of the 2

nd order statistics proposed in [84] as a basis for

traffic characterization and modeling.

C.1 Packet and Flow Measurement for YouTube and Facebook

In [142], the authors evaluated measurements from a link in the aggregation of the IP network of Telekom Deutschland GmbH (TDG) by means of a hardware tool for quality management which captures headers and timestamps of IP packets being exact to the microsecond. In a case study, they analysed traffic on three parallel 1 Gb/s links connecting aggregation routers at the boundary of the backbone. The monitored links connected residential users to the IP network with DSL access bandwidths of up to 16

Mb/s. The results presented in this section are based on a typical measurement trace from December, 18

th 2011, in a busy hour period 7–9 p.m.

Within these two hours, around 600 million IP packets have been counted in the downlink. For the purpose of our statistics, packets are aggregated into flows. A flow is a single TCP connection or UDP application specified by source/destination IP addresses, TCP ports and a QoS marking in the packet headers. This quintuple enables the evaluation of statistics per flow and for aggregates of flows.

Traffic profiles were not only captured for the total traffic, but also for components contributed by relevant applications. The authors examined different classes of the total downstream traffic for the popular application platforms YouTube and Facebook. Traffic has been classified based on IP addresses, ports and traffic pattern rather than using Deep Packet Inspection (DPI). Due to this deliberately simple approach, only a subset of each considered application is identified. Statistics on packets (number, size) and flows (number, size, rate, 2

nd order statistics) have been collected and summarized along with

parameters selected in Table C-1.

Table C-1: Packet and flow measurement for YouTube and Facebook [142].

Measurement

statistics

Number of Packets

Number of Flows

Mean Packet Size [Byte]

Mean Flow Volume [MB]

Mean Flow Rate [Mb/s]

YouTube 87 201 701 34 977 1 462 3.9 2.0

Facebook 11 594 608 14 671 891 0.6 0.2

Complete Traffic 632 705 587 257 010 1 150 2.8 1.0

Table C-1 gives an overview of the mean values for the complete traffic as well as for portions of traffic which have been identified to belong to YouTube and Facebook applications. Moreover, Figure C-1 gives more detailed insights into the cumulative distribution function (CDF) of the packet and flow size. The graphs show that Facebook traffic has shorter packets and flows than the total traffic. In YouTube traffic, 90% of large 1500 Byte packets are dominant corresponding to a large number of full size packets in

long flows. Facebook IP packet sizes are partly small (25% at 50 Byte) with only one half of the packets around 1500 Byte, which indicates smaller flows and/or a mix of different applications. The flow rate distribution shown in Figure C-2 is also largely different with 50% Facebook flows at a mean rate below 10kb/s comparing to less than 10% YouTube



flows in the 10kb/s range, whereas 50% of the YouTube flows are streaming at more than 1Mb/s while only 5% of Facebook flows achieve 1Mb/s.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 500 1000 1500

CDF of the Packet Size [Byte]

Facebook

YouTube

Complete Traffic

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0,1 1 10 100

CDF of the Flow Volume [MByte]

Facebook

YouTube

Complete Traffic

Figure C-1: Statistics on packet and flow sizes [142].

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0,001 0,01 0,1 1 10 100

CDF of the Flow Rate [Mb/s]

Facebook

YouTubeComplete Traffic

Figure C-2: Statistics on the mean traffic rates per flow [142].

C.2 2nd order statistics results for YouTube and Facebook traffic variability

In this section, the variability of traffic components is considered, which has to be taken into account in the dimensioning of network links. Link bandwidth must cover the mean expected traffic rate plus an additional portion depending on traffic variability for sufficient QoS support, e.g., the mean plus 3-fold standard deviation of the traffic rate in an appropriate time scale as a dimensioning rule. Moreover, the 2

nd order statistics of traffic

flows can be used as a criterion that helps to differentiate and identify application types.

C.2.1 Definition of 2nd

order statistics and autocorrelation function

The 2nd order statistics is a basic characteristic of traffic variability [142], [144], [145], [146]. Therefore, a covariance stationary stochastic process

R = {Rt; t = 0, 1,2, …}

with mean = E(R) and variance 2 = E(R 2) –

2 is considered. For N = 1, 2, …, we

denote the average over sequenced blocks of size N in the time series Rt as



R(N)

= {Rt(N)

; t = 0, 1,2, …} with Rt(N)

= (R tN + R tN + 1 + + R tN + N – 1) / N.

The variance N2 of R

(N) is denoted as the 2nd order statistics of the process including 1

2 =

2. An alternative and equivalent measure for the correlation of a process in different time

scales is the autocorrelation function

(k) = cov(R t, R t+k) / 2 = E((Rt – )(R

t+k – ))/

2.

The relationship is due to computation rules for the variance of a sum of N random variables R tN + R tN + 1 + + R tN + N – 1 = N Rt

(N) via covariance coefficients cov(R tN+i, R

tN +j) [142]:

1

1

21

0

1

0

22 )( )()(2),cov(N

kjtNitN

N

i

N

jN kkNNRRN

This equation determines N2 from(1), …, (N–1) and

2. Vice versa, it can also be

transformed in order to obtain (N) from ( 2 =)1

2, 2

2, …,N+1

2.

C.2.2 2nd order statistics of self-similar processes

Self-similar models are defined based on properties of their 2nd

order statistics. A process

R is exactly 2nd order self-similar with Hurst parameter H (0.5 H < 1) [145], [146], if

.)(: 222)(22 HN

N NRN

The same property is also valid for each of the derived processes R(N)

. In this way, the result is transferrable to time scales of coarser granularity, i.e., longer time frames with increasing N. The autocorrelation function is again proven to be invariant for all derived process R

(N) [145], [146].

C.2.3 2nd order statistics for Gilbert-Elliott and 2-state (semi-)Markov models

The 2nd order statistics of a Gilbert-Elliott model has been derived in [143]:

)(

)1(11

)(

))(1(213

222

qpN

qp

qp

hhqppqpp

N

NGB

EEN

with state specific error rates error hG and hB, total error rate pE = (p hG+q hB)/(p+q) and

2

= pE(1 – pE), where p and q are the transition probabilities from state G to B and vice versa from B to G.

As preconditions for the derivation, the 2-state Markov chain is assumed to start in steady state conditions, i.e., Pr{S0 = G}= p/(p+q) and Pr{S0 = B}= q/(p+q) for the initial state S0.An extended result of the same type holds for the 2nd order statistics of 2-state semi-Markov traffic models under the same steady state assumptions. As a general 2-state semi-Markov result derived in [142], it is obtained:

)(

)1(11

)(

))(1(213

222

qpN

qp

qp

qppq

N

NBG

N

qp

qp

qp

qp BGBBGG

;

)()( 2

2222

2

where p and q are again the transition probabilities and µG and µB are the state specific mean traffic rates. As a main result of traffic modeling by self-similar, full 2-state semi-



Markov, Gilbert-Elliott and 2-state Markov-modulated Poisson processes, the adaptation flexibility of full 2-state semi-Markov models is shown to be much better than for all mentioned alternatives, since 2-state semi-Markov models in general provide 2 parameters, which form the shape of their 2

nd order statistic curve, whereas special 2-

state as well as self-similar processes have only a single adaptation parameter for the 2nd

order statistics.

C.2.4 Measurement evaluations of the 2nd

order statistics

The burstiness of traffic measurements was also evaluated in terms of the ratio of the

standard deviation to the mean / µ in different time scales. Figure C-3 shows that the

total traffic has a low burstiness of / µ = 0.3 already in the smallest considered time scale of 1ms. Therefore a Byte count of all packets arriving per 1ms intervals was taken as the traffic volume on this time scale, from which 1ms samples of the variable traffic rate were computed. When statistics became available on the 1ms time scale, then the authors easily summed up the mean traffic rate on larger time scales of 0.01s, 0.1s, 1.0s, …, etc., and thus, obtained the traffic variability on those time scales as well. Generally, the standard deviation of the rate is decreasing for longer time scales.

0

0,5

1

1,5

2

0,001 0,01 0,1 1 10 100 1000Time scale [s]

Sta

nd

ard

Dev

iati

on

/ M

ean

Ra

te

Facebook

YouTube

Total Traffic

=110 100

1000

0

0,5

1

1,5

2

0,001 0,01 0,1 1 10 100 1000Time scale [s]

Sta

nd

ard

Dev

iati

on

/ M

ean

Ra

te

Facebook

YouTube

Total Traffic

=110 100

1000

Figure C-3: Burstiness over time scales for YouTube, Facebook traffic parts [142].

According to [142], evaluation results showed that the identified partial Facebook traffic volume has a burstiness starting from 2.2 for 1ms and stays above the maximum of 0.3 for the total traffic up to the 10s time scale. On the other hand, the burstiness of the YouTube traffic portion lies between the Facebook and the total traffic curves over the entire time scales. The burstiness of traffic aggregates is influenced by the statistical multiplexing effect, i.e., variability is smoothing down in larger traffic aggregates, which is confirmed in the results of Figure C-3.

Moreover, the burstiness also for single flows has been computed. In average, the standard deviation of the data rate of a single Facebook flow is 9.5 times higher than its mean data rate.



C.3 Consequences for SmartenIT

We have evaluated traffic profiles for different fractions of traffic that can be identified to belong to application platforms. Measurement statistics on packets and flows confirm that e.g. YouTube traffic has larger flow volumes than the total traffic mix, whereas Facebook has much more short flows than in the average. The 2

nd order statistics of traffic fractions

is included as a measure of traffic variability which is relevant for proper link dimensioning in order to provide enough extra capacity for temporal overload. An update of packet and flow measurement evaluation is planned for further study within the SmartenIT project with a more detailed view on the current traffic and application mix.