patronus: controlling thousands of geo-distributed micro...

Patronus: Controlling Thousands of Geo-Distributed Micro Data Centers

Sangeetha Abdu Jyothi†, Ruiyang Chen†, Aditya Akella‡, Brighten Godfrey†

†University of Illinois, Urbana-Champaign ‡University of Wisconsin, Madison

Abstract

Micro Data Centers (MDCs), currently being deployed atthe edge of service provider networks, are expected to num-ber in the thousands to support latency-sensitive VNFs,geo-distributed monitoring and analytics, connected cars,and other emerging applications. Managing geographically-separated MDCs is challenging due to the combination ofconstrained resources in each MDC, latency and bandwidthbetween MDCs and stringent requirements of applicationsrunning within and across MDCs. In this paper, we show theneed for a dynamic, scalable resource management schemein this emerging environment, which we call a WAN as aNetwork of Data centers (WAND). The geo-distributed na-ture of the infrastructure and applications necessitate novelAPI and control schemes. To handle the dual burden of lim-ited resource availability and stringent application needs, wedesign Patronus, a closed-loop control mechanism that accu-rately predicts demands, efficiently allocates resources, andadapts to variations, at each individual MDC and collectivelyacross thousands of MDCs. Using central office locations ofa cellular provider and realistic traffic data, we show that Pa-tronus can improve the peak resource usage by up to 47%while meeting application requirements.

1 Introduction

Micro data centers (MDCs) at the edge are emerging asprominent components in the Internet infrastructure. Tradi-tionally, MDCs were used for content caching and video per-formance optimization in Content Delivery Networks [12]and for user load-balancing at entry/exit of content providernetworks such as Google [38], Facebook [31], and Mi-crosoft [21]. With the emergence of novel applications withstringent performance requirements, the role of MDCs is ex-panding. MDCs providing limited compute resources veryclose to users are increasingly used for supporting a vari-ety of low-latency applications, often augmenting, and some-times replacing, the traditional hyperscale clouds.

An important use case for increased processing at theedge is the burgeoning Internet of Things (IoT). IoT devicesproduce data in the form of video, voice, sensor informa-tion, etc., which needs to be analyzed and acted upon, of-ten within tight delay constraints. Processing and compress-ing data from such applications at the edge will also reducethe bandwidth demand in the core. Availability of computein close proximity is a must-have for offloading Augmentedand Virtual Reality (AR/VR) that operate at timescales com-parable to the sensitivity of human perception. Self-drivingand connected cars will also increasingly rely on these edgeDCs [5, 6]. MDCs can also prove beneficial to other latency-sensitive applications with heavy compute requirements suchas real-time video analytics and online gaming.

Today, carrier networks are spearheading large-scale de-ployments of MDCs for supporting Virtualized NetworkFunctions (VNFs). Service providers like AT&T [4] andVerizon [34] are converting traditional Central Offices (COs)with dedicated hardware to MDCs with off-the-shelf servers.A single carrier operates a few thousand COs – AT&T has4700 COs in the US [8] – and with the advent of 5G, thenumber of MDC sites is expected to grow further. In addi-tion to the critical cellular services, these MDCs are also de-signed to support emerging applications like those above [6].

Thus, Wide Area Networks (WANs) are morphing from anetwork of routers to an interconnection between a multitudeof geographically distributed micro data centers. We refer tothis emerging model as a WAN As a Network of Data centers(WAND). WAND is indispensable to providing support forthe emerging hyperconnected world with billions of devicesand geo-distributed applications.

In order to meet the goals of geo-distributed applicationswhile simultaneously utilizing resources efficiently, we needan efficient WAND resource management scheme. However,this is difficult for several reasons. First, the combinationof scale and geographic spread has not been addressed byprior large-scale systems (§ 2.1 Table 1). Second, the envi-ronment needs to support a motley set of applications withdiverse requirements. This includes long-running streaming

applications (e.g., cellular VNFs, other middlebox servicechains), batch analytics (e.g., cellular log analytics, Hadoopjobs) and Lambda-like short-lived jobs (e.g., elastic webservers). To meet the requirements of such geo-distributedhigh-performance applications, resource allocation on dis-tributed MDCs and the interconnecting WAN will need to becoordinated (§ 2.2). Third, the smaller size of MDCs and thepotential for demand bursts mean that resource availabilityin any particular MDC will be more dynamic and variablethan in a hyperscale DC. In other words, MDCs enjoy lim-ited benefits of statistical multiplexing (§ 2.2). Thus, WANDis characterized by its scale, geographic spread, diversity ofapplications, and limited resources at MDCs. While oneor two of these challenges have been addressed in existinglarge-scale systems [17, 15, 21, 31, 38], the combination ofall characteristics calls for novel resource management tech-niques in WAND.

We design an autonomous WAND control system, Pa-tronus, which enables high utilization of the infrastructurewhile improving the performance of applications through(a) fast and efficient resource allocation, and (b) intelligentadaptation during variations. The centralized control planeof Patronus has several components. First, Patronus intro-duces a simple and expressive API which can support rep-resentation of diverse requirements of WAND applicationsusing WAND tags. Tags encode bounds on application re-quirements such as latency, bandwidth, location preference,deadline, etc. Second, Patronus leverages predictability oftraffic and resource usage patterns inherent to the WAND en-vironment/applications to obtain a long-term perspective onresource availability and usage.

Third, Patronus provides fast and efficient resource allo-cation in the complex WAND environment through divisionof labor across an instantaneous scheduler and a long-termscheduler. The long-term scheduler relies on long-term pre-dictions for packing of jobs across time while complyingwith deadlines, fairness, etc. The instantaneous schedulerallocates resources to a subset of tasks deemed active in thecurrent instant by the long-term scheduler. Both schedulersrely on hierarchical optimization for handling diverse ap-plication objectives and a simple mechanism for convertingWAND tags to constraints.

In this paper, we make the following contributions:

• We identify control challenges in an emerging envi-ronment with geo-distributed MDCs and performance-sensitive applications, which we call WAND (WAN as aNetwork of Data Centers), and its differentiating featurescompared to other large-scale infrastructure.

• We design a simple and expressive WAND API for cap-turing the requirements of diverse WAND applications.

• We design Patronus, a dynamic WAND controller provid-ing scalability and high utilization for the WAND providerand high performance for WAND user applications.

• Using realistic data, we evaluate trade-offs in performanceand control overhead using closed-loop control in WANDto meet the high-level goals of applications and operators.

• We show that Patronus can schedule resources acrossthousands of MDCs in seconds while reducing peak re-source usage by up to 47%.

2 Background & Motivation

In this section, we illustrate the differentiating features ofWAND that necessitate novel control solutions.

2.1 Related DomainsOne may draw parallels between control of thousands ofdata centers and interconnecting network in WAND and sev-eral well-studied control domains such as clusters, traditionalISPs, private WANs, geo-distributed analytics, etc. Next, weargue why control solutions in these domains cannot be bor-rowed easily for WAND (overview in Table 1).

Traditional ISPs: Traffic engineering is a well-studiedproblem in traditional ISP networks with a single resource(geographically-distributed network). While the problem hasbeen tackled from different perspectives ranging from obliv-ious routing [3, 20] to traffic-adaptive schemes [18, 36, 11,25], these solutions do not apply to the WAND model. Ex-tending network traffic classes and prioritization to an envi-ronment with demands and performance constraints on mul-tiple resources (WAN network and DC resources like CPU,memory, etc.) is a non-trivial challenge. Besides, the pres-ence of new applications over which the service provider hascomplete control (eg, background analytics jobs) will helpthe provider to drive the network and DCs to high utilization.This was difficult in traditional ISPs which mostly dealt withuser-driven inelastic traffic. As ISPs move to SDN-basedcontrol, there are also efforts to scale SDN with distributedcontrollers. Recursive SDN [23] proposed a hierarchical so-lution for geo-distributed carrier networks. RSDN and otherwork [9, 19] in distributing SDN control, however, only con-siders network control.

Private inter-data center WANs: Inter-DC private WANsolutions such as B4 [17] and SWAN [15] optimize resourcesacross multiple applications similar to WAND. However, pri-vate WANs with a sparse network connecting tens of hyper-scale data centers operate at a much smaller scale (in termsof number of distinct sites) compared to the carrier environ-ment composed of thousands of micro data centers and inter-connecting links. Moreover, the proposed solutions only fo-cus on the bandwidth requirements of the private WAN sincethe DCs are hyperscale and relatively less constrained (or atleast have more reliable statistical multiplexing). In WAND,both DCs and the WAN are frequently constrained. Further-more, quickly adapting to traffic variations is more critical

2

Property WANDTradi-tionalISPs

PrivateInter-DC

WAN

ClusterSchedulers

Application-specific (e.g.,geo-distributed analytics,

NFV platforms, etc.)

Infrastructure

Geographicallyfragmented resources Yes Yes Yes Yes

Large scale Yes Yes Yes Yes

Cross-applicationoptimization for highutilization

Yes Yes Yes

Applications

Demand overmultiple resources Yes Yes Yes

Location constraints Yes Yes

Table 1: Comparison of features — WAND and other infrastructure

in WAND due to the limited capacity of MDCs that cannotaccommodate bursts.

Cluster schedulers: Cluster schedulers [32, 14, 27] are re-sponsible for allocation of server resources within hyper-scale DCs. They consider demands over multiple types ofresources and strive to drive the cluster to high utilization,often with prioritization across applications. They deal withlarge scale, e.g., allocating at the machine or rack granularityacross many racks. However, the resources are highly local-ized (typically within a single DC) in contrast with a geo-distributed WAND. Moreover, cluster schedulers are typi-cally decoupled from the network controllers which managesymmetric high-bisection bandwidth intra-DC interconnectbetween servers. This separation of control may not workwell in a WAND environment which has an irregular topol-ogy and expensive WAN links.

Traffic-aware VM placement in data centers: Traffic-aware VM placement is a related domain with joint serverand network resource management within a DC. However,this problem is NP-hard, while the WAND resource man-agement can be tackled using a Linear Program. In intra-DC environments, flexibility in VM placement translates toflexibility in the location of source and destination of theflow [24]. This optimization problem can be reduced to theQuadratic Assignment Problem (QAP) which is NP-hard. Inthe WAND model, the source and the destination of the floware fixed (typically at the end-user). We have flexibility inchoosing way-points, i.e., DCs. Thus, WAND resource man-agement is a resource-augmented Multi-Commodity Flow(MCF) problem with location restrictions. The challengesin WAND are related to scalability and quick adaptation.

Application-specific solutions: There exist several solu-tions tailored to specific geo-distributed applications.(a) NFV solutions: Elastic Edge [28] is an application-agnostic framework which allocates resources across NFVs.However, this solution is restricted to scheduling within asingle DC. PEPC [7] introduced a new architecture for the

cellular core to solve state duplication across EPC com-ponents and improved throughput at bottlenecked VNFs.KLEIN [30], another cellular-specific solution, put forwarda 3GPP-compliant solution for load-balancing cellular trafficacross MDCs. However, this does not take into account thepresence of multiple applications with diverse performanceneeds in the DCs, or the constraints on the interconnectingnetwork between DCs.(b) Geo-distributed analytics: Solutions for big data ana-lytics across the wide area [29, 35] offer optimization forlow-latency processing of queries, but do not deal with thedifficult problem of cross-application optimization.

WAND: Having established that existing resource man-agement techniques cannot be directly extended to genericWAND, we give a brief overview of current domain-specificsolutions offered by provider networks.(a) Carriers: AT&T [4] and Verizon [34] have publishedwhite papers on SDN/NFV reference architectures and arebuilding platforms with similar features such as network andDC controllers, data monitoring, a policy module, an orches-tration module, etc. In addition to cellular traffic, cellularMDCs also intend to support other micro-services and appli-cations such as connected cars [6]. These platforms, how-ever, are in the early stages of development and do not cur-rently offer solutions for resource management at the scaleof thousands of MDCs.(b) Content Providers: Points of Presence (PoPs) of largecontent providers like Google [38], Microsoft [21], andFacebook [31] also constitute a WAND. However, solu-tions in this domain primarily focus on load-balancing trafficacross edge locations. Currently, they do not offer computeresources for other applications at the edge.(c) CDNs: Content Delivery Networks cache objects at theedge to serve user requests with minimal latency. CDNs area sub-class of WAND where traffic is bandwidth-heavy, andedge resources are mostly used for storage and delivery. Likecontent providers, they do not typically support other userapplications. Application-specific video delivery optimiza-

3

��

��

��

��

��

��

��

��

��

��

��

(a) 15-min samples

��

��

��

��

��

��

��

��

��

��

��

(b) 60-min samples

Figure 1: Coefficient of Variation (CV) of traffic load based on areal-world DNS dataset evaluated at different scales — MDC-level,area-level, and the entire country.

tion schemes [12] have been proposed for this environment.

2.2 Features of WAND

Next, we crystallize the key characteristics of the WAND en-vironment. In WAND, (i) WAN and DC components canbe coordinated, (ii) MDCs are small to moderate in size butlarge in number, (iii) demand includes network and DC com-ponents, (iv) some traffic may be elastic, and (v) at individ-ual MDCs, small scale means limited statistical multiplexingand high demand variability.

A federated system: A WAND includes both a WAN andMDCs, whose operation may be coordinated in a single sys-tem or may be composed of federated subsystems.

Scale: WAND comprises a network of a large number ofdata centers — typically hundreds to a few thousands —which are individually small to moderate in size. CDNs havetens to hundreds of locations. Carrier networks operate thou-sands of DCs. There are efforts to install micro data centersat every base station, which will increase the scale to tens ofthousands [16].

Demands with multiple resource requirements: WANDapplications use WAN bandwidth as well as multiple re-sources within the DC, including compute, memory, andbandwidth, among others. While this is true of applicationsoutside of WAND as well, in a WAND these resources maybe coordinated in a single system. In addition, the particularmix of resources needed is likely to evolve with new appli-cations that employ MDCs: cellular control traffic adds loadon NFVs at DCs, uses limited WAN throughput but is latencysensitive; an IoT data-aggregation application may have highthroughput to the edge DC, high resource consumption in theDC and moderate throughput to remote hyperscale DCs. Ap-plications may also be categorized as single-user and multi-user applications. Cellular VNFs are shared across users ina region. On the other hand, an enterprise VPN connectionwill be composed of dedicated VMs and tunnels for a singleenterprise.

Elasticity of traffic: Depending on the types of applicationsrunning on WAND, some traffic may be elastic — a provider

can control the rate and time of resource allocation in DCsand WAN based on current resource availability. This can beused for low priority traffic such as backups.

Limited statistical multiplexing: Edge DCs do not form asingle large pool of compute. Each DC is small, and there-fore subject to more load variability due to less statisticalmultiplexing. They are distributed geographically, hencethey are non-interchangeable for latency-sensitive applica-tions. This may also lead to more variable utilization onindividual DCs during regional hotspots. Hence, efficientdynamic resource management is paramount.

We quantify the impact of limited statistical multiplex-ing in Figure 1. We use a real-world dataset from a DNSprovider comprising of DNS queries from across the USover a single day tagged with origin zip code and timestamp(more details about the dataset in § 6.1.1 b). For measuringvariability, we use the Coefficient of Variation, CV , definedas the ratio of standard deviation to the mean. CV is a com-parable measure for the extent of variability for distributionswith different means.

We divide the data with samples every minute into chunksof different lengths (15 min and 60 min) and compute CVover each chunk at different geographic scales — per edgelocation, per area1 and over the complete dataset (entirecountry). In Figure 1, we plot the CDF of CV over all chunksin one day across all edges/areas. We observe that CV is thehighest when locations are considered independently and thelowest in the complete dataset. When the timescale is in-creased from 15 min to 60 min, CV remains nearly the sameat larger scales while it increases more than 100× for DC-level traffic. This shows that edges are subjected to highervariability compared to traditional clouds and therefore, ben-efits accruable from statistical multiplexing are limited.

3 WAND API

Before delving into the design of a WAND API that helpsapplications specify their needs to the WAND controller, wediscuss common requirements of WAND applications.

3.1 Application requirementsGeo-distributed applications in WAND can be of two gen-eral types: (a) streaming with real-time traffic or (b) batchanalytics which processes offline data. Each has a variety ofperformance requirements; we describe examples in order tounderstand how to build an expressive WAND API.

Latency: When an application prefers an MDC at the edgeover a distant hyperscale cloud with cheaper resources, themain motivating factor is often latency. For user-facing ap-plications, the latency constraint is primarily related to end-

1More details about area boundaries in § 6.1.2. Load in an area is thesum of loads across all edge locations in that area

4

to-end latency experienced by users. Non-user facing ap-plications may have latency requirements between differentconcurrent modules.

Bandwidth: An application running in an MDC may havebandwidth requirements to users or between different inter-nal components.

Deadlines: Deadlines are primarily associated with batchprocessing jobs. They denote the final time before which theapplication expects an output. In applications with buffer-ing capability, this can also be the maximum time to fillthe buffer. For example, in video feeds to be processed theedge, the cameras may have limited storage capacity. In suchcases, when the analytics is not time-constrained, the videomay be pulled by the edge MDC with some time flexibility.

Intra-application dependencies: In traditional batch pro-cessing jobs, the application is represented using a DirectedAcyclic Graph (DAG) where the nodes are tasks and theedges denote dependencies. The same notation may beused in WAND. However, in distributed analytics, we mayhave additional constraints based on location of data andlatency/bandwidth constraints. Geo-distributed applicationsof streaming nature (NFV service chains, traditional streamprocessing, etc.) can also be represented using a graph. Thekey difference between streaming and batch jobs is that instreaming all components need to be active at the same time,whereas in batch jobs the tasks are executed sequentially inan order determined by dependencies.

In addition to task-level dependencies, WAND applica-tions may also have state-based dependencies. For exam-ple, cellular customers directed to their home MDC does notincur control overhead related with state transfer, whereasthose redirected to an MDC farther away will cause mobility-related overheads.

External dependencies: An application may have exter-nal dependencies on other jobs, type of resources, etc. Forexample, an analytics job on cellular traffic will prefer tobe colocated with the cellular service chain. In additionto such affinity constraints, an application may have anti-affinity constraints (two customers may prefer to not havetheir VPN provider edges colocated).

Evictions: One of the fundamental characteristics a WANDhas to deal with is limited statistical multiplexing due to thesmall size of MDCs. This manifests as high load variabilityand for medium- or low-priority applications, a risk of beingevicted by a burst of load on a high-priority application. Ap-plications may have different preferences regarding this risk,in at least two ways.

First, applications can differ in their tolerance for eviction,relative to the value they place on having resources close tousers. An opportunistic application may prefer to grab re-sources at the edge whenever possible, if evictions are not es-pecially costly (e.g., an opportunistic web cache at the edge

can serve requests at lower latency when nearby MDC re-sources are available). On the other hand, a more conser-vative application might prefer to run in an MDC only whenthe probability of eviction is low (e.g., a real-time applicationserving self-driving cars).

Second, an application may wish to lessen the damage dueto eviction by indicating which of its constituent modules areless critical. A video analytics application which processesCCTV feeds may have different levels of resource configura-tion [39] or a critical minimal set of cameras which providehigh coverage. Specifying such intra-application criticalitycan increase the controller’s scheduling flexibility and thusincrease the application’s chances of being scheduled.

Grouping requirements: We define a set of users whichshare the same characteristics, and therefore have a com-mon set of requirements (latency, bandwidth, etc.), as a usergroup. For example, in cellular environment, the set of usersassigned to a specific HSS may be one user group. Suchgrouping makes it more convenient for applications to spec-ify their needs.

3.2 Defining a WAND APIWe devise an API that allows geo-distributed WAND appli-cations to represent their diverse requirements. The key op-erations supported by the API are given in Table 2.

Both streaming and batch applications are represented us-ing a DAG. In streaming the dependencies denote the con-nections between tasks running concurrently. In batch appli-cations, the dependencies denote the sequence of execution.A tag may be associated with a node (task) or an edge (de-pendency) in the application graph. An application can usethese tags to denote its preferences such as bandwidth andlatency requirements. A tag may be of the type min, max orsum. When a tag specifies a min value, it is the minimum re-quirement on that attribute required by the application (e.g.,minimum bandwidth). When a tag is specified as sum, thelimit applies to sum of the attribute across all resources withthe same tag. For example, the end-to-end latency require-ment of a service chain can be specified using this type, withno constraints on individual dependencies.

The app id also acts as the tag for the entire application.Hence, to represent external dependencies, an applicationmay use the app id of the external application in the SetCon-straints() function. A specific task of an external applica-tion can be represented using the format app id:tag. In orderto represent location constraints, the infrastructure providercan provide location tags, either at per-MDC level or at aregional-level.

Our WAND API also supports an extendable list of appli-cation preferences. Current supported preferences are evic-tion tolerance, which specifies an approximate acceptableprobability of eviction of a resource allocated to the appli-cation; and optional modules, which specifies constituent

5

Operation Descriptioncreate () Returns app idsetType (app id, app type) app type ∈ {streaming, batch}setUserGroups (app id, list[ (user group idi, list[ <tagi >) ]] ) user group idi may be IP prefix, tags are optionalsetTasks (app id, list [ (task id j, list[ <tag j > ] ) ] ) each task may have one or more optional tagssetDependencies (app id, list[ (task idm, task idn , list[<tagmn > ]) ] ) each dependency may have one or more optional tags

setConstraints (app id, list[ (tagi, constraint name, constraint type,constraint value) ] )

constraint name ∈ { containers, latency, bandwidth,affinity, deadline}constraint type ∈ { min, max, sum}For latency, value is time in millisecondsFor bandwidth, value is bandwidth in MbpsFor affinity, value is 0 for anti-affinity, 1 for affinityFor deadline, value is UNIX epoch time

UpdateConstraints (app id,list[ (tagi, constraint name, constraint type, constraint change) ] )

constraint type ∈ { latency, bandwidth } andSigned integer indicating change in latency/bandwidth

setRedirectionOptions (app id, redirect) redirect ∈ {drop, edge, remote}

setApplicationPreferences (app id, preference, value)

Extendable list of preference attributes. Current set:preference ∈ {eviction tolerance, optional modules}For eviction tolerance, value ∈ (0,1]For optional modules: value is list[ (tag j, weight j)]

setMonitorInterval (app id, interval) monitoring interval in milliseconds

Table 2: WAND API fields

tasks within the application that it prefers to be evicted first,rather than evicting the entire application.

Currently, Patronus supports linear constraints only sincethe schedulers rely on specialized Linear Programs. How-ever, this is sufficient to handle the requirements of mostWAND applications we considered.

4 WAND Control Plane

We design an automated control system, Patronus, to achievehigh efficiency in the WAND environment. Patronus is acentralized controller managing resources across distributedMDCs and the interconnecting WAN. Patronus is designedfor a private WAND infrastructure, such as that operated bycellular providers, with a variety of streaming and batch ap-plications over which the infrastructure provider has somecontrol and visibility. We do not currently consider externalworkloads, and hence adversarial traffic. Automated con-trol in Patronus is realized with a sense-control-actuate loopcomposed of three main components: the prediction module,the resource allocation module, and the monitoring module.

4.1 Prediction Module

The prediction module is responsible for accurately esti-mating the requirements of applications. Depending on theinformation made available by applications, the predictionmay involve several phases. For all applications, the predic-tion module estimates the resource needs for future instancesbased on history. The prediction module is also responsible

for estimating user traffic patterns, such as diurnal behavior,across time to facilitate long-term resource planning. Forblackbox applications where the dependencies and require-ments are not explicitly known, the prediction module alsoestimates these dependencies between modules.

4.2 Resource Allocation ModuleThe resource allocation module is responsible for effi-ciently allocating resources across WAND applications, bothstreaming and batch jobs with a wide range of requirements.The scheduling involves two phases: long-term schedul-ing and instantaneous scheduling. The long-term schedulermaintains a long-term plan of the entire WAND system. It isresponsible for handling complex requirements such as dead-lines, location constraints, fairness, etc. At the beginning ofeach scheduling interval, the long-term scheduler shares thesubset of tasks and the predicted loads to the instantaneousscheduler for immediate scheduling. The unallocated tasksare returned to the long-term scheduler for future schedul-ing or offloading to remote data centers. This separation al-lows Patronus to have fast and simple instantaneous schedul-ing, while conforming to complex application requirementsacross longer timescales.

The pre-processing phase in resource allocation involvesthe conversion of application details obtained from the APIto a concise set of actionable requirements easily parsableby the schedulers. The min, max and sum values associ-ated with tags provide bounds on latency and bandwidth fortasks/dependencies. Latency bounds also determine a subsetof DCs where the task may be scheduled.

6

Before we delve into the schedulers, we discuss hierarchi-cal optimization and the conversion of application require-ments to constraints which form the core of both long-termand instantaneous schedulers.

4.2.1 Hierarchical optimization

Patronus employs incremental multi-objective optimizationfor scheduling diverse applications with a wide range of re-quirements. Unlike traditional Linear Programming (LP)with a single objective, this technique supports multipleobjectives within the same optimization. This allows theWAND controller to allocate resources of different typesacross diverse applications in a single optimization with pri-orities on applications/resources/locations determined by theoperator.

Objectives: In the multi-objective optimization, each objec-tive has four tunable parameters: priority (P), weight (W ),absolute tolerance (α) and relative tolerance (ρ). The LPsolver repeats the optimization n times in the order of pri-orities, where n is the number of priority classes. Whenmultiple objectives belong to the same priority class, theweight denotes the relative weight of each objective withinthe class. In this case, the aggregate objective of the priorityclass with multiple sub-objectives is given by the weightedsum. The absolute tolerance of an objective represents theabsolute value of degradation in the optimal value toleratedwhile optimizing lower priority classes. The relative toler-ance, ρ ∈ [0, inf], denotes the degradation tolerable as a mul-tiplicative factor of the optimal value.

A simple example with two objectives on DC utiliza-tion for a single application with a single task — nearestMDC assignment and load balancing at DCs is given be-low. The highest priority objective (higher P value) mini-mizes the fraction of traffic allocated to each DC weightedby the distance from each user-group to the MDC. This isthe nearest-MDC objective. Allowing 50% tolerance on thisoptimal value, the next round of optimization minimizes K,the difference in utilization between all pairs of MDCs. Inshort, this optimization supports nearest-MDC assignmentwith load-balancing across MDCs up to a certain limit of de-viation from nearest MDC distance. For this example, theLP is:

Obj0: P=2, W=1, α=0, ρ=0.5

Minimize ∑m∈M,g∈Ga

Da,gm ∗ f a,g

m (1)

Obj1: P=1, W=1, α=0, ρ=0

Minimize K (2)

Symbol DescriptionM The set of MDCsE The set of WAN linksCm Capacity of MDC m (#containers)Bl Capacity of link l (Gbps)

Nak

Number of containers consumed by taskk of app a per unit traffic

Ga The set of user-groups, g, of app a

Da,gm

Distance from user-group g of app a toMDC m

uag(t) Traffic at t for user-group g of app a

f a,gm (t)

Traffic from user-group g of app aassigned to MDC m at t

Table 3: ILP Notation

Subject to:

∀m ∈M : Na1 ∗ f a,g

m ≤Cm (3)∀m1,m2 ∈M : Na

1 ∗ f a,gm1−Na

1 ∗ f a,gm2

≤ K (4)

∀g ∈ Ga : ∑m∈M

f a,gm = ua

g (5)

Thus, hierarchical optimization allows combining mul-tiple objectives within and across applications in Patronuswith varying levels of tolerance determined by the operator.This technique forms the core of both long-term and instan-taneous schedulers.

4.2.2 Long-Term Scheduler

Long-term scheduling in WAND offers two benefits: (i)Analysis of historical traffic patterns provides the long-termscheduler with an approximate estimate of resource usagefar ahead in time. Most user-facing applications in WANDhave diurnal traffic patterns with predictable patterns acrossdays. (ii) For deadline-bound jobs, it allows planning re-source allocation ahead of time to meet the deadline. TheWAND environment may host a large number of opportunis-tic and flexible jobs which are not time-critical. Analyzingthe complete set of such jobs and their extent of flexibilityduring instantaneous resource optimization will increase thecomplexity of the scheduler considerably. In order to speedup instantaneous allocation and ensure fairness/cost-sharingacross time, scheduling of flexible jobs are handled throughlong-term resource planning.

In this phase, we also mitigate the impact of limited statis-tical multiplexing at individual MDCs. While each individ-ual MDC in WAND suffers from high variability due to itssmall size, a pool of MDCs in a region enjoys less variabilitywhen aggregated resources are considered (as discussed in§ 2.2 Figure 1). In addition to identifying the most suitableMDC for a task, the long-term planner also identifies areasfor backup scheduling. This loose allocation of tasks to ofa group of MDCs during the long-term planning phase also

7

helps avoid the need for a global search for an alternative al-location during instantaneous scheduling when the primarychoice MDC is unavailable. Thus, long-term planning re-duces the number of active low-priority tasks and their pos-sible locations to be considered during a given instant whileensuring that the deadlines and other constraints are met inlonger timescales.

The long-term scheduler can operate in two modes: (i)fast mode, (ii) replan mode. When a new job arrives atthe long-term scheduler, first, the fast mode is initiated.The aggregate load at all DCs for all previously allocatedjobs is used to determine the available capacity and an at-tempt is made to place the job. If its requirements (la-tency/bandwidth/deadline etc.) are met with allocation atpreferred MDCs, the allocation is finalized. If the fast modefails, a replan is invoked over applications at or below thepriority class of the new job. In this phase, the resourcesare shared fairly across multiple jobs belonging to the sameclass. If the new job is not completely accommodated at theMDCs, parts of it may be scheduled on a regional or remoteclouds in this phase. The long-term scheduler may also in-voke replan of the schedule if long-term resource predictionschange significantly relative to prior estimates.

The long-term scheduler is also responsible for provid-ing inputs to instantaneous scheduling at each schedulinginterval. It provides a limited set of feasible locations foreach task based on traffic estimates. It also includes a setof optional tasks to ensure work-conservation during under-estimation of traffic.

4.2.3 Instantaneous Scheduler

The instantaneous resource allocation module is responsiblefor per-instance scheduling of application components deter-mined to be scheduled by the long-term scheduler. This in-cludes (a) a subset of batch tasks, (b) loads on active stream-ing applications, and (c) optional tasks which are not criticalin the current instance but may be scheduled if resources areavailable. These application components may belong to dif-ferent priority classes and may have different objectives. Forhighest priority applications guaranteed to be accommodatedby an MDC (such as cellular traffic which is guaranteed tohave a utilization below a low threshold, say 50%), optimiza-tion is not invoked unless there is a failure in the system.

The instantaneous scheduler also relies on hierarchical op-timization, but typically it has fewer objectives than the long-term planning mode. This is possible because deadlines, fair-ness, and the complete set of preferences are not handledby the instantaneous scheduler (long-term scheduler sharesa critical subset). Even within the same class, the long-termplanner will include relative priorities that determine whichtask is to be eliminated if sufficient capacity is not available.Such tasks are determined by long-term fairness estimates.

4.2.4 Other Optimization Features

We discuss two features in the context of WAND optimiza-tion: eviction tolerance of WAND applications and geo-distributed fairness objectives employed by WAND provider.

Eviction Tolerance: Lower priority applications in WANDwill be evicted when load of applications in higher prior-ity classes spikes high enough that the MDC becomes over-loaded. Therefore, being allocated resources in MDCs nearusers comes with some risk. We allow applications to con-trol this tradeoff through the eviction tolerance parameter inthe WAND API. During resource allocation, this high-levelapplication requirement is incorporated in the optimizationbased on statistical analysis.

The goal of the eviction tolerance, θ , is to represent themaximum eviction probability per unit-time tolerable by theapplication (θ ∈ [0,1]). However, evictions are not perfectlypredictable. When considering scheduling an application inan MDC, Patronus attempts to predict eviction probabilitybased on three factors: (i) mean load (M) of applicationswith priority ≥ c, (ii) variability of their load (V ), and (iii)resource requirements (R) of app a. Mean and variability arecomputed over a period of length equal to the length of thetask being scheduled. i.e., if a task is 5 min long, the meanand variability of historical data from past 5 min is estimated.Variability is measured as Coefficient of Variation (§ 2.2).

The application with demand R is evicted when the higherpriority load exceeds C−R where C is the capacity of theMDC. Assuming that the variability of this load follows anormal distribution (with mean M and coefficient of varia-tion V ), the probability of eviction at an instant is P(L >C−R) = 1−P(L < C−R) = 1−Φ(C−R−M

MV ), where Φ isthe cumulative probability of a standard normal distribution,X ∼ N(0,1), Φ(x) = P(X < x).

Note that all the terms in the probability expression areconstants that can be determined apriori based on historicaldata. An MDC is considered a candidate for placement onlyif this estimated probability of eviction for the app is smallerthan the application’s eviction tolerance (θ ). This is an ap-proximate heuristic since the distribution of load may not beperfectly normal. However, the application can still adjustits eviction probability by increasing or decreasing θ .

Fairness: The second optimization feature is the fairnessobjective for geo-distributed applications. The distributednature of the applications and the underlying infrastructurecalls for new notion of fairness. Defining fairness in WANDcan be challenging for two main reasons: (i) WAND is amulti-resource environment with requirements on multipleresources, and (ii) in with most applications composed ofgeo-distributed components, it is difficult to define a domainof fairness. Does each application get a fair-share at eachMDC? Can underallocation for an app in one MDC be com-pensated by overallocation at another location?

Patronus employs a simple notion of fairness. Since

8

DCs are the bottleneck in current cellular WANDs, we usefair-share allocation across DC resources for applications.However, instead of fair-sharing at each individual DC, thefairness domain is areas (collective group of neighboringMDCs). The choice of area as the domain for determin-ing fairness is justified by the behavior of applications inWAND and the structure of the underlying topology. Stream-ing applications such as cellular VNFs are capable of load-balancing across adjacent DCs. When a single edge is over-loaded, the application can obtain its fair-share through aneighbor.

4.3 Monitoring Module

Monitoring of applications serves two purposes: (i) en-able quick reaction to changes in application behavior and(ii) provide input for prediction and long-term planning.The monitor interval can be explicitly set by an applicationthrough the WAND API or determined by the provider dur-ing runtime based on the extent of variability. We developa simple tunable monitoring module in Patronus which canrun alongside applications in WAND. We assume that thismonitored information is made available centrally to the re-source allocation module. The prediction module that relieson the output of monitoring may be co-located in the sameMDC or centralized.

5 Implementation

We implement Patronus as a stand-alone control system withpluggable modules for each component. The current imple-mentation of various modules is described below.

Resource allocation module: Both the instantaneous andlong-term schedulers are Java applications with an integratedGurobi [13] environment for solving the Linear Programs as-sociated with resource allocation. Both schedulers take as in-put application requirements as JSON files containing task-level/user-group level requirements and dependencies.

The long-term scheduler holds a plan which contains esti-mated allocations for all applications across time. We storea plan for 1 day in 1 minute time-slots. The instantaneousprediction runs every minute. These parameters are tunable.When a new application arrives in the system, it is initiallyplaced in this plan by the long-term scheduler. At each allo-cation interval, the long-term scheduler sends the list of cur-rently active tasks to the instantaneous scheduler for actualplacement. The schedulers convert requirements to linearobjectives and constraints in a hierarchical linear programand solve the multi-objective LP using Gurobi. The out-put from the instantaneous scheduler is the set of allocatedtasks, their location and the amount of resources allocated.Currently, the schedulers consider resources at the scale ofVMs/containers within data centers.

��

��

��

��

��

��

��

��

��

��

��

(a) Cellular dataset

0

0.2

0.4

0.6

0.8

1

0 5 10 15 20

Norm

aliz

ed L

oad

Time (hours)

New York Chicago

(b) DNS dataset

Figure 2: Sample traffic

��

��

��

��

��

��

��

��

��

��


��

��

��

��

��

��

��

��

��

��

��

(b) DNS dataset

Figure 3: Traffic Characteristics

��

��

��

��

��

��

��

��

��


��

��

��

��

��

��

��

��

��

(b) Simulation Environment

Figure 4: DC Sizes

Prediction module: The prediction modules takes as inputtimeseries data on traffic volume/resource usage characteris-tics and predicts the resources required for the next instance.Currently, the prediction module can operate in two ways: (i)learning using a Neural Network (NN) or (ii) statistical esti-mates as mean, variance, weighted average etc. Dependingon the characteristics of the application, one of these tech-niques is used. A 4-layer NN with 16, 8, 4, and 1 nodes ateach of the layers respectively with full connection betweenthe layers is used. Other cloud-based solutions [33, 2] can beeasily integrated with this module.

Monitoring module: We implement the monitoring moduleas a lightweight Linux application which will run on the ap-plication VM to record its resource and traffic patterns. Themonitoring interval is a tunable parameter. Note that the Pa-tronus evaluation is based on simulation that does not usereal-time monitoring.

6 Experiments

In this section, we evaluate scalability and efficiency of thePatronus system.

9

6.1 Methodology

The first step of evaluation is the development of a realis-tic test environment. We use real-world traffic datasets andpublicly available infrastructure information to generate a re-alistic WAND topology.

6.1.1 Workloads

We integrate several real-world workloads to generate thetopology and the traffic in WAND.

(a) Cellular dataset: The cellular dataset is obtained froma large cellular provider in China and consists of utilizationinformation at 50 virtualized DCs over a period of 5 days.At each DC, it provides information on (a) the mean and themaximum CPU utilization per server, and (b) total data sentand received from the DC. The utilization values are moni-tored every 10s by the cellular provider. The dataset containsthe mean and the maximum at 15 min intervals over this data.The utilization at two locations are plotted in Figure 2 (a). InFigure 3 (a), we plot the CDF of mean and maximum CPUutilization across all 50 locations on all days. We observethat the utilization is approximately between 10% and 30%.The distribution of normalized DC sizes (# servers) is plottedin Figure 4 (a). The number of servers range from < 10 tohundreds. The largest DC has more than 1000 servers.

(b) DNS dataset: This dataset contains DNS queries re-ceived by a single DNS provider at locations across the globeover a period of 24 hours (508GB of compressed data). En-tries include the timestamp (at microsecond granularity) andthe origin zipcode of each DNS request. From this globaldataset, we filter requests originating in the continental US,which include 17,440 origin zipcodes with more than 20 bil-lion queries in the 24-hour period. The number of queriesare aggregated at minute-scale and load at two zipcodes areplotted in Figure 2 (b). The CDF distributions of minimumand mean with respect to the maximum load are shown inFigure 3 (b). We observe that the DNS queries have highervariability compared with the cellular dataset. The maximumload at a DC can be more than 10× the mean in certain cases.

(c) Video camera dataset: Video processing at the edge isa most prominent application that will benefit from WANDdeployments. The number of security cameras in the US in2016 is estimated to be 62 million [22]. We use the loca-tions of red-light and speeding cameras in the US availablein a public GPS forum. This dataset includes 4932 camerasacross the continental US. While these cameras take snap-shot pictures during traffic violations, we use this as a realis-tic proxy for video camera locations.

(d) Big data analytics: We use publicly available TPC-DSmicrobenchmarks [26] with synthetic location constraints tosimulate batch analytics workloads in WAND. We use tracesof 16 TPC-DS queries originally run on 20 machines.

6.1.2 Generating the Test Environment

We run the Patronus controller on a Dell PowerEdge R440machine with 32 cores and 128GB RAM.

(a) Topology: We generate a realistic WAND topologybased on publicly available information on cellular infras-tructure. The locations of MDCs are determined using Cen-tral Office (CO) information of a large cellular provider inthe US [1]. We obtain 1864 Central Office locations associ-ated with the 17440 zip codes in the DNS dataset.

Each MDC is assigned a capacity which is proportional tothe population associated with that zip code (2016 popula-tion [37]). The distribution of data center sizes is shown inFigure 4 (b). We note the MDC sizes based on population inthe US and the real size distribution in China (Figure 4 (a))follow similar distributions. The number of servers in the USsimulation environment ranges from 2 to 2200. In line withregional offices of cellular providers, we also add 15 regionalDCs (publicly available locations of colocation centers of thesame cellular provider in the US) with capacity proportionalto the sum of MDCs that are closest to it.

The core topology of the same cellular provider is ob-tained from the Intertubes dataset [10] and has 116 nodesand 151 links. Each DC is connected to its closest corenode. Conversations with cellular operator reveal that cur-rently cellular MDCs follow a hub-and-spoke-like modelwith MDCs connecting to the core. With 5G, MDCs areexpected to have more interconnections among them. Cur-rently, we do not consider such interconnections in our testenvironment. The bandwidth of the outgoing link of an MDCis set proportional to its capacity.

(b) Workload: The cellular data for the test environmentis obtained by applying the percentage of utilization from arandomly chosen DC in the Chinese dataset to the DC. Thesemapping are done a priori to obtain the timeseries for a day.VPN/other high-priority applications of cellular provider isgenerated synthetically. We generate 100 such applications,each with a random number of sites between 3 and 15 with arandomly chosen load at each location. These are streamingapplications where all sites are concurrently active.

The scaling factors for datasets are determined propor-tional to their loads in such a way that the load (as number ofcontainers) is neither too low nor too high at most DCs. Forthe DNS dataset, the load in containers is proportional to thenumber of requests. For the camera dataset, the number ofcontainers is proportional to the cameras assigned to a DC.

6.2 ResultsWe evaluate the ability of Patronus controller to utilize theresources efficiently and meet the application requirements.

Efficiency of Hierarchical optimizationVPN and other applications which belong to the highest pri-ority class are slightly more flexible than the cellular traffic.

10

��

��

��

��

��

��

��

��

��

��

��

Figure 5: Comparison of Patronus opti-mization, random placement, and nearest-DC placement on VPN (Highest priority).

��

��

��

��

��

Figure 6: Eviction Tolerance vs. Latency(latency based on geodesic distance)

��

��

��

��

��

��

��

��

��

��

Figure 7: Geodesic distance-based la-tency for distributed video processingbased on DC placement (3rd priority class)

��

��

��

��

��

��

��

��

��

��

Figure 8: DNS dataset Prediction Error.With regional balancing of load, impact oferror mitigated.

��

��

��

��

��

��

��

��

Figure 9: Scheduling delay as a functionof number of variables.

��

��

��

��

��

��

��

��

��

��

Figure 10: Coefficient of Variation ofpenalty (resources scheduled in cloud dueto lack of edge resource)

We assume that the cellular traffic need to be processed at thenearest MDC (when resources are available) while the VPNallocation may be at any MDC within 1ms from the originof the traffic. Hence, to determine long-term placement fornon-cellular high priority applications, we run a two-levelhierarchical optimization at the peak period of cellular load.

The optimization has 2 objectives. The highest priorityobjective is shortest path allocation of cellular traffic. InFigure 5, we observe that shortest path allocation on cellu-lar traffic load results in a maximum edge DC utilization of50%. For the VPN traffic, we evaluate several techniques.Minimizing the maximum load of DCs with latency-basedplacement constraints in Patronus can accommodate all thenon-cellular applications while maintaining the maximumutilization at 53.6%. On the other hand, placing the VPNload at the closest MDC to the origin or at a randomly cho-sen MDC within the prescribed radius (1ms) leads to highutilization in some DCs. Note that jointly optimizing the cel-lular and VPN loads may result in some cellular traffic beingredirected to non-shortest DCs. Hence, multi-level optimiza-tion may be essential even within the same priority class.

Eviction Tolerance/Latency Trade-offThe trade-off between eviction tolerance (4.2.4) and latencyat a single MDC in third priority class is given in Figure 6(first priority is cellular and VPN applications, second prior-ity is DNS workload). We consider the video analytics appli-cation load in third priority class at a single MDC. Evictiontolerance is computed based on the mean and standard devia-tion of total traffic in the higher priority classes for an interval

of 15min. The figure shows mean latency over a period of 19hours at each data point. When the eviction tolerance is thehighest at 1.0, the latency is 0.15ms. As the eviction toler-ance decreases, the latency of the application increases. Ateviction tolerance value of 0.7, the latency increases 10× to1.5ms. When the application is very conservative (θ < 0.1),the latency increases to approximately 3ms.

Latency of applicationsWe measure the latency perceived by the video analyticsdataset in priority class 3 (Figure 7). We compare Patronusoptimization with nearest-MDC placement. In both scenar-ios, when an MDC is not available, the task is placed on thenearest regional DC. We observe that the tail latency is morethan 2× lower in Patronus. Note that the latency is computedbased on geodesic distance here. With additional infrastruc-ture overhead, the differences will be more pronounced sincePatronus places tasks at the edge at a higher rate.

Prediction EfficiencyWe evaluate the predictability of the two real-world datasets.In the sparse cellular dataset, we use statistical measures. Weconsider two scenarios: (a) past hour on the same day, or (b)Time of Day (ToD) in past n days. When the prediction isdefined as the maximum observed load, ToD of 4 days isa better predictor. Over a day, prediction based on recenthistory can lead to under-prediction up to 6.5% and over-allocation up to 8% while ToD prediction limits the errors to3.1% and 6% respectively. Hence, we use ToD demand of 4days with a 5% over-allocation to meet the demand.

11

��

��

��

��

��

��

��

��

��

��

(a) CPU

��

��

��

��

��

��

(b) Disk

Figure 11: Performance monitoring overhead at various monitor-ing intervals

In the DNS dataset, the error in prediction using NN isgiven in Figure 8. In the dataset with samples every minute,data from past 10 min is used to predict the expected loadas number of DNS requests in the current instance. The er-ror is high due to high variability in the workload (shown inFigure 2). However, if the application can effectively load-balance across MDCs in an area, the error reduces consider-ably. This shows that applications with high load variabilitycan benefit significantly from load-balancing at the edge.

ScalabilityWe test the scalability of both hierarchical and long-termschedulers in Patronus resource allocation module. Sinceboth the schedulers rely on hierarchical optimization, we plotthe scalability in terms of number of variables (Figure 9).The scheduling latency was measured with 4 optimizationobjective priority levels in the optimizer.

The number of variables depends on a variety of factors:number of MDCs and links, number of applications, numberof tasks in batch jobs, number of user-groups in streamingjobs, number of timeslots, etc. Even with millions of vari-ables, the scheduling latency is in seconds. Note that thenumber of variables in hierarchical optimization is 2-3 ordersof magnitude smaller in instantaneous scheduler comparedto the long-term scheduler. For example, if we consider 1000DCs and 100 applications with a components in every DC,we have approximately 0.1 million variables. In hierarchicaloptimization, not all of them may be active at the same time,reducing the complexity even further. In this range, the opti-mization takes seconds. The corresponding long-term sched-uler with a one-day plan has 0.1 million ∗1440 variables.

FairnessWe compare the fairness achieved with Patronus fairnessscheme compared with a scheme where no fairness is en-forced (arbitrary sharing of resources across applications inthe same class determined by the LP solver). We assume thatfor batch applications, when the edge resources are unavail-able, they can run in the cloud, but this is expensive. Wedefine the penalty as the amount of resources allocated oncloud when edge resources are not available. In Figure 10,we observe that the coefficient of variation of penalty acrossapplications (a heuristic for unfairness) is reduced with fair-sharing in Patronus.

Performance monitoringWe measure the overhead associated with monitoring using alightweight Linux module which measures applications’ us-age of CPU, memory, network, and disk. The CPU and diskutilization of the monitoring module at various monitoringintervals are given in Figure 11(a) and 11(b) respectively.The overhead is negligible (< 0.5% in CPU and 9Kbps inIO) when monitoring interval is 1s.

7 Discussion

Patronus is a first step towards scalable WAND resourcemanagement. Several challenges remain open.

Offering WAND services: Patronus is designed for a pri-vate WAND environment. If a WAND provider chooses tooffer resources to external applications, similar to the cloud,we need a novel service characterization. MDCs have differ-ent compute capabilities and latency profiles between them.The cost and performance using the same amount of resourcevary widely depending on the size and location of the DCsand within the same DC across time due to limited statisticalmultiplexing.

More stringent application constraints: Patronus can re-spond to application changes at the scale of seconds. Tohandle emerging applications that require millisecond-levelapplication response times and for improving resource uti-lization when fast VM/container resource scaling is feasible,Patronus may be extended with edge controllers that supple-ment the centralized controller.

Handling the data deluge: Monitoring and profiling ofthousands of geo-distributed applications generate signifi-cant amount of data. Patronus assumed that this data is avail-able to the centralized controller. In practice, we need effi-cient techniques to collect and monitor such data. The tem-poral and spatial dimension of generated data in WAND callsfor novel database techniques for efficient storage and queryresponse-times.

8 Conclusion

In this paper, we identify control challenges in an emergingdynamic environment of interconnected Micro Data Centerswhich we refer to as WAND (WAN As A Network of DataCenters). We build Patronus, an automated control system,for efficient resource management in WAND. We address thescalability challenge by partitioning the scheduling probleminto two temporal levels: instantaneous scheduling handlingimmediate allocation and long-term scheduling for meetingcritical application requirements across time. We show thatPatronus is scalable, resource-efficient with balanced loadacross MDCs and capable of meeting stringent applicationrequirements.

12

References[1] Telecommunications Database. https://www.telcodata.us, 2018.

[2] ALIPOURFARD, O., LIU, H. H., CHEN, J., VENKATARAMAN, S.,YU, M., AND ZHANG, M. Cherrypick: Adaptively unearthing thebest cloud configurations for big data analytics. In 14th USENIX Sym-posium on Networked Systems Design and Implementation (NSDI 17)(Boston, MA, 2017), USENIX Association, pp. 469–482.

[3] APPLEGATE, D., AND COHEN, E. Making Intra-domain RoutingRobust to Changing and Uncertain Traffic Demands: Understand-ing Fundamental Tradeoffs. In Proceedings of the 2003 Conferenceon Applications, Technologies, Architectures, and Protocols for Com-puter Communications (New York, NY, USA, 2003), SIGCOMM ’03,ACM, pp. 313–324.

[4] AT&T. ECOMP (Enhanced Control, Orchestration, Management &Policy) Architecture White Paper. https://goo.gl/KUmNq2, 2016.

[5] AT&T. The Cloud Comes to You: ATT to Power Self-Driving Cars,AR/VR and Other Future 5G Applications Through Edge Computing.http://bit.ly/attEdge, 2017.

[6] AT&T. ATT Connected Car. http://bit.ly/attConn, 2018.

[7] AYYUB QAZI, Z., WALLS, M., PANDA, A., SEKAR, V., RAT-NASAMY, S., AND SHENKER, S. A High Performance Packet Corefor Next Generation Cellular Networks. In Proceedings of the 2017ACM SIGCOMM Conference (2017).

[8] CORD. Central Office Re-architected as a Datacenter (CORD).https://goo.gl/qzzChH, 2016.

[9] DIXIT, A. A., HAO, F., MUKHERJEE, S., LAKSHMAN, T., ANDKOMPELLA, R. Elasticon: An elastic distributed sdn controller.In Proceedings of the Tenth ACM/IEEE Symposium on Architecturesfor Networking and Communications Systems (New York, NY, USA,2014), ANCS ’14, ACM, pp. 17–28.

[10] DURAIRAJAN, R., BARFORD, P., SOMMERS, J., AND WILLINGER,W. intertubes: A study of the us long-haul fiber-optic infrastructure.

[11] ELWALID, A., JIN, C., LOW, S., AND WIDJAJA, I. Mate: Mplsadaptive traffic engineering. In Proceedings IEEE INFOCOM 2001.Conference on Computer Communications. Twentieth Annual JointConference of the IEEE Computer and Communications Society (Cat.No.01CH37213) (2001), vol. 3, pp. 1300–1309 vol.3.

[12] GANJAM, A., SIDDIQUI, F., ZHAN, J., LIU, X., STOICA, I., JIANG,J., SEKAR, V., AND ZHANG, H. C3: Internet-scale control plane forvideo quality optimization. In 12th USENIX Symposium on NetworkedSystems Design and Implementation (NSDI 15) (Oakland, CA, 2015),USENIX Association, pp. 131–144.

[13] GUROBI OPTIMIZATION, I. Gurobi optimizer reference manual,2016.

[14] HINDMAN, B., KONWINSKI, A., ZAHARIA, M., GHODSI, A.,JOSEPH, A. D., KATZ, R., SHENKER, S., AND STOICA, I. Mesos:A platform for fine-grained resource sharing in the data center. In Pro-ceedings of the 8th USENIX Conference on Networked Systems Designand Implementation (Berkeley, CA, USA, 2011), NSDI’11, USENIXAssociation, pp. 295–308.

[15] HONG, C.-Y., KANDULA, S., MAHAJAN, R., ZHANG, M., GILL,V., NANDURI, M., AND WATTENHOFER, R. Achieving High Uti-lization with Software-driven WAN. In Proceedings of the ACM SIG-COMM 2013 Conference on SIGCOMM (New York, NY, USA, 2013),SIGCOMM ’13, ACM, pp. 15–26.

[16] IO, V. Vapor IO Chamber. https://www.vapor.io/chamber/, 2015.

[17] JAIN, S., KUMAR, A., MANDAL, S., ONG, J., POUTIEVSKI, L.,SINGH, A., VENKATA, S., WANDERER, J., ZHOU, J., ZHU, M.,ZOLLA, J., HOLZLE, U., STUART, S., AND VAHDAT, A. B4: Expe-rience with a Globally-deployed Software Defined WAN. In Proceed-ings of the ACM SIGCOMM 2013 Conference on SIGCOMM (NewYork, NY, USA, 2013), SIGCOMM ’13, ACM, pp. 3–14.

[18] KANDULA, S., KATABI, D., DAVIE, B., AND CHARNY, A. Walkingthe Tightrope: Responsive Yet Stable Traffic Engineering. In Proceed-ings of the 2005 Conference on Applications, Technologies, Architec-tures, and Protocols for Computer Communications (New York, NY,USA, 2005), SIGCOMM ’05, ACM, pp. 253–264.

[19] LEVIN, D., WUNDSAM, A., HELLER, B., HANDIGOL, N., ANDFELDMANN, A. Logically centralized?: State distribution trade-offsin software defined networks. In Proceedings of the First Workshopon Hot Topics in Software Defined Networks (New York, NY, USA,2012), HotSDN ’12, ACM, pp. 1–6.

[20] LI, Y., HARMS, J., AND HOLTE, R. A simple method for balancingnetwork utilization and quality of routing. In Proceedings. 14th In-ternational Conference on Computer Communications and Networks,2005. ICCCN 2005. (Oct 2005), pp. 71–76.

[21] LIU, H. H., VISWANATHAN, R., CALDER, M., AKELLA, A., MA-HAJAN, R., PADHYE, J., AND ZHANG, M. Efficiently deliveringonline services over integrated infrastructure. In 13th USENIX Sym-posium on Networked Systems Design and Implementation (NSDI 16)(Santa Clara, CA, 2016), USENIX Association, pp. 77–90.

[22] MARKIT, I. North American security camera installed base to reach62 million in 2016. http://bit.ly/cctvInUS, 2016.

[23] MCCAULEY, J., LIU, Z., PANDA, A., KOPONEN, T., RAGHAVAN,B., REXFORD, J., AND SHENKER, S. Recursive sdn for carrier net-works. SIGCOMM Comput. Commun. Rev. 46, 4 (Dec. 2016), 1–7.

[24] MENG, X., PAPPAS, V., AND ZHANG, L. Improving the scalabilityof data center networks with traffic-aware virtual machine placement.In 2010 Proceedings IEEE INFOCOM (March 2010), pp. 1–9.

[25] MICHAEL, N., AND TANG, A. HALO: Hop-by-Hop Adaptive Link-State Optimal Routing. IEEE/ACM Transactions on Networking 23, 6(Dec 2015), 1862–1875.

[26] OUSTERHOUT, K., RASTI, R., RATNASAMY, S., SHENKER, S.,AND CHUN, B.-G. Making sense of performance in data analyticsframeworks. In 12th USENIX Symposium on Networked Systems De-sign and Implementation (NSDI 15) (Oakland, CA, 2015), USENIXAssociation, pp. 293–307.

[27] OUSTERHOUT, K., WENDELL, P., ZAHARIA, M., AND STOICA,I. Sparrow: Distributed, low latency scheduling. In Proceedings ofthe Twenty-Fourth ACM Symposium on Operating Systems Principles(New York, NY, USA, 2013), SOSP ’13, ACM, pp. 69–84.

[28] PALKAR, S., LAN, C., HAN, S., JANG, K., PANDA, A., RAT-NASAMY, S., RIZZO, L., AND SHENKER, S. E2: A framework fornfv applications. In Proceedings of the 25th Symposium on OperatingSystems Principles (New York, NY, USA, 2015), SOSP ’15, ACM,pp. 121–136.

[29] PU, Q., ANANTHANARAYANAN, G., BODIK, P., KANDULA, S.,AKELLA, A., BAHL, P., AND STOICA, I. Low latency geo-distributed data analytics. In Proceedings of the 2015 ACM Confer-ence on Special Interest Group on Data Communication (New York,NY, USA, 2015), SIGCOMM ’15, ACM, pp. 421–434.

[30] QAZI, Z. A., PENUMARTHI, P. K., SEKAR, V., GOPALAKRISHNAN,V., JOSHI, K., AND DAS, S. R. KLEIN: A Minimally DisruptiveDesign for an Elastic Cellular Core. In Proceedings of the Symposiumon SDN Research (New York, NY, USA, 2016), SOSR ’16, ACM,pp. 2:1–2:12.

[31] SCHLINKER, B., KIM, H., CUI, T., KATZ-BASSETT, E., MAD-HYASTHA, H. V., CUNHA, I., QUINN, J., HASAN, S., LAPUKHOV,P., AND ZENG, H. Engineering egress with edge fabric: Steeringoceans of content to the world. In Proceedings of the Conference ofthe ACM Special Interest Group on Data Communication (New York,NY, USA, 2017), SIGCOMM ’17, ACM, pp. 418–431.

[32] VAVILAPALLI, V. K., MURTHY, A. C., DOUGLAS, C., AGARWAL,S., KONAR, M., EVANS, R., GRAVES, T., LOWE, J., SHAH, H.,SETH, S., SAHA, B., CURINO, C., O’MALLEY, O., RADIA, S.,

13

REED, B., AND BALDESCHWIELER, E. Apache hadoop yarn: Yetanother resource negotiator. In Proceedings of the 4th Annual Sympo-sium on Cloud Computing (New York, NY, USA, 2013), SOCC ’13,ACM, pp. 5:1–5:16.

[33] VENKATARAMAN, S., YANG, Z., FRANKLIN, M., RECHT, B., ANDSTOICA, I. Ernest: Efficient performance prediction for large-scaleadvanced analytics. In 13th USENIX Symposium on Networked Sys-tems Design and Implementation (NSDI 16) (Santa Clara, CA, 2016),USENIX Association, pp. 363–378.

[34] VERIZON. SDN-NFV Reference Architecture, Verizon Network In-frastructure Planning. https://goo.gl/YCeDBr, 2015.

[35] VULIMIRI, A., CURINO, C., GODFREY, P. B., JUNGBLUT, T.,KARANASOS, K., PADHYE, J., AND VARGHESE, G. Wanalytics:Geo-distributed analytics for a data intensive world. In Proceedingsof the 2015 ACM SIGMOD International Conference on Managementof Data (New York, NY, USA, 2015), SIGMOD ’15, ACM, pp. 1087–1092.

[36] WANG, H., XIE, H., QIU, L., YANG, Y. R., ZHANG, Y., ANDGREENBERG, A. COPE: Traffic Engineering in Dynamic Networks.In Proceedings of the 2006 Conference on Applications, Technologies,Architectures, and Protocols for Computer Communications (NewYork, NY, USA, 2006), SIGCOMM ’06, ACM, pp. 99–110.

[37] WORLD, D. U.S. Population by zip code, 2010-2016.http://bit.ly/usPop, 2016.

[38] YAP, K.-K., MOTIWALA, M., RAHE, J., PADGETT, S., HOLLIMAN,M., BALDUS, G., HINES, M., KIM, T., NARAYANAN, A., JAIN, A.,LIN, V., RICE, C., ROGAN, B., SINGH, A., TANAKA, B., VERMA,M., SOOD, P., TARIQ, M., TIERNEY, M., TRUMIC, D., VALAN-CIUS, V., YING, C., KALLAHALLA, M., KOLEY, B., AND VAH-DAT, A. Taking the edge off with espresso: Scale, reliability and pro-grammability for global internet peering. In Proceedings of the Con-ference of the ACM Special Interest Group on Data Communication(New York, NY, USA, 2017), SIGCOMM ’17, ACM, pp. 432–445.

[39] ZHANG, H., ANANTHANARAYANAN, G., BODIK, P., PHILIPOSE,M., BAHL, P., AND FREEDMAN, M. J. Live video analytics at scalewith approximation and delay-tolerance. In 14th USENIX Sympo-sium on Networked Systems Design and Implementation (NSDI 17)(Boston, MA, 2017), USENIX Association, pp. 377–392.

14

patronus: controlling thousands of geo-distributed micro...

Documents