How to Increase the Value of Volatile Cloud Resources:Resource Management and Information Disclosure

Chaojie ZhangDept. of Computer Science

University of Chicago

chaojie@uchicago.edu

Varun GuptaDept. of Computer Science

University of Chicago

guptav@uchicago.edu

Andrew A. ChienDept. of Computer Science

University of Chicago

Argonne National Lab

achien@cs.uchicago.edu

AbstractCloud providers sell unreliable or “volatile” resourcesthat are unused by foreground (reserved/high priority)workloads. The value users can extract from these re-sources depends on (i) the volatile resource manage-ment algorithm, and (ii) the information provided to usersabout the volatile resources. We describe and evalu-ate four volatile resource management approaches (Ran-dom, FIFO, LIFO, LIFO-pools) using commercial cloudresource traces drawn from 608 Amazon EC2 instancepools. We also consider several information models(MTTR, limited statistics, Full distribution, and Oracle)that statistically characterize the resources for users.

Our results show volatile resource management algo-rithms can increase user value by 30 to 45%. Slightlyricher information models (90pctile) combined with LIFOand LIFO-pools volatile resource management increaseuser value by as much as 10x. Our results suggest thatcloud providers should pay significant attention to whatstatistical information they provide to users. And, theseresults broadly characterize the vast majority (475 of 608)of instance pools. Finally, we provide a detailed drill-down showing how the volatile resource management al-gorithms affect resource interval durations, and thus po-tential user value. We further show how the informationmodel shapes user targeting, success rate, and user value.

1 IntroductionModern cloud datacenters are continually expandingtheir computing resources to meet growing needs for e-commerce, web search, social networking, and big dataanalytics. IT leaders have rapidly adopted public cloudresources that match traditional reliable resource modelssuch as on-demand and reserved instances from Ama-zon’s Elastic Compute Cloud and Google Compute En-gine [1, 5]. As the cloud market grows, these opera-tors continue to build capacity to meet unpredictable load

spikes, peak, and growing demand. Accurate forecastingis difficult, producing fluctuating quantities of excess re-sources as in Figure 1.

Figure 1: Cloud operators serve a varied foreground load,producing a variable “excess” resources.

To increase resource utilization, cloud providers sell ex-cess resources to increase revenue. For example, all cloudproviders practice overprovisioning, the overcommittal ofresources to increase revenue, and if done well it has neg-ligible customer performance impact. Reserved instancesguarantee a customer access to a resource, but those re-sources are frequently idle. Excess resources are oftensold as unreliable cloud instances. These resources varyin name (spot instances, Preemptible virtual machines),but share the property that they can be revoked as illus-trated in Figure 2. This produces a more complex volatileresource usage interface. Requests for volatile resourcescan be delayed or fail. Even after a successful request andongoing use, a volatile resource can be revoked, causingloss of some application work. Because of this potentialrevocation, we call these resources “volatile”.

The ability to reclaim volatile resources enables cloudproviders to meet ramps in foreground (high priority)load. The design of volatile resources varies acrosscloud providers. For example, prices differ – Google’sPreemptible VM’s have a fixed discounted price[7];Amazon EC2’s Spot Instances use an auction biddingmechanism[3]. However, all volatile resources exhibit re-

1

Figure 2: High-level view of how Users and Cloudproviders interact on Volatile and Reliable Resourcesvocation, so users (applications) must monitor resources,respond to revocations, and ensure application progress.

Volatile resources have been available at scale since2009 [3], so many studies explore efficient applicationuse. For example, a number of efforts focus on biddingstrategies in Spot Instance markets [20, 21, 12, 30] orpredicting price dynamics [14, 31]. These bidding strate-gies attempt to keep instances running, but because theyfail, other systems employ migration[16, 28] and check-pointing [29, 13, 15, 17, 28] to save application work,and enable application resumption after revocation. Someefforts attempt statistical characterization and prediction[25, 26], and another proposed the idea of using resourcemanagement to shape volatile resource properties, provid-ing MTTR as a statistical characterization to increase uservalue [19]. Our work builds on these ideas.

Volatile resource properties are not natural; they areproduced by interaction of foreground load and volatileresource management algorithms. Yet, the temporal, sta-tistical properties of the volatile resources affect theirvalue to users. We call the information that a cloudprovider reveals about the volatile resources an “Informa-tion Model”. Thus, a cloud provider faces two key ques-tions:

1. What volatile resource management algorithm max-imizes the value of my excess resources?

2. How does the information model for a volatile re-source pool affect resource value for users?

In this paper, we formulate and explore the questionof volatile resource management that creates volatile re-source pools. We study several volatile resource manage-ment (VRM) algorithms, evaluating their impact on in-terval distributions, variability, and value. We also varythe information model, exploring the tradeoff between en-abling accurate user targeting and the desire to shield pro-prietary data center specs. Because cloud providers donot release resource management data, we derive resourceavailability profiles from a large collection of 608 Spot In-stance price traces, drawn from the full breadth of Ama-zon’s EC2.

Our results show that volatile resource management hasdramatic effect on the statistical properties of volatile re-

sources, and that good choices can significantly increaseuser value. Our studies also show the choice of infor-mation model is critical for effective volatile resource ex-ploitation.

Specific contributions include:1. Formulation of the volatile resource management

and information model design problem.2. Using four exemplar instance pools, design and eval-

uation of four volatile resource management algo-rithms (Random, FIFO, LIFO, and LIFO-pools), in-cluding impact on resource availability intervals andvariability. The best-performance VRM algorithm,LIFO-pools, can increase user value by 30-80%(MTTR info model), and up to 2-10x across infomodels and exemplars.

3. With four exemplars, design and evaluation of infor-mation models, MTTR and 10pctile produce pooruser value because of the skew in resource inter-val distributions, but 90pctile approaches the valueof providing the Full distribution, outperforming theothers.

4. A drill-down on interval statistics, showing VRM re-source properties, and how information models in-form user targeting and improve success rates forhigher user-value.

5. Study of 608 instance pools, Amazon EC2’s full USbreadth, characterizing of periodicity and availabil-ity. Full analysis of these pools with the four VRMsand info models, showing that our results for the twoStable exemplars are representative of 80% of the in-stance pools.

The rest of the paper is organized as follows. Key back-ground is covered in Section 2. Section 3 describes ourvolatile resource management and information model ap-proaches. Methods are covered in Section 4. Section 5evaluates these approaches using commercial cloud tracedata, followed by a drill-down on a number of key ques-tions in Section 6. In Section 7 we discuss related work,and then summarize results and future research directionsin Section 8.

2 Background

2.1 Volatile Resources in CommercialClouds

Cloud operators have introduced volatile resource ser-vices to utilize and garner revenue from idle resources.However, the way they are offered differs greatly. Table1 summarizes several major volatile resource services onthe market. Amazon’s Spot Instances are offered and re-voked based on a market mechanism where users placebids, and resources are allocated to users in descending

2

Figure 3: A 3-month trace of price history and derived resource availability profile.

bid price order. Given the quantity of available volatileresources, eventually they are exhausted, and the lowestbid receiving a resource defines the market clearing price.In other words, all bids greater than or equal to the mar-ket clearing price receive resources [3]. All active SpotInstances pay the market clearing price for a given hour.Significant academic research has explored the Spot mar-kets [8, 14]. Amazon has disclosed no further detail. Weuse the 90 days of clearing price history that Amazon pro-vides to inform user bidding.

Volatile Resource Service Duration Info CommentAmazon

Spot Instance None Revoke on price riseDefined Duration Select duration Pay fixed priceInstance (≥ 1,≤ 6hrs) per hour

GooglePreemptible VM ≤ 24hrs No guarantee

Table 1: Cloud provider volatile resource servicesGoogle offers Preemptible Virtual Machines with a

much simpler pricing scheme. Preemptible VM’s havefixed prices, set at 20% of on-demand price for each vir-tual machine type; no information is provided about howrevocation decisions are made, and Preemptible VM’s re-ceive a notification 30 seconds before termination [6].

2.2 Traditional Resource Management

Traditional resource managers or resource managementalgorithms are important in any large scale computing in-frastructure. Such resource managers typically match re-source requests (with partial to nearly complete informa-tion about requirements) to a fixed set of resources thatare owned by the resource manager. To do so, resourcemanagers track resource status, load, and sometimes fail-ure state, and employ a variety of information (lookaheadinto a queue of requests, priority, wait time, prediction, re-quest characteristics) and sophisticated algorithms (spacefitting, back-filling, online, simulation, etc.) to schedulethe resource requests with multi-dimensional attributes

(compute, memory, storage, parallelism, etc.) onto datacenter resources.

The problem we consider here differs in that volatileresources vary in quantity rapidly due to external forces(the fluctuations of the foreground workload demand). Avolatile resource manager cannot control these, but rathermust respond to them by adapting the outstanding set ofresources granted to users. In many cases, this adaptationrequires resource revocation. Such revocation is a rare,rather than common feature of traditional resource man-agement systems. The lack of awareness or control overforeground load makes the task of volatile resource man-agement significantly different from traditional resourcemanagement.

2.3 Resource Availability ProfilesVolatile resource availability is defined by datacenter ca-pacity and foreground resource load. Both of these areconsidered to be highly proprietary trade secrets by allmajor cloud providers. Thus only limited summary infor-mation is available [24, 11]; insufficient to do the broadstudies considered here. On the other hand, plentiful pric-ing information is available for Amazon Spot Instancemarkets. We exploit this pricing data to infer the resourceavailability traces.

We use 90-day traces of Amazon EC2 Spot marketsfrom 5/5/2017 to 8/3/2017 that include market clearingprices for each 5-minute interval for 608 instance types.Each instance type is an independent resource type withan arbitrary amount of resources associated to it, fromfour US Regions. For each instance type, this comprised25,920 price samples, or 15.8 million prices overall. Weused the pricing data to infer an availability profile basedon a simple linear price-supply relationship, assuming themarket is always cleared. At the maximum clearing pricein the trace, the volatile resources are assumed to be zero;at the minimum price the volatile resources are assumedto be maximum (arbitrarily set at 5000 units, commensu-rate with a total data center capacity of 10000 units). 1

1This choice affects the absolute value of available resources, but

3

Hence, the resource availability at time t is:

VR(t) = (max price− price(t)) ∗ Max volatile resourcesmax price − min price

(1)

For example, in Figure 3, traces in red show Spotprices for instance type r3.4xlarge,i3.4xlarge,r3.xlarge,and r4.16xlarge for a 3 month period for one availabilityzone of region us-west-2. The traces in blue show the in-ferred resource availability (shown in percentage of totalcapacity) updated every 5 minutes at a datacenter where50% of the resources are always claimed by foregroundload (on-demand and reserved instances).

3 ApproachWe describe our approach to studying the two key ques-tions posed earlier: the volatile resource managementproblem, that is, what VRM algorithm maximizes thevalue of the excess resources?; and the information modelproblem, that is, what information is needed by usersto accurately target volatile resources and extract mostvalue? We begin by decomposing the complexities ofthe volatile resource management problem, including thekey requirements and challenges, and then describe thealgorithms for volatile resource management and modelsfor information sharing evaluated in this paper.

Figure 4: Volatile resource managers (VRMs) make deci-sions to allocate and revoke resources, producing volatileresource intervals of varying lengths.

3.1 Volatile Resource ManagementAvailable volatile resources are determined by the fluctua-tion of foreground workload (i.e. on-demand and reservedinstances) (see Figure 4), and the task of the volatileresource manager (VRM) algorithm is to grant and re-voke the available volatile resources. When a resourceis needed for foreground load, the VRM must revoke avolatile resource, causing a volatile interval to end. When

does not have any impact on the metrics evaluated in the paper, as allof our results are comparative across approaches; each with the samequantity of available volatile resources.

a resource is no longer needed by the foreground load, theVRM starts a new interval. Different VRMs can producedifferent interval distributions via their choice of whichvolatile resource to revoke first. For example, considertwo resources m,n that are added to the volatile resourcepool at times t and t + i, respectively. Subsequently, attimes t + i + j and t + i + k, there are new foregroundload resource requests. By choosing which resource tobe revoked first, the volatile resource manager can createtwo intervals of durations {i + j, k} or {j, i + k}. Thus,choice of volatile resource management produces differ-ent volatile resource properties.

We consider four different volatile resource manage-ment algorithms:

Volatile Resource Management Algorithms

Random Revoke a randomly chosen resource from theactive volatile resources.

FIFOAll volatile resources form a single pool.Revoke the volatile resource allocated first intime (FIFO, or oldest-first)

LIFOAll volatile resources form a single pool.Revoke the volatile resource allocated mostrecently (LIFO, or last-first)

LIFO-PoolsSeparate volatile resources into five pools.Management within each pools is LIFO, andmanagement across the pools is LIFO.

Of these algorithms, LIFO-Pools bears some explana-tions. A VRM can create multiple resource pools, de-signed to create more attractive interval properties forusers. In this case, as necessary, the VRM selects a pooland a resource within it to revoke. LIFO-Pools createsa stack of N pools, with the topmost having the short-est intervals and lowest resource availability. We use N=5pools, as our experiments with more pools provided negli-gible further change. Further down the stack, both of theseproperties improve. The LIFO-pools VRM seeks to createnarrower distributions of interval duration, enabling bettertargeting.

Data center evolution activities such as large-scale up-grades, system migrations, and system reconfigurations,require basic resource flexibility, so we cap maximum in-terval length at 48 hours for all our VRM algorithms. Ad-ditional key assumptions and requirements for the volatileresource management include:

1. Cloud neutrality: All volatile resource customersshould be treated equally. That is, there should beno prejudice based on whether job came from Mor-ganStanley or GoldmanSachs or internal cloud prop-erties (analogous to net neutrality).

2. Foreground load is a quantity requirement: A fore-ground load resource request only specifies an in-stance type (cpu speed, cores, memory, local IO), nota particular physical machine.

4

Figure 5: Information models can provide users with sta-tistical characterization of a volatile resource pool.

3.2 Information Models

Information about the volatile resource interval durationsaffects users’ ability to capture value from them. The in-terval lengths for volatile resources in a pool can be char-acterized statistically, so the users maximize value sta-tistically; for example, enabling users to select job run-times to maximize the expected value. Current cloudoperators disclose essentially no statistical informationabout their volatile resources2. However, cloud opera-tors could disclose a statistical characterization of eachvolatile resource pool, as shown in Figure 5. While onemight propose full transparency as a good policy for cloudproviders, they are reluctant to reveal such information,viewing their internal resource pool sizes and resourceloading as a critical competitive secret.

Users exploit the available information to target jobruntimes, tune checkpointing strategy, etc. If the distri-bution of durations is highly skewed, simple informationmodels may lead users to make poor decisions.

We consider several different information models, andevaluate how they affect the ability of users to derive valuefrom resources.

Information Models

MTTR Basic statistics, MTTR (mean-time torevocation) of the resource pool.

10pctile Median and the 10th percentile (duration forwhich 10% of the intervals are shorter)

90pctile Median and the 90th percentile (duration forwhich 90% of the interval are shorter)

Full Full histogram, equivalent to the probabilitydistribution function of interval durations.

Oracle Exact duration length for each interval, inadvance (unrealizable).

2See Section 7 for a discussion of defined duration instances.

3.3 Example VRM: Simple AvailabilityProfile

To build intuition, we consider a simple resource avail-ability profile (see Figure 6a) that includes a periodic dailyramp that might result from diurnal usage. Resources areclaimed and released at a constant rate until the maximumor minimum levels of volatile resources is reached. Themagnitude of variation is one-half the maximum volatileresource level. On reaching the minimum volatile re-source level (2,500 in Figure 6a), the system remains therefor a period of time before resuming its variation.

For this availability profile, FIFO volatile resourcemanagement would produce a narrow distribution withall intervals of length close to 1,800 minutes. Other re-source management algorithms can produce interval dis-tributions that vary in number of intervals, mean intervallength, variance in interval length, and the overall distri-bution (see Figure 6b). In short, the volatile resource man-agement algorithm can shape the interval duration distri-bution, and thus is a critical element in creating value fromvolatile resources.

(a) Simple availability with pe-riodic (daily) variation

(b) Interval statistics, Volatileresource management ap-proaches

Figure 6: Simple Model

4 MethodologyIn this section we outline key elements of our evalua-tion methodology: user value functions (Section 4.1), howusers optimize them for each information model (Sec-tion 4.2), and the metrics we study to compare the VRMand information models (Section 4.3).

4.1 User Value FunctionThe value function for a user’s job maps the actual intervalduration assigned to the job to a value. We use a stepfunction that depends on the target duration of the job asdefined below:

f(L) =

{v0(target) L ≥ target ,

0 otherwise,(2)

5

(a) Step value function (b) Comparing growth rates forStep value function

Figure 7: Value Functions

where target represents the target duration, andv0(target) the maximum achievable value (see Fig-ure 7a). That is, the job delivers no value until the run-time L exceeds a target duration, and execution beyondthe target accrues no further value. The step function isa good model for batch and workflow computations. Asis evident, accurate targeting is critical for applications tomaximize value.

We consider two scalings of the maximum achiev-able value v0(target): linear (target) and polynomial(target1.5) as shown in Figure 7b. Linear scaling mod-els computations whose value grows in direct proportionto the runtime. Polynomial scaling models computationsthat benefit from longer runtimes enabling solution ofhigher value, larger problem sizes, as well as effects suchas overhead amortization, etc. For experiments, we nor-malize the v0(target) value so that they are equal for thelinear and target1.5 scalings for target = 1 hour.

4.2 User Targeting Strategies

Given a value function, users can select target duration tomaximize value based on statistical information availableabout volatile resources. We assume users employ the fol-lowing targetting strategies for each information model:

1. MTTR: target = 0.9× (Mean interval length)Users normally checkpoint or run workloads thatare slightly shorter than the given MTTR; Assum-ing standard deviation is 10% of MTTR and Gaus-sian distribution, targeting 0.9×MTTR would givea success rate around 67% .

2. 10pctile: Select target to maximize expected valueassuming Gaussian distribution with given 10th per-centile and median

3. 90pctile: Select target to maximize expected valueassuming Gaussian distribution with given 90th per-centile and median

4. Full: Select target to maximize expected valuegiven the Full distribution of interval durations

5. Oracle: Users are told the exact interval duration andtarget it to extract maximum value (unrealizable)

Experiments assume that demand is sufficient to utilizeall volatile resources – same assumption as the AWS mar-ket clearing [3].

4.3 MetricsWe use several performance metrics in our study:

Volatile Resource Availability Profile: Number of in-stances available at each 5-minute interval. Induced byforeground load; the input to the volatile resource man-ager.Total Available Volatile Resources (resource-hours):the total available instance hours over the entire periodof study, used to compare different instance pools.Interval Statistics: the VRM induced interval statisticsfor an instance pool – includes mean (MTTR), %-tiles,distributions, standard deviation.Target (minutes): Target duration selected by usergiven a resource pool information model.Success rate: Fraction of computations achieving non-zero value.Total Value: Sum of the computation values over inter-vals of an instance pool.

5 EvaluationWe employ a trace-driven method to evaluate volatile re-source management algorithms (VRM), exploring theirimpact on interval statistics such as mean duration(MTTR) and standard deviation. We then compareachievable user value for each VRM, varying the infor-mation models and value function scaling.

Evaluation is based on industrial cloud price tracestaken from Amazon’s EC2 for a 3-month period [4, 26].We studied volatile reource management on 608 instancetypes, drawn from Amazon’s four US regions, each ofwhich contains 2-6 availability zones. Comprehensive re-sults for these resource pools are too large to be presented,so we identified four exemplars based on two statisticaldimensions: volatility (standard deviation of the resourceavailability) and periodicity (ratio of fourier coefficientsfor a period of 1 day vs. average)3 that capture importantcharacteristics commonly found in cloud resources, suchas diurnal periodicity. Scatterplots showing the distribu-tion of instance pools along these two dimensions for thefour US regions are shown in Appendix A.

Distribution of the 608 instance pools show two keysimilar characteristics: 1) a wide spread of volatility, butwith many instance pools with low volatility, and 2) var-ied magnitudes of diurnal load variation. To reflect thisdistribution, we selected four exemplars of instance pools

3Computed by taking the fast-fourier transform (FFT) of the resourceavailability profile, and then the ratio of the frequency bands

6

Figure 8: Scatter-plot of Instance pool pricing behavior.

PriceInstance Dynamic Mean Std.

Pool Range Dev.Stable 1(r3.4xlarge) 13.798 0.378 0.674Stable 2(i3.4xlarge) 0.436 0.162 0.038Periodic(r3.xlarge) 0.043 0.049 0.005

Unstable(r4.16xlarge) 42.135 12.802 17.836

Table 2: Key statistics for Instance Pools Exemplars.

- Stable 1, Stable 2, Periodic(diurnal), and Unstable - fordetailed study from the us-west-2 region, availability zonea. The scatterplot for us-west-2a is shown in Figure 8, andthe selected exemplars are highlighted in red. The twoStable instances reflect the predominance of low volatil-ity pools, and the Periodic and Unstable instances capturethe range in those respective dimensions. We return to thelarge set of instance pools in Section 5.6, showing howthe two Stable exemplars are representative of 80% of theAmazon EC2 instance pools.

Key price statistics for each exemplar are shown in Ta-ble 2. Price data for each exemplar is processed (as inSection 2.3) creating a resource availability profile (seeFigure 3). Statistics for resource availability, and the num-ber of intervals created by each VRM are shown in Table3.

5.1 Volatile Resource Management and In-terval Properties

The VRM algorithm creates availablity intervals in avolatile resource pool. In Figure 9, we present the re-sulting interval duration statistics for the four exemplarpools, applying the four VRMs. The basic statistics in-clude mean, median, and standard deviation. The x-axisshows three VRM’s (Random, FIFO, LIFO) on the left,and LIFO-pools on the right with five lighter-colored bars

InstancePool

TotalVolatileHours

VRMAlgo-rithm

# ofIntervals

Stable 1(r3.4xlarge) 1.05× 107

RandomFIFOLIFO

366,720347,529414,713

Stable 2(i3.4xlarge) 9.74× 106

RandomFIFOLIFO

387,225360,889451,394

Periodic(r3.xlarge) 6.38× 106

RandomFIFOLIFO

416,876390,007485,364

Unstable(r4.16xlarge) 7.46× 106

RandomFIFOLIFO

723,939722,159732,563

Table 3: Total Resource and Intervals Statistics (Exem-plars).

– one for each sub-pool within LIFO-pools. In all casesthe whiskers depict standard deviation. The VRM canmake a large difference in all of these statistics, and theLIFO-pools’ partition into 5 sub-pools can create differ-entiated statistics for each sub-pool.

On the left side of Figure 9 is the Stable 1 resource pool.Stable 1 (r3.4xlarge) has small price fluctuations muchof the time, and a handful of big price spikes in the 3-month period. Three VRMs (Random, FIFO, LIFO) pro-duce similar mean and median interval durations. Theyalso produce standard deviations close to their mean, sothe variability of intervals durations is high. The stan-dard deviation of LIFO is 1.3x greater than FIFO. LIFOcreates both more short intervals and long intervals whileFIFO creates more medium-length intervals. LIFO-poolsdifferentiates pools. Pool 0 has a much smaller mean, andPools 1-4 have much higher means and medians than bothPool 0 and the basic VRMs. All 5 sub-pools are signifi-cantly less variable, exhibiting lower standard deviation.

Stable 2 (i3.4xlarge) is similar to Stable 1 but has rel-atively more and stronger price fluctuations. Again, thethree VRMs (Random, FIFO, LIFO) produce compara-ble means, but the medians tell a different story withLIFO producing dramatically more short intervals, andthereby a much smaller median (3x smaller than FIFO).Of course LIFO must produce a number of correspond-ing longer intervals to achieve a mean close LIFO. Thethree basic VRM algorithms produce standard deviationsclose to their means, so interval variability is very high.Consistent with Stable 1, LIFO-pools successfully sepa-rates resources into differentiated pools with Pools 0 and1 exhibiting low means and medians and Pools 2-4 muchhigher and low variability means and medians comparedto both Pools 0 and 1 as well as the basic VRMs.

Third subplot in Figure 9 is Periodic (r3.xlarge, a

7

Figure 9: Basic Interval statistics, 4 Instance Pool Exemplars (Stable 1, Stable 2, Periodic, and Unstable) and 4 VRMs

diurnal load), exhibiting periodic resource availabilitychanges. These patterns produce shorter mean and me-dian interval lengths (∼1000 minutes or approximately16 hours). As before, LIFO produces a mean similar toRandom and FIFO, but much small median (10x smaller),and comparable standard deviation. Again, this effect isdue to LIFO producing many more very short and verylong intervals. LIFO-pools again successfully creates dif-ferentiated resource pools. Pools 0, 1, and 2 with lowermean and median interval lengths. Pools 3 and 4 havemuch higher means and medians than the basic VRMs(Random, FIFO, LIFO). Combined with a lower standarddeviation, this makes Pools 3 and 4 much better resourcepools. Note that this is not quite as good as Stable 1 where4 pools were superior.

Finally the rightmost graph in Figure 9 presents inter-val statistics for Unstable (r4.16xlarge). Unstable hasfrequent, extreme price spikes, and many moderate pricefluctuations. With such frequent spikes, all four VRM’sproduce similar characteristics, comparable means andmedians that are consistently lower than other exemplars.LIFO-pools fails to create distinct resource pools.

Thus, we see that VRMs shape the distribution of inter-val durations, and LIFO-pools can sometimes create dif-ferentiated pools. In the next section, we assess the impactof these interval duration properties on the ability of usersto extract value.

5.2 Volatile Resource Management Algo-rithms and User Value

We compare our four volatile resource management algo-rithms (Random, FIFO, LIFO, and LIFO-pools), using themetric of derived user value for the Step value function asdefined in Section 4.1. We simplify this initial compari-son, providing only basic instance pool information to theuser (MTTR or mean interval duration), and comparing toan ideal (“Oracle”), where the user is given precise dura-tion information for each interval, as it begins, enablingperfect targeting. “Oracle” level performance is of courseunachievable.

In Figure 10, we present the achieved user value. Along

Figure 10: Comparing four VRMs, MTTR info model.the x-axis, the major steps are the four exemplar instancepools with each of the VRMs clustered for convenientcomparison. The y-axis is the achieved user value for theentire 3-month period. For LIFO-pools we present theaggegated value from the five sub-pools.

Beginning with the Stable 1 and Stable 2 exemplarpools we see that Random, FIFO, and LIFO all achievecomparable value, but LIFO-pools gives a 30% improve-ment. With the Periodic and Unstable exemplars, overallachieved value is much lower, reflecting the difficulty ofexploiting these resources. For Periodic, FIFO and LIFO-pools give best value. For both Periodic and Unstable allof the four VRMs achieve comparable user value. For allexemplars, the best value is well short of “Oracle”, show-ing the impact of statistical unpredictability in resources.

A deeper look at the interval statistics for each of theVRMs and instance pools in Figure 9 can explain someof the differences in value. LIFO-pools separates smallintervals and long intervals in different subpools, so themean/MTTR captures the pool properties better, enablingmore accurate targeting. We will explore this deeper incoming sections, considering different information mod-els.

5.3 Information Models and User ValueBecause volatile resources vary statistically, informationabout interval duration distribution is critical to maxi-mize derived value. As mentioned in Section 4.2, weassume users do optimized targeting based on the infor-

8

Figure 11: User value vs. Information model, Linear scaling of Step value

Figure 12: User Value vs. Information model, Polynomial scaling(d1.50 ) of Step value

mation available to maximize value. Our four informa-tion models each reveal a different amount of information,and would require a cloud operator to choose to releasesuch resource characterization information (perhaps leak-ing proprietary, competitive information). To inform suchchoices, we compare four information models (MTTR,10th percentile, 90th percentile, Full distribution, and Or-acle), combined with our volatile resource managementalgorithms.

In Figure 11, along the x-axis, we show four VRMs,and for each in a cluster, we vary the information modelfor easy comparison. Consider Stable 1, leftmost in Fig-ure 11: the clear trend with all four VRMs is that richerinformation models increase value. Both Random andFIFO yield modest benefits as the information availableincreases, peaking at 0.55-.67 of the ideal potential value(Oracle). Because FIFO has a low standard deviation, thedifferent information models provide similar real informa-tion, producing comparable value. LIFO is quite different,increasing nearly 2x from the MTTR information modelto Full, and achieving nearly 0.90 of the Oracle value.LIFO-pools is close to LIFO, starting higher, and reach-ing a similar peak of 0.90 of the Oracle. Stable 2, sec-ond from left in Figure 11 shows a more complicated, butsimilar story. Random and FIFO see small benefits fromincreased information, but LIFO and LIFO-pools see sig-nificant benefits and peak at 0.85 of ideal potential value.

For Periodic (second from right in Figure 11), Ran-dom and FIFO have similar properties, benefiting littlefrom increased information. LIFO improves the deliveredvalue as we vary information model, but shows that mis-

leading information (10th percentile for a heavily skeweddistribution) causes poor targeting, sharply reducing de-rived value. LIFO-pools delivers significant improve-ment, peaking at nearly 0.90 of ideal potential value, andavoids the misdirection for targeting. It is worth notingthat the total volatile resources in Periodic is lower than inStable 1 and Stable 2 (see in Table 3), so the lower idealpotential value is in line with expectation. For Unstable,shown in at far right, all four resource management al-gorithms do well with a Full information model, peakingat 0.67 of ideal with LIFO and LIFO-pools slightly bet-ter. They all do progressively better with 90th percentilemodels, showing around 0.45 of the Oracle value.

Looking across the full range of exemplars, it is clearthat the information model makes a dramatic difference inuser value, providing a 30% increase in numerous cases,and more than 10-fold in extreme cases. The Full dis-tribution information model achieves the best user value(within realizable information models), as much as 1.5xbetter than the next best information model. FIFO andLIFO-pool are insensitive to information models; on theother hand, LIFO’s value depends strongly on the infor-mation model. Information about the 90th percentile isrobust across both Stable and Periodic instance pools, be-cause LIFO produces distributions that are skewed long,making it an attractive tradeoff of limited information andgood value. To achieve value as close to Oracle, a goodcombination of VRM and information model is needed.

In Figure 12 we present user value results for polyno-mial scaling of the step value function as described in Sec-tion 4.1. Several differences from linear scaling in Figure

9

(a) VRM / Random (b) Info Models / MTTRFigure 13: Relative Value increases for VRM and InfoModel

11 are worth noting. First, because we scaled the linearand target1.5 growth of step value to match at 1 hour, thetotal achievable value for cases shown in Figure 12 arearbitrarily much higher and should not be compared di-rectly with those in Figure 11. Second, under polynomialscaling, the user value achieved under the “Oracle” in-formation model differs across volatile resource manage-ment algorithms within each exemplar. This is becausethe LIFO algorithms create more long intervals, and as thestep value increases superlinearly, this causes an increasein the total potential value for the LIFO algorithms. Third,polynomial scaling does not change the qualitative resultscomparing information models or volatile resource man-agement algorithms. In all cases, the 90th percentile andFull information models give best user value and achieve50% to as much as 90% of Oracle value. Comparingthe volatile resource management algorithms, LIFO andLIFO-pools are significantly better for the stable and peri-odic exemplars with LIFO-pools doing particularly well.For the unstable exemplar, results are similar to the linearscaling.

5.4 VRM and Information Model Benefits

Consider VRM algorithms normalized to Random, fo-cused on instance pool Stable 1 (see Figure 13a). GoodVRM can increase value beyond Random significantly,by up to 1.7x and LIFO-pools consistently achieves thehighest value (1.5x increase), but LIFO is good as well.Consider information models normalized to MTTR, alsofocused on instance pool Stable 1 (see Figure 13b). Allof the information models increase value, including thesimple information models (MTTR, 10pctile, 90pctile).Combining a good VRM such as LIFO or LIFO-poolsyields 1.2x to 1.45x value increase with simple infor-mation models. Further while Full consistently achieveshighest value, it may not be desirable for cloud providers.After Full, the best by a significant margin is 90pctile (upto 45% value increase).

5.5 Overall Four Exemplar Results

Our study of four exemplar instance pools demonstratesthat both volatile resource management algorithm and in-formation model can make a large difference in achieveduser value. LIFO and FIFO vs Random can make morethan 2x difference, and LIFO-pools provides the highestperformance broadly up to 2-10x better compared to LIFOor FIFO. The information model is also critical suggest-ing that cloud providers should consider providing statis-tical characterization of their volatile resources (Amazon,Google, and Microsoft currently do not). Even simplestatistics, such as 90th percentile, can increase achiev-able value by 10% to as much as 5x, and the Full dis-tribution can increase it by as much as 10x. Design of aninformation model that maximizes value whilst protectingcloud provider’s internal proprietary secrets is an interest-ing topic for future research.

5.6 Relating Exemplars and All InstancePools

We have done full studies of interval statistics, VRM, andinformation models on 608 Amazon EC2 instance pools.We believe these results show that the majority of thepools are substantially similar to our two stable exem-plars. To demonstrate, in Figure 14 we present the meanand median results across the 475 instance pools closestto Stable 1 and Stable 2 in the normalized volatility, peri-odicity space.4 These results, normalized to Oracle, showthat many of these pools are similar to the two Stable ex-emplars. For example, for mean user-value, the results arethe same ordering for VRMs and information model as inStable 1 and Stable 2, and likewise, but with smaller gaps,for median. Standard deviations across the collection ofinstance pools are small, showing the significance of thedifferences.

Considering ordinal statistics between different VRMand information model configurations can also show howsimilar the pools are to Stable. The frequency of variousrelations that were our key conclusions for the Stable ex-emplars within the 475 instance pools are shown below.

4Of the 608 pools, we removed 5 that had no price changes, and thetwo Stable exemplars, so 475 is approximately 80% of the remainingpools.

10

Figure 14: Mean, Median, and Standard deviation performance for 475 instance pools closest to Stable 1 and Stable 2exemplars

Configuration Relationship Count PercentageLIFO-Pools+Full Top 1 468 98.5%Full ExcludedLIFO-Pools + 90pctile Top 2 365 77%LIFO + 90pctile Top 2 388 82%LIFO-Pools + 90pctile within 10% best 399 84%LIFO + 90pctile within 10% best 403 85%FIFO + 10pctile within 10% best 200 42%Random + 10pctile within 10% best 223 47%FIFO + MTTR within 10% best 15 3.2%Random + MTTR within 10% best 14 2.9%

Within 10% of best is defined as greater than or equalto (90% * best value achieved). These results supportthat LIFO-Pools, Full is best overall in nearly all instancepools. If we exclude the Full information model as in-feasible, LIFO-Pools and LIFO do best with the simpleinformation models (90pctile best, 10pctile and MTTR)in more than 80% of these instance pools. The 10pctileand MTTR information models don’t achieve Top 2 per-formance in the majority of the instance pools. Randomdoes poor uniformly. These ordinal results match our de-tailed analysis of Stable 1 and Stable 2 well.

6 Drilling Deeper

6.1 Information Models and Optimal Tar-geting

We have seen that choice of VRM algorithm and infor-mation model has direct impact on realizable value. Toexplore why, we consider how these choices affect opti-mal target duration and success rate. In Figure 15, alongthe x-axis, the major steps are the four information modelswith the four VRM algorithms in each cluster in differentcolors. For LIFO-pools, one target is shown for each sub-pool for a total of 5 target durations.

Leftmost in Figure 15 is the Stable 1 exemplarr3.4xlarge. Here LIFO-pools’ advantage in separatingshort and long intervals is clear, enabling better target-ing. All other three VRMs have comparable target length,

but LIFO pools 1-4 have up to 1.6X longer targets, evenunder weaker information models. A similar story playsout for Stable 2 exemplar i3.4xlarge (2nd from left), withLIFO-pools targeting quickly (and accurately) growing to1.75x with even small amounts of information.

The same basic trend occurs for the Periodic exemplar(Figure 15, 2nd from right), but only LIFO-pools 2,3,4have long targets, whilst targets for other volatile resourcemanagement algoritms are relatively low. The benefits ofLIFO-pool, separating short and long intervals, and de-scribing them well are clear. However, Unstable (Figure15, right) is a different story. The VRMs and the informa-tion models fail to produce significant differences in targetinterval length. Even LIFO-pools cannot aid targeting.

6.2 Information Models and Success Rate

The complement of targeting is user computation successrate. Maximizing user value balances the value derived onsuccessful completion (a function of target) against suc-cess rate. Thus, maximizing success rate does not nec-essarily maximize user value. With Stable 1 (see Figure16, left), Random, LIFO, and FIFO never achieve highsuccess rates, but the better targeting enabled by LIFO-pools achieves nearly 2x higher success rate. The reasonfor this is large standard deviations of the interval dura-tion (recall Figure 9). For Stable 2 (see Figure 16, 2ndfrom left), a similar story prevails; LIFO-pools’ differen-tiated pools are slightly less favorable so the net benefit isapproximately 1.5x.

For Periodic (see Figure 16, 2nd from right), FIFO canpick a long target, but because LIFO-pools can target fivedifferentiated pools, the custom target for each one allowsa clever trade-off, optimizing for a lower success rate inpools in lieu for a higher payback (larger target). Unstableis a different story (see Figure 16, right) as pool statisticsthat produce high variability cannot be effectively char-acterized, even with a range of information model andVRMs. So, the result is uniformly low success rates.

11

Figure 15: Optimal Target Interval vs. VRM algorithm and Information Model

Figure 16: User Computation Success rate vs. VRM algorithm and Information Model

6.3 Exemplars and Realizable ValueA key question for volatile resource users is how the dy-namics of the instance pools affect the ability to extractvalue. First, it is important to take into account the totalavailable resource hours within each instance pool (seeTable 3). In Figure 17, the plain green bars show theuser value achieved using the Full distribution informa-tion model, divided by the total available resource hoursin the corresponding pool. This normalizes out the to-tal resource availability. Comparison across the four in-stance pool exemplars then shows clearly that the Un-stable pool’s violent fluctuations make capturing valuedifficult, causing a nearly 30% decrease. The Stable 1,Stable 2 and Periodic exemplars exhibit a small decrease(< 10%), but are similar to each other. The dotted greenbars in Figure 17 show the same ratio, selecting the high-est value achieved across information models (the win-ner is 90th percentile in all cases), and shows a simi-lar trend. This information model is more realistic, re-quiring only minimal information disclosure by the cloudprovider. Despite its attractiveness, it does achieve a lowervalue due to the lesser information available to users fortargeting.

7 Discussion and Related WorkA broad range of research explores use of volatile re-sources and cloud resource management. Topics includestatistical characterization and duration prediction, effi-cient volatile resource exploitation in public clouds, and

Figure 17: Relative User value per Resource-hour(VRM/Random) for the four exemplars.

efficient cloud resource management from the perspec-tive of cloud operators. In contrast, our work takes theperspective of a cloud operator, assumes foreground re-source management for reliable cloud services as given,and explores a variety of volatile resource managementalgorithms and information models on 608 Amazon EC2instance pools.

Characterization None of the commercial volatile re-source services provide statistical characterization of re-sources, though numerous external and academic effortshave sought to create such characterization. Many papershave conducted empirical studies of Amazon’s Spot mar-kets, constructing statistical models from historical price[21, 14] and revocation behavior [10]. Examples of keystatistical variables are interval durations (MTTR), varia-tion, and prediction of interval duration. Brevik and Wol-ski [25] observe that stationary statistics are a poor model,and use time-series techniques to predict the bid price

12

needed for reliable spot interval durations with a given re-liability SLO (also see [26]). These efforts are hamperedby the sparse information provided by cloud operators.In contrast, our work explores how different informationmodels affect achievable user value, complementing theseefforts. Amazon’s recent defined duration spot instancesthat allow a specification of 1 to 6 hours [2] are an exam-ple of a limited information model – where the durationdistributions are all longer than the defined duration.

Engineering Reliable Resources A number of re-search studies have explored reliability mechanisms suchas checkpointing [29, 13, 15, 17], replication [22, 27], andmigration [28, 16] to shield applications from resourcevolatility. These techniques use historical empirical char-acterization of resource volatility to reduce the overheadof reliability mechanisms. For example, Carvalho et al.[9]construct “economy class” computing from unused datacenter resources using statistical characterization of fore-ground load to predict future volatile resource availabil-ity, thus creating a class of nearly reliable resources. Incontrast, our work uses VRMs to engineer the propertiesof volatile resources, and explores the information modelspace cloud operators could provide to users to enable in-telligent user-management approaches.

Differentiated Pools and Information Models Shas-tri et al.[19] describe a statistically characterized volatileresource, using MTTR, called a“transient guarantee”. Us-ing LIFO resource management to divide resources (sim-ilar to LIFO-pools) on a single Google data center tracewith different MTTRs. Using a superlinear value model,they showed significant resource value increases.

Our work goes further in several key dimensions. First,we consider not just MTTR but four information mod-els (four different “transient guarantees”), showing howthe choice affects user value and that two (90pctile, Full)yield much greater value than MTTR. Second, we go be-yond LIFO, studying four different VRM algorithms, andshowing that they create significant differences in inter-val distribution and thereby user value. Third, we studythe VRMs and information models for over 600 instancepools. And fourth, we use a more challenging user valuemodel. In Shastri, value increases superlinearly withMTTR, so partitioning that produces higher MTTR in-creases overall user value. However, our linear value anduser-targeting is a more challenging hurdle. Increasingvalue requires reducing interval variability, which we dealwith directly in statistical characterization, targeting, andsuccess rate.

Value Functions Prior work [9, 19] that quantifies thevalue obtainable from volatile resources models computa-tions as long-lived. Carvalho et al. [9] model the loss invalue of a job assigned to a volatile resource in an ad hocmanner, asserting that availability < 99% entails a loss of30% (mimicing Amazon EC2 charging scheme). Shastri

et al. [19] endogenize the loss in value via the cost ofcheckpointing due to finite interval durations, and recom-putation due to unpredictability of interval durations.

In contrast, we model computations as having finitedurations without any checkpointing, assuming no un-certainty in the computation time. Thus, the value ob-tained by a user is a step function of the interval dura-tion with the only degree of freedom being the maximumvalue obtainable as a function of the resources required(v0(target)), for which we pick two representative cases:linear growth (v0(target) ∝ target), and super-linearpolynomial growth (v0(target) ∝ target1.5). In [19], theauthors use an exponential value function. In [18], the au-thors use a linear growth assumption, but assume that jobshave heterogeneous priorities with the slope of the lineargrowth an increasing function of job priority (however,that paper focuses on utility models that are a function oflatency; see also [23]). We instead assume jobs of a sin-gle priority that can saturate the volatile resources. Thisallows a somewhat cleaner comparison of resource man-agement policies and information models.

8 Summary and Future Work

The properties of the volatile resources that a cloudprovider offers to users with weak service guarantee arenot exogenous, but an outcome of the resource manage-ment algorithms used by the provider. Further, the abil-ity of users to extract value from these resources dependson the nature of statistical information provided by thecloud provider. Our broad study of 608 Amazon instancepools characterizes periodicity and availability based onspot price traces. Using four exemplars, we study fourvolatile resource management approaches and four infor-mation models that can be provided to the users. Ourstudy is based on Amazon EC2 spot price traces showsthat the choice of resource management algorithms canhave a large impact on the achieved value, up to ∼2X.We also show that simple statistical models do not cap-ture real distributions of volatile resources, leading to lowvalue to users. By comparing information models, weshow that the ability of customers to extract value arehighly-dependent on information model, with differenceup to ∼ 20X. We also compare all schemes to ideal real-izable value, and the results show that one can get veryclose to the ideal value with a carefully chosen combina-tion of VRM and infomation model. We compared theresults of the two Stable exemplars to 80% of the 608 in-stance pools, showing that results for these exemplars arebroadly representative of these 475 pools. Thus, a cloudoperator has to deliberately choose the volatile resourcemanagement approach and information model for users inorder to maximize value.

13

There are numerous directions in which the presentstudy can be extended. One direction is a more fine-grained value model of users’ utility incorporating uncer-tain job durations, their ability to checkpoint compute in-tensive jobs, and manage a portfolio of VMs across mul-tiple instance types for parallel jobs. A second directionin which the study can be extended is to incorporate thecloud provider’s ability to predict idle resources insteadof using historical aggregate statistics, and thereby pro-viding more targeted service level guarantees for volatileresources. How such a better targeting can be achievedwhile not revealing the utilization and other characteris-tics of the cluster resources is also an open challenge. Oneresource management algorithm that we have not stud-ied is “defined duration” type volatile resources where thecloud provider offers a hard upper bound on the durationof the intervals on the order of hours. Slicing longer in-tervals into such smaller intervals allows the provider totrade-off higher service level at the cost of lost value fromlonger intervals. Finally, taking a holistic view, it wouldbe interesting to explore the cloud provider’s problem ofjointly optimizing the menu of guaranteed (reserved/on-demand) resources and high quality volatile resources, thelatter of which may cannibalize the demand for former.

References[1] Amazon cloud services. https://aws.

amazon.com/.

[2] Amazon spot blocks. https://aws.amazon.com/cn/blogs/aws/new-ec2-spot-blocks-for-defined-duration-workloads/.

[3] Amazon spot instances. https://aws.amazon.com/ec2/spot/.

[4] Amazon spot instances price his-tory. http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances-history.html.

[5] Google compute engine. https://cloud.google.com/compute/.

[6] Google compute engine. https://cloud.google.com/compute/docs/instances/preemptible.

[7] Google preemptible vms. https://cloud.google.com/preemptible-vms/.

[8] O. Agmon Ben-Yehuda, M. Ben-Yehuda, A. Schus-ter, and D. Tsafrir. Deconstructing amazon ec2 spotinstance pricing. ACM Transactions on Economicsand Computation, 1(3):16, 2013.

[9] M. Carvalho, W. Cirne, F. Brasileiro, and J. Wilkes.Long-term slos for reclaimed cloud computing re-sources. In Proceedings of the ACM Symposium onCloud Computing, pages 1–13. ACM, 2014.

[10] N. Chohan, C. Castillo, M. Spreitzer, M. Steinder,A. N. Tantawi, and C. Krintz. See spot run: Usingspot instances for mapreduce workflows. HotCloud,10:7–7, 2010.

[11] E. Cortez, A. Bonde, A. Muzio, M. Russinovich,M. Fontoura, and R. Bianchini. Resource cen-tral: Understanding and predicting workloads forimproved resource management in large cloud plat-forms. In Proceedings of the 26th Symposium on Op-erating Systems Principles, SOSP ’17, pages 153–167, New York, NY, USA, 2017. ACM.

[12] W. Guo, K. Chen, Y. Wu, and W. Zheng. Biddingfor highly available services with low price in spotinstance market. In Proceedings of the 24th Inter-national Symposium on High-Performance Paralleland Distributed Computing, pages 191–202. ACM,2015.

[13] I. Jangjaimon and N.-F. Tzeng. Effective cost re-duction for elastic clouds under spot instance pricingthrough adaptive checkpointing. IEEE Transactionson Computers, 64(2):396–409, 2015.

[14] B. Javadi, R. K. Thulasiramy, and R. Buyya. Sta-tistical modeling of spot instance prices in publiccloud environments. In Utility and Cloud Comput-ing (UCC), 2011 Fourth IEEE International Confer-ence on, pages 219–228. IEEE, 2011.

[15] D. Jung, S. Chin, K. Chung, H. Yu, and J. Gil. Anefficient checkpointing scheme using price historyof spot instances in cloud computing environment.In IFIP International Conference on Network andParallel Computing, pages 185–200. Springer, 2011.

[16] D. Jung, S. Chin, K. S. Chung, and H. Yu. Vmmigration for fault tolerance in spot instance basedcloud computing. In International Conference onGrid and Pervasive Computing, pages 142–151.Springer, 2013.

[17] S. Khatua and N. Mukherjee. A novel checkpoint-ing scheme for amazon ec2 spot instances. In Clus-ter, Cloud and Grid Computing (CCGrid), 201313th IEEE/ACM International Symposium on, pages180–181. IEEE, 2013.

[18] C. B. Lee and A. E. Snavely. Precise and realisticutility functions for user-centric performance analy-sis of schedulers. In Proceedings of the 16th interna-

14

tional symposium on High performance distributedcomputing, pages 107–116. ACM, 2007.

[19] S. Shastri, A. Rizk, and D. Irwin. Transient guaran-tees: maximizing the value of idle cloud capacity. InHigh Performance Computing, Networking, Storageand Analysis, SC16: International Conference for,pages 992–1002. IEEE, 2016.

[20] Y. Song, M. Zafer, and K.-W. Lee. Optimal biddingin spot instance market. In INFOCOM, 2012 Pro-ceedings IEEE, pages 190–198. IEEE, 2012.

[21] S. Tang, J. Yuan, and X.-Y. Li. Towards optimal bid-ding strategy for amazon ec2 cloud spot instance. InCloud Computing (CLOUD), 2012 IEEE 5th Inter-national Conference on, pages 91–98. IEEE, 2012.

[22] W. Voorsluys and R. Buyya. Reliable provisioningof spot instances for compute-intensive applications.In Advanced information networking and applica-tions (aina), 2012 ieee 26th international conferenceon, pages 542–549. IEEE, 2012.

[23] J. Wilkes. Utility functions, prices, and negotiation,2008.

[24] J. Wilkes. More google cluster data. GoogleResearch Blog, November 2011. http://googleresearch.blogspot.com/2011/11/more-google-cluster-data.html.

[25] R. Wolski and J. Brevik. Providing statistical reli-ability guarantees in the aws spot tier. In Proceed-ings of the 24th High Performance Computing Sym-posium, page 13. Society for Computer SimulationInternational, 2016.

[26] R. Wolski, J. Brevik, R. Chard, and K. Chard. Proba-bilistic guarantees of execution duration for amazonspot instances. In Proceedings of the InternationalConference for High Performance Computing, Net-working, Storage and Analysis, SC ’17, pages 18:1–18:11, New York, NY, USA, 2017. ACM.

[27] Z. Xu, C. Stewart, N. Deng, and X. Wang. Blend-ing on-demand and spot instances to lower costs forin-memory storage. In Computer Communications,IEEE INFOCOM 2016-The 35th Annual IEEE In-ternational Conference on, pages 1–9. IEEE, 2016.

[28] S. Yi, A. Andrzejak, and D. Kondo. Monetarycost-aware checkpointing and migration on amazoncloud spot instances. IEEE Transactions on ServicesComputing, 5(4):512–524, 2012.

[29] S. Yi, D. Kondo, and A. Andrzejak. Reducingcosts of spot instances via checkpointing in the ama-zon elastic compute cloud. In Cloud Computing(CLOUD), 2010 IEEE 3rd International Conferenceon, pages 236–243. IEEE, 2010.

[30] M. Zafer, Y. Song, and K.-W. Lee. Optimal bids forspot vms in a cloud for deadline constrained jobs. InCloud Computing (CLOUD), 2012 IEEE 5th Inter-national Conference on, pages 75–82. IEEE, 2012.

[31] Q. Zhang, Q. Zhu, and R. Boutaba. Dynamic re-source allocation for spot markets in cloud comput-ing environments. In Utility and Cloud Computing(UCC), 2011 Fourth IEEE International Conferenceon, pages 178–185. IEEE, 2011.

15

Appendix A Summary of All In-stance Pools

All of the presented results are based on EC2 Spot In-stance price trace[4, 26] over 3 month-long period. Weperformed a broad comparison of volatile resource man-agement for 608 instance types, drawn from Amazon’sfour US regions, each which contains 2-6 availabilityzones. We use two statistical characterizations, volatility(standard deviation of the resource availability) and fre-quency ratio (period of 1 day vs. average)5 The frequencyratio captures the diurnal periodicity of load often seen incloud workloads. Scatterplots showing the distribution ofinstance pools for the four US regions are show in Fig-ure 18,19,20, and 21 from all availability zones in us-east-1, us-east-2, us-west-1, and us-west-2 accordingly.These distributions show similar characteristics: 1) a widespread of volatility, but with many instance pools with lowvolatility and 2) varied magnitudes of diurnal load.

Note that Figure 8 from Section 5 is a scatter plot ofone particular availability zone from us-west-2, so it is asubset plot of Figure 21 here. X-axis denotes the standarddeviation of availability, hence the availability volatility,and Y-axis represents the level of periodicity at a diurnalpattern.

Figure 18: Scatter-plot of Instance pool pricing behaviorin us-east-1 region

5We computed by taking the fast-fourier transform (FFT) of the re-source availability profile, and then the ratio of the frequency bands.

Figure 19: Scatter-plot of Instance pool pricing behaviorin us-east-2 region

Figure 20: Scatter-plot of Instance pool pricing behaviorin us-west-1 region

Figure 21: Scatter-plot of Instance pool pricing behaviorin us-west-2 region

16