min c: heterogeneous concentration policy for...

204

ISSN 0361-7688, Programming and Computer Software, 2017, Vol. 43, No. 3, pp. 204–215. © Pleiades Publishing, Ltd., 2017.Original Russian Text © F.A. Armenta-Cano, A. Tchernykh, J.M. Cortes-Mendoza, R. Yahyapour, A.Yu. Drozdov, P. Bouvry, D. Kliazovich, A. Avetisyan, S. Nesmachnow, 2017,published in Programmirovanie, 2017, Vol. 43, No. 3.

Min_c: Heterogeneous Concentration Policy for Energy-Aware Scheduling of Jobs with Resource Contention1

F. A. Armenta-Canoa, *, A. Tchernykha, **, J. M. Cortes-Mendozaa, ***, R. Yahyapourb, ****,A. Yu. Drozdovc, *****, P. Bouvryd, ******, D. Kliazovichd, A. Avetisyane, and S. Nesmachnowf

aCICESE Research Center, Ensenada, MexicobUniversity of Gottingen, Gottingen, Germany

cMoscow Institute of Physics and Technology, Moscow, RussiadUniversity of Luxembourg, Luxembourg

eISP RAS, Moscow, RussiafUniversidad de la Republica, Montevideo, Uruguay

*e-mail: [email protected], armentac, [email protected]**e-mail: [email protected]

***e-mail: [email protected]****e-mail: Pascal.Bouvry, [email protected]

*****e-mail: [email protected]******e-mail: [email protected]

Received June 2, 2016

Abstract—In this paper, we address energy-aware online scheduling of jobs with resource contention. We pro-pose an optimization model and present new approach to resource allocation with job concentration takinginto account types of applications and heterogeneous workloads that could include CPU-intensive, disk-intensive, I/O-intensive, memory-intensive, network-intensive, and other applications. When jobs of onetype are allocated to the same resource, they may create a bottleneck and resource contention either in CPU,memory, disk or network. It may result in degradation of the system performance and increasing energy con-sumption. We focus on energy characteristics of applications, and show that an intelligent allocation strategycan further improve energy consumption compared with traditional approaches. We propose heterogeneousjob consolidation algorithms and validate them by conducting a performance evaluation study using theCloud Sim toolkit under different scenarios and real data. We analyze several scheduling algorithms depend-ing on the type and amount of information they require.

DOI: 10.1134/S0361768817030021

1. INTRODUCTION

Cloud computing is an on-demand distributedmodel widely accepted by public and private organiza-tions. The cloud providers offer computationalresources and services that guarantee the Quality-of-Service (QoS) requirements. One of the main compo-nents of the operational cost of the system, and thusone of the major concerns of the cloud provider is theenergy expenditure.

Inefficient resource management has a direct neg-ative impact on performance and cost. In the sharedenvironments, it is often difficult to optimize energyconsumption of physical resources and virtualmachines (VMs) with different type of tasks (CPU-intensive, disk-intensive, I/O-intensive, memory-

intensive, network-intensive, etc.). Detailed energymanagement at granular levels should be used to opti-mize resource usage and profitability [1].

In this paper, we propose an energy optimizationmodel and heterogeneous job consolidation algo-rithms for energy-aware scheduling. Both of them takeinto account types of applications.

We evaluate energy efficiency of our strategies onreal data under different scenarios and compare themwith ones proposed in the literature.

The paper is structured as follow. Section 2 reviewsrelated work on the energy optimization. Section 3presents the problem definition, while the proposedscheduling algorithms are described in Section 4. Sec-tion 5 provides details of the experimental setup. Sec-tion 6 describes the methodology used for the analysis.Experimental validation is reported in Section 7.1 The article is published in the original.

PROGRAMMING AND COMPUTER SOFTWARE Vol. 43 No. 3 2017

Min_c: HETEROGENEOUS CONCENTRATION POLICY 205

Finally, Section 8 concludes the paper by presentingmain contribution and future work.

2. RELATED WORK

Energy required by computers for their operation,power supply, and cooling contribute significantly tothe total operational costs. Reducing energy con-sumption has emerged as one the main research issuesboth in industry and academia. Here, we briefly dis-cuss energy-aware resource allocation algorithms pre-sented in the literature.

EMVM – Energy-aware resource allocation heu-ristics for efficient management, by Beloglazov et al.[2]. The authors present resource provisioning andallocation algorithms utilizing the dynamic consolida-tion of VMs and describe principles of energy-efficientmanagement in Cloud computing environments. It isshown that the consolidation leads to a substantialreduction of energy consumption in comparison withstatic resource allocation techniques.

The used power consumption model is

where is the maximum power consumption when theserver is fully utilized; is the fraction of power con-sumed by the idle server (i.e. 0.7); and is the CPU uti-lization.

The total energy consumption is defined as an inte-gral of the power consumed over a given time interval:

When VMs do not use all resources, they can beconsolidated to the minimum number of physicalnodes. Idle nodes can be switched to the sleep mode toeliminate the idle power consumption.

= + − ,max max( ) (1 )P u kP k P u

= .∫1

0

( ( ))t

t

E P u t dt

Figure 1 descripts the behavior of energy consump-tion according to the described model varying CPUutilization.

HSFL – Hybrid Shuffled Frog Leaping algorithmby Luo et al. [3]. The resource management schemeused in the paper guarantees user quality of service(QoS) specified by SLAs, and achieves energy saving.The authors use VM migrations technology to consol-idate resources. Low utilized and idle hosts areswitched to power saving mode to achieve energy sav-ing while ensuring QoS.

The authors assume that the host energy consump-tion exhibits an almost linear proportion to CPUenergy consumption. Moreover, the energy consump-tion of an idle host accounts for 70% of full load oper-ation energy consumption, considering the energyconsumed during VM migrations. Energy consump-tion at any given time h is defined as follows:

is the energy consumption when host is in fullload. is the average utilization rate of the hostprocessor, is the collection of VM migrations, and

is the migration time of VM i. The relationbetween CPU utilization and energy consumption issimilar to one presented in Fig. 1.

AETC – Algorithm of Energy-aware Task Consol-idation, by Hsu et al. [4]. The authors propose anenergy-aware task consolidation (ETC) technique tominimize energy consumption. ETC is designed towork in a data center for VMs that reside on the samerack or on racks with permanent bandwidth. It restrictsCPU use above a specified peak threshold of 70% byconsolidating tasks amongst virtual clusters. Further-more, the energy cost considers network latency, whena task migrates to another virtual cluster. The idle state

∈

= ++ .∑

max max

max

( ) 0.7 ( ) 0.3 ( ) ( )0.1 ( )

i

E h E h Utlz h E h

E T iv

max( )E h( )Utlz hv

( )T i

Fig. 1. Linear function of energy consumption versus CPUutilization (%).

100%95%90%85%80%75%70%65%

100%90%

80%

70%

60%

50%

40%

30%

20%

10%0%

Ene

rgy

cons

umpt

ion

CPU utilization

Fig. 2. Stepwise function of energy consumption versusCPU utilization (%).

Ene

rgy

cons

umpt

ion 100%

10%

90%80%70%60%50%40%30%20%

100%90%

80%

70%

60%

50%

40%

30%

20%

10%0%

CPU utilization

206


ARMENTA-CANO et al.

of virtual machines and network transmission areassumed to have a constant part of the total energyconsumption.

The simulation results show that ETC can reducepower consumption by managing task consolidationfor Cloud systems.

Figure 2 shows the percentage of energy consump-tion depending on the CPU utilization used in ETCalgorithm.

The model assumes energy consumption E(Vi) = in the idle state. An additional energy is

required for executing tasks when CPU utilization isincreased.

The energy consumption of a virtual machine during the time period is defined as follows:

For a given a virtual cluster, which consists of VMs,the energy consumption is calculated as follows:

CTES – Cooperative Two-Tier Energy-AwareScheduling, by Hosseinimotlagh et al. [5]. Theauthors address a cooperative two-tier task schedulingapproach with regulation of the execution speeds oftasks in order to reach an optimum level of utilizationinstead of migrating tasks to other hosts. Several pre-dictive global task scheduling policies are proposed tomap arrived tasks to feasible VMs. Simulation resultsshow that this approach reduces the total energy con-sumption of a Cloud.

The utilization of a host at time is defined as:

, where is the allocated MIPS of

the host i, and is the maximum computing powerof the host i. The authors assume that an idle hostchanges its state to be powered off immediately. Thus,the total power of a host is defined as:

α /W s β

α⎧⎪β + α < ≤⎪

β + α < ≤⎪⎪= β + α < ≤⎨⎪ β + α < ≤⎪

β + α < ≤⎪⎪ β + α < ≤⎩

/ , if is idle,/ , if 0% CPU util 20%,

3 / , if 20% CPU util 50%,( ) 5 / , if 50% CPU util 70%,

8 / , if 70% CPU util 80%,11 / , if 80% CPU util 90%,12 / , if 90% CPU util 100%.

i

W sW sW s

E V W sW sW sW s

iV,0[ ]mt t

,=

= .∑0

0

( ) ( )m

m i t i

t

E V E V

, ,=

= .∑0 0

0

( ) ( )n

m k m i

i

E VC E V

( )iu t

= ( )( ) ii

i

ah tu tmh

( )iah t

imh

is the power consumed during the idle time of a

computing node defined as: = isthe power consumed when a host works with its maxi-mum utilization. is the constant ratio of the static powerof a host to its maximum power (0 < α ≤ 1) whichdepends on the physical characteristics of a host.

Dynamic power consumption is:

If the system uses the power P(u), the energy con-

sumption will be E = , where tmin is the timein which a host works at its maximum computingpower. Therefore, the energy consumption of host a isobtained by:

DVMA – A Decentralized Virtual Machine Migra-tion Approach, by Wang et al. [6]. The authors addressa decentralized virtual machine migration approach.They define a system model and power model basedon the load vectors, load information collection, VMselection, and destination determination.

The power consumption of a physical node is cal-culated as follows:

where is the power consumption of the nodewhen it is fully utilized (i.e., it reaches 100% of CPUutilization). α is the fraction of power consumed by anidle node compared to a full utilized node, and θ is thecurrent CPU utilization. The percentage of energyconsumption depending on the CPU utilization issimilar to the one shown on Fig. 1. Performance eval-uation results illustrate that the approach can achievebetter load balancing and less power consumptionthan other strategies.

EDRP – Energy and Deadline-aware ResourceProvisioning, by Gao et al. [7]. The authors focus onthe problem of minimizing the operation cost of aCloud system by maximizing its energy efficiencywhile ensuring that user deadlines as defined in Ser-vice Level Agreements are met. They take into accounttwo types of workload models, independent batchrequests and task graphs with dependencies.

The model of power consumption at time includesthe static power consumption , and thedynamic power consumption . Both of them

⎧ + , >= ⎨ ,⎩

static dynamic ( ) 0,( ( ))0 o. w.

i i ii i

P P u tP u t

staticiP

staticiP α static max

i iP P

= − .dynamic max static( ) ( )i i i iP P P u t

∫min

0( )

tu P u dt

= α + − α .max min[ (1 ) ] tE u Pu

= α + − α θ.max max(1 )i i iP P P

maxiP

static( )xP t

dynamic( )xP t



are correlated with the CPU utilization rate Utilx(t) attime t. Utilx(t) considers only CPU requirements of thehosted VMs, and does not differentiate between VMsthat are running tasks and idle VMs, since backgroundCPU activity is needed even during idle periods.

is a constant, when Utilx(t) > 0, and 0, other-

wise. The relationship between and Utilx(t)is complex. Servers have optimal utilization levels Optx

in terms of performance-per-Watt.

It is commonly accepted that for modern serversOptx ≈ 0.7, and the increase in power consumptionbeyond this operating point is more drastic than whenUtilx(t) < Optx. Even for identical utilization levels, theenergy efficiency of different servers may vary. This iscaptured by the coefficients αx and βx, representingthe power consumption increase when Utilx(t) < Optx

and Utilx(t) ≥ Optx, respectively. is calcu-lated as:

The authors point out that the exact formulation of

does not undermine the analysis, since itsincrement is faster when than when

The total energy consumption is the sum of thepower consumption across all servers throughout theoperation time interval:

static( )xP t

dynamic( )xP t

dynamic( )xP t

α <⎧⎪ α + − β⎨⎪ ≥⎩

2

( ) , if ( ( ) ),

( ( ) ) ,if ( ( ) ).

x x x x

x x x x x

x x

Util t Util t Opt

Opt Util t OptUtil t Opt

( )xdynamicP t

≥( )x xUtil t Opt< .( )x xUtil t Opt

where is the upper bound of the maximumschedule length.

Figure 3 shows the energy consumption dependingon the CPU utilization.

BFDP – Best Fit Decreasing Power, by Luo et al.[8]. The authors propose a simulation-driven method-ology with an energy model based on polynomialLASSO (least absolute shrinkage and selection opera-tor) regression. The resource scheduling algorithmBFDP aims to improve the energy efficiency withoutdegrading the QoS considering four type of jobs:CPU-intensive, Memory-intensive, Network-inten-sive and I/O-intensive. The authors introduced utili-zation thresholds to alleviate the over-consolidationissue in the Best-Fit strategy. The results showed thatBFDP creates less SLA violations than the BFDR inlight workloads.

= =

⎛ ⎞= + ,⎜ ⎟

⎜ ⎟⎝ ⎠

∑ ∑max

static dinamic

1 1

COSP ( ( ) ( ))M L

x x

x t

P t P t

maxL

Table 1. Characteristics of the related work algorithms

Application domain Characteristics Evaluation criteria

RefD

ata

cent

ers

Clo

ud

Hyb

rid

Clo

ud

Cen

tral

ized

Dec

entr

aliz

ed

On

line

Off

line

Cla

irvo

yant

Non

clai

rvoy

ant

QoS

Mig

ratio

n

Stat

ic

Dyn

amic

CPU

-Int

envi

ve

I/O

-Int

ensi

ve

Util

izat

ion

Ene

rgy

Dea

dlin

e

EMVM • • • • • • • • • • • [2]HSFL • • • • • • • • • [3]AETC • • • • • • • • • [4]CTES • • • • • • • • • • [5]DVMA • • • • • • • [6]EDRP • • • • • • • • • • • [7]BFDP • • • • • • • • • • [8]PAHD • • • • • • • [9]

Fig. 3. Nonlinear function of energy consumption (%) ver-sus CPU utilization (%).

100%90%

80%

70%

60%

50%

40%

30%

20%

10%0%

100%95%90%85%80%75%70%65%

Ene

rgy

cons

umpt

ion

CPU utilization

208


ARMENTA-CANO et al.

The nonlinear energy model is defined by:

where is the kernel function, xi is the CPU andmemory utilization. is the parameter determinedthrough the model training process. is a constant.

Figure 4 presents a relationship between CPU,memory utilization, and full-system power.

PAHD – Power-aware Applications HybridDeployment, by Liu et al. [9]. The authors presentI/O-Intensive and CPU-Intensive applications tooptimize resource utilization within virtualized envi-ronments. The experimental evaluation uses Xen asthe Virtual Machine Monitor. The experimentalresults show that power efficiency is improved up to2% to 12% under different resource allocation config-urations when compared with the default deployment.The main conclusion of the work is that allocatingtwice as much CPU for CPU-Intensive applicationscompared to I/O-Intensive application allows obtain-ing a significant improvement of the power efficiency.

Table 1 presents the summary of the related work aboutenergy-aware resource allocation algorithms. The tableincludes information about the application domains, themain characteristics of the proposed algorithms, and thecriteria used to evaluate their performance.

3. PROBLEM DEFINITIONWe consider a computing system that consists of m

homogeneous servers described by tuples {s, mem,band, eff} , where is a computational power, charac-terized by a number of operations per unit of time it iscapable to perform (MIPS), mem is the amount of mem-

=

= β + β φ + ε ,∑0

1

( )m

i j j i i

j

y x

φ ( )j ixβi

ε i

s

ory (MB), band is the available bandwidth (Mbps), andeff is energy efficiency (MIPS per watt). We also assumethat computers have enough resources to execute any job.The main objective of the proposed strategies is to mini-

mize the total energy consumption of running work-loads, while guarantee the QoS requirements.

3.1. Job Model

We consider independent jobs Thejob is described by a tuple , where

is the released time; is a compute capacityguaranteed for the application. It represents a userquality of service (QoS) specified by correspondingSLA (Service-Level Agreement). The release time of a job is not available before the job is submitted.

characterizes a job as CPU-intensive, disk-intensive, memory-intensive, network-intensive, I/Ointensive, etc.

opE

, , , .1 2 iJ J … J

jJ = , ,( )j j j jJ r type c≥ 0jr jc

jr

jtype

Fig. 4. Energy consumption versus CPU and memory utilization [8].

280260240220200180160140

Pow

er, W

1.0 0.80.6

0.40.2 0.2

0.40.6

0.8 1.0

0 Utilization of cpuUtilization of mem

Fig. 5. Normalized power consumption of jobs A and Bversus CPU utilization.

Type A Type B

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0CPU utilization

1.00.90.80.7

0.10.20.30.40.50.6

Nor

mal

ized

pow

erco

nsum

tion



3.2. Energy Model

We follow a nonlinear hybrid model of energy con-sumption proposed in [25]. Our model takes intoaccount power consumption of individual jobs andtheir combinations.

We assume that applications of different types withthe same CPU utilization contribute differently to thetotal power consumptions due to using different hard-ware.

In this paper, we consider two types of applications(type A and type B). Figure 5 descripts the normalizedpower consumption of these jobs versus CPU utiliza-tion.

Characteristics of these jobs influence differentlyon power consumption due to corresponding hard-ware characteristics. Moreover, allocation of two dif-ferent applications on the same server could causereduced power consumption, less than the sum of theirindividual power consumptions.

It also has an impact on performance enhancementavoiding creation of a bottleneck (either in CPU, diskor network) that may result in additional degradationof the system performance and increased energy con-sumption.

The power consumption at time consists of twoparts: the idle power consumption when the processor

is turned on, but not used , and the power con-

sumption when the processor is in use :

where , if the processor is on at time t, and otherwise. is the utilization at time t, r

is a coefficient proposed in [11] to use non-linearpower profiles.

i

procidlee

i

procused ( )e t

= +i i

proc proc procidle used( ) ( )( ( ) ( ) )r

i i ie t o t e e t U t

=( ) 1io t= ,( ) 0io t ( )iU t

= − β α ,i i i

proc proc procused max idle(( ) ( ( )))Ae e e t

Table 2. Job allocation strategies

Type Strategy Description

Know ledge Free Rand Allocates job j to a suitable machine randomly using a uniform distri-bution in the range [1…m]

FFit (First Fit) Allocates job j to the first machine available and capable to execute it;RR (Round Robin) Allocates job j to the machine available and capable to execute by

Round Robin strategyMin_L (Min load) Allocates job у to the machine with the least load at time rj:

mini = 1…m{ni}

Energy aware Min_Te (Min-Total_energy) Allocates job j to the machine with minimum total energy consumption

at time rj:

Min_e (Min-energy) Allocates job j to the machine with minimum power consumption

at time rj:

Utilization aware Min_u (Min-utilization) Allocates job j to the machine with minimum total utilization at time rj

Max_u (Max-utilization) Allocates job j to the machine with maximum total utilization at time rj

Min_ujt (Min-util_job_type) Allocates job j to the machine with minimum utilization of jobsof the same type at time rj

Min_c (Min-concentration) Allocates job j to the machine with minimum concentration of jobsof the same type at time rj

( )= =∑…

proc1 1min ( )jr

i m it e t

= …

proc1min ( ( ))i m i je r

= …

proc1min ( )i m iu

= …

proc1max ( )i m iu

Fig. 6. Normalized of power consumption versus propor-tion of jobs A.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0CPU utilization

1.00.90.80.7

0.10.20.30.40.50.6

Nor

mal

ized

pow

erco

nsum

tion

210


ARMENTA-CANO et al.

where is the maximum power consumption whenthe processor is fully utilized. is the coeffi-cient that represents the increment of power con-sumption when a processor runs different types of

applications. is the concentra-

tion (proportion) of jobs of type A among jobs allo-cated to the processor at the time t.

Due to the diversity of applications types, and theircombinations, we propose to use the aggregated utili-zation of each application type instead of consideringall possible job combinations. We consider the aggre-gated utilization as a sum of utilizations of jobs of thesame type. The aggregated utilization of all jobsof type A allocated to the processor at time is:

The utilization of a processor contributed by a job at

time is defined as: , where is the comput-

ing capacity guaranteed for the application, and is themaximum computing power (MIPS) of the processor.Figure 6 shows the power consumption depending onthe concentration of jobs of type A, when the proces-sor runs jobs A and B.

The total power consumption is the integral of thepower consumed during operation time :

i

procmaxe

β α( ( ))A t

α = +( )( )

( ) ( )A

AA B

U ttU t U t

( )AU tt

∈

= .∑( ) ( )A j

j A

U t u t

=( )

( ) jj

c tu t

s

maxC

==

= , = .∑∫max

op op op proc

11

with ( ) ( )c m

i

it

E E dt E t e t

We define as a reference value .We set for (when all jobs are type A),and for (when all jobs are type B).

Figure 7 shows the normalized power consumptiondepending on the CPU utilization and proportion of jobsA, when the processor runs two types of applications.

In Figure 7, we can see that when the CPU utiliza-tion is low, the power consumption is low. In this case,the combination of jobs has no significant impact onthe total power consumption; the lower blue zone hasa f lat surface.

Left and right zones of the surface reflect the pre-dominance of type A jobs and type B jobs, respec-tively. In both cases, we can observe that when utiliza-tion increases to its highest level, power consumptionalso increases to its highest level.

On the other hand, if the concentration stays inbalance between the two types of jobs, even, if the uti-lization increases to its highest level of 0.8–1, thepower consumption does not reach its highest level.

4. SCHEDULING ALGORITHMS

In this section, we describe our schedulingapproach and the energy-aware algorithms.

4.1. Scheduling Approach

We address a basic two-level scheduling approach[12–15]. At the upper level, the system, having a gen-eral view of job requests and available resources, allo-cates jobs to suitable (admissible) machines using agiven selection criteria. The local resource manage-ment system executes jobs on the lower level.

= .i i

proc procidle max0 2e e

β α = ,( ) 1A α = ,1A

β α = .( ) 0 9A α = 0A

Fig. 7. Normalized power consumption of job A and B versus CPU utilization and proportions of jobs A.

1.00.8

0.6

0.4

0.201 0.90.8 0.70.6 0.5 0.4 0.30.2 0.2

0.40.6

0.8

0.1 0 0CPU Utilization

αA

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

Nor

mal

ized

pow

erco

nsum

tion



4.2. Allocation StrategiesThe decision making process for job allocation is

based on different criteria. In this paper, we study tenallocation strategies Rand (Random), FFit (First Fit),RR (Round Robin), Min_L (Min Load), Min_Te(Min Total_energy), Min_e (Min energy), Min_u(Min utilization), Max_u (Max utilization), Min_ujt(Min utilization of job type), and Min_c (Min-con-centration) (see Table 2).

We categorize them in three groups by the type andamount of information used for allocation decision:(1) knowledge-free, with no information about appli-cations and resources [16–18, 26]; (2) energy-aware,with power consumption information; and (3)utiliza-tion-aware with CPU utilization information.

5. EXPERIMENTAL SETUPIn this section, we present the experimental setup,

including workload and evaluation scenarios, anddescribes the methodology used for the analysis. Allexperiments are performed using the Cloud Sim test-bed: a framework for modeling and simulation ofcloud computing infrastructures and services. It is astandard trace based simulator used to study cloudresource management problems. We have extendedCloud Sim to include our algorithms using the java(JDK 7u51) programming language.

5.1. WorkloadPerformance evaluation of scheduling strategies

requires careful experimentation in testbed environ-ments. A critical element of these experiments is theuse of realistic workloads that can represent thedeployment conditions. Our workload is based ontraces of real HPC from Parallel Workloads Archive[21], and grids from Grid Workload Archive [22]. It issuitable for our scenarios, where machines areintended to execute jobs traditionally executed onGrids and HPC machines.

The workloads include nine traces from: DAS2-University of Amsterdam, DAS2–Delft University ofTechnology, DAS2–Utrecht University, DAS2–Leiden University, KHT, DAS2–Vrije UniversityAmsterdam, HPC2N, CTC, and LANL. Detailsinformation about the log characteristics and work-loads can be found in [21] and [22].

Some workload aspects such as the job demanddistribution over the time of a day, day of a week, andtime zones can affect the evaluation of the strategies.Therefore, we use time normalization by shifting theworkloads by a certain time interval to represent amore realistic setup. We consider time-zone normal-ization, recording time normalization, and invalidjobs filtering. We fit all workloads to the same timezone, so all traces begin at the same weekday and timeof the day. Note that the alignment is related to thelocal time, hence, the time differences correspondingto the original time zones are maintained.

Several filters are applied to remove jobs withinconsistent information, for example, negative valuesof submit time, runtime, number of allocated proces-sors, requested time and user ID; job with failed orpartial execution, and canceled jobs (either beforestarting or during run).

We also add two fields to the Standard Workload For-mat (swf) to describe the type of jobs and CPU utilization.

We address a non-clairvoyant on-line schedulingproblem, hence, scheduling decisions must be madewithout complete information about the entire prob-lem instance and any future jobs. Jobs arrive one byone over time. The processing times of jobs areunknown initially and during run time. They becomeknown only when jobs actually complete.

Figure 8 shows the characteristics of the workloads,reporting the job distribution per week. In order to obtainvalid statistical values, we consider a thirty weeks work-load. It depicts total number of jobs, the number of jobsof type A and type B. Figure 8 shows the job distributionper week. In order to obtain valid statistical values, weconsider a thirty weeks workload. It depicts total numberof jobs, the number of jobs of type A and type B.

In Fig. 8, we observe that there is no predominanceof one type of jobs in logs. Some weeks have more jobsof type A, and others, more jobs of type B.

Additional features of the studied workloads arepresented in the workload characterization in FiguresA1 to A5 (in the Appendix). Figures A1, A2 and A3report histograms for the total number of jobs perhour, mean number of jobs per day, and per hours,

Fig. 8. Number of jobs per week.

50

40

30

20

10

01 4 7 10 13 16 19 22 25 28

Weeks

Jobs

(×10

3 )Job A Job B

212


ARMENTA-CANO et al.

respectively. Figures A4 and A5 depict the distributionof each type of job per week and hour, for a moredetailed analysis.

5.2. ScenariosFollowing the power consumption of a Fujitsu

PRIMERGY TX300 S7 processor [11], for our scenar-

ios, we set the values = 300, , andfor the non-linear power profiles r = 1.5.

6. METHODOLOGY OF THE ANALYSIS6.1. Degradation in Performance

In order to provide effective guidance in choosingthe best strategy, we perform an analysis of the powerconsumption according to the mean degradationmethodology proposed in Tsafrir et al. [24], andapplied for scheduling problems in [12, 17, 20].

First, we evaluate the degradation in performance (rel-ative error) of each strategy under the metric. This is donerelative to the best performing strategy for the metric:

We average these values and rank the strategies.The best strategy with the lowest average performancedegradation has rank 1. Note that we try to identifystrategies which perform reliably well in different sce-narios; that is, we try to find a compromise that con-siders all our test cases. The rank of the strategy may benot the same for all scenarios.

We present metric degradation averages to evaluateperformance of the strategies and study if some strate-gies tend to dominate results.

i

procmaxe = .

i i

proc procidle max0 2e e

γ − ×

γ = .

( 1) 100 withstrategy metric value

best found metric value

6.2. Performance Profile

The degradation approach allows analyze resultsbased on the mean values. To eliminate the influenceof a small portion of data with large deviation on thebenchmarking process, and help with the interpreta-tion of the data, we present performance profiles ofour strategies.

The performance profile is a non-decreasing,piecewise constant function that presents the proba-bility that a ratio is within a factor of the best ratio[23]. The function is the cumulative distributionfunction. Strategies with large probability forsmall are to be preferred.

7. EXPERIMENTAL VALIDATION

In this section, we report the experimental resultsof the proposed strategies. First, we evaluate themaccording to degradation in performance methodol-ogy described in Section 6.1 and present their ranking.

ρ τ( )

γρ τ( )

ρ τ( )

Fig. 9. Power consumption degradation per week.

21%18%15%12%9%6%3%0%

1 4 7 10 13 16Weeks

19 22 25 28

Rand FFit RRMin_u Min_ujtMax_uMin_e Min_Te

Min_LMin_c

1

5

9

2

6

3

7

4

8

6

10

10

5 4

12 3

9

87Pow

er d

egra

datio

n

Fig. 10. Mean power consumption degradation per strategy.

15%12%9%6%3%0%

Min_ujt

Min_c

Min_u

Min_L

Min_e

Min_Te

Max

_uRan

dFFitRR

Pow

er d

egra

datio

n

Strategies

Fig. 11. Performance profile of the power degradation of10 strategies.

Rand FFit RRMin_u Min_ujtMax_uMin_e Min_Te

Min_LMin_c

1 2 3 4

5 6

62

10

4

78

3

5 91

7 8

9 10

10.8

0.60.4

0.20

00.

01

0.20

0.02

0.19

0.03

0.18

0.04

0.17

0.05

0.16

0.06

0.15

0.07

0.08

0.09

0.10

0.11

0.12

0.13

0.14

τ = [0...0.2]

δ



Then, we present more detail analysis based on perfor-mance profiles of the strategies described in Section 6.2.

7.1. Power Consumption egradation AnalysisFigure 9 shows the power consumption degrada-

tion per week. A small percentage of degradation indi-cates that the performance of a strategy is close to thebest result obtained by all strategies. Therefore, smalldegradations represent better results.

We observe that the degradations vary greatly weekper week, and from strategy to strategy. However,Max_u shows worst behavior almost in all weeks, andMin_c and Min_ujt show best behavior. Last two strat-egies dominate in all test cases.

Figure 10 shows the mean power consumption deg-radation over all weeks. Max_u, Min_Te, and FFitstrategies show worst power degradations and ranked10, 9, 8, respectively. Min_c and Min_ujt are the beststrategies with the lowest average performance degra-dation and have ranks 1 and 2, respectively.

7.2. Performance profile analysisAs mentioned in Section 6, conclusions based on

the average values may have some negative aspects.A small portion of problem instances with large deviationcould change the mean value and change conclusionsbased on averages. To analyze results in more details, wepresent performance profiles of the strategies.

Figure 11 shows the performance profiles of our 10strategies according to power consumption in theinterval It displays large discrepanciesin the power consumption degradation on a substan-tial percentage of the problems.

Min_c has the highest ranking and the highestprobability of being the better strategy. If we choosebeing within a factor of 0.01 from the best result as thescope of our interest, the probability that it is the win-ner on a given problem is close to 1. Min_ujt has thesecond highest ranking of being the best strategy.Within a factor of 0.01 of the scope of our interest, itwins with probability of 0.96.

8. CONCLUSIONSIn this paper, we propose new approach to resource

allocation taking into account application characteris-tics. The main idea of our approach is based on the factthat different applications use different resources.Applications might be computing-bound, heavily usingCPUs, I/O-bound, requiring high bandwidth, memory-bound, disk-bound, etc. When jobs of one type are allo-

τ = , , . .[0 0 2]…

cated to the same resource, they may create a bottleneckand resource contention either in CPU, memory, disk ornetwork. It may result in degradation of the system per-formance and increasing energy consumption.

Current schedulers are unaware of the applicationtype. We propose to allocate them in controllable way,and mix them in such a way that use resources moreefficiently with reduced power consumption.

In this paper, we propose and analyze a variety ofjob allocation strategies that take into account bothdynamic resource state information and job proper-ties. We conduct a comprehensive performance evalu-ation study of 10 strategies using the Cloud Sim toolkitunder different scenarios and real data. Our experi-mental analysis results in several contributions:

(a) We identify the problem of the allocation of jobswith different characteristics to make sophisticatedscheduling decisions with respect to power consump-tion optimization;

(b) To provide effective guidance in choosing agood strategy, we perform, first, an analysis based onthe degradation in performance of each strategy, then,based on their performance profiles.

(c) We find that the information about the utiliza-tion of resources without knowledge of jobs types doesnot help to improve significantly the allocation strate-gies. Based on these results, we show that being awareof types of applications, we can apply intelligent allo-cation strategies using job concentration informationto improve energy consumption, and performanceenhancement avoiding creation of bottlenecks andresource contentions either in CPU, disk or network.

(d) Simulation results presented in the paper revealthat in terms of minimizing power consumption, minconcentration allocation strategy Min_c outperformsother algorithms. It dominates in almost all test cases.We conclude that the strategy is stable under differentconditions. It provides minor performance degrada-tion and copes with different demands.

However, further study for energy consumption ofmultiple job types and their concentration on the het-erogeneous resources is required to assess the actualefficiency and effectiveness of the proposed method.Careful profiling and characterization of applicationsinto several categories is important to share resourceswith least resource contention and power consump-tion. This will be the subject of future work for betterunderstanding of the resource contentions and theirimpact on the energy consumption.

Acknowledgment. This work is partially supportedby CONACYT (Consejo Nacional de Ciencia y Tec-nologia, Mexico), grant no. 178415. The work of

214


ARMENTA-CANO et al.

Drozdov is partially supported by the Ministry of Edu-cation and Science of Russian Federation under con-tract nos. 02.G25.31.0061 12/02/2013 (GovernmentRegulation no. 218 from 09/04/2010). The work of Nes-machnow is partly supported by ANII (project ANIIFSE_1_2013_1_10794) and PEDECIBA, Uruguay.

APPENDIX. WORKLOAD CHARACTERIZATION

REFERENCES1. Kliazovich, D., Pecero, J.E., Tchernykh, A., Bouvry, P.,

Khan, S.U., and Zomaya, A.Y., CA-DAG: Modelingcommunication-aware applications for scheduling inCloud computing, J. Grid Comput., 2015.

2. Beloglazov, A., Abawajy, J., and Buyya, R., Energy-aware resource allocation heuristics for efficient man-agement of data centers for Cloud computing, FutureGener. Comput. Syst., 2012, vol. 28, no. 5, pp. 755–768.

3. Luo, J., Li, X., and Chen, M., Hybrid shuffled frogleaping algorithm for energy-efficient dynamic consol-idation of virtual machines in cloud data centers, ExpertSyst. Appl., 2014, vol. 41, no. 13, pp. 5804–5816.

4. Hsu, C.-H., Slagter, K.D., Chen, S.-C., andChung, Y.-C., Optimizing energy consumption withtask consolidation in clouds, 2014, Inf. Sci., vol. 258,pp. 452–462.

5. Hosseinimotlagh, S., Khunjush, F., and Hosseinimot-lagh, S., A cooperative two-tier energy-aware schedul-ing for real-time tasks in computing Clouds, Proceed-ings of the 2014 22Nd Euromicro International Confer-ence on Parallel, Distributed, and Network-Based Pro-cessing, Washington, DC, USA, 2014, pp. 178–182.

6. Wang, X., Liu, X., Fan, L., and Jia, X., A DecentralizedVirtual Machine migration approach of data centers forCloud computing, Math. Probl. Eng., 2013, vol. 2013,p. e878542.

Fig. A4. Number of jobs per day of the week.

Jobs

(×10

3 )

5101520253035

00 2 4 6 8 10 12

Hours14 16 18 20 22

1

1

2

2

3

3

Job A Job BAll jobs

Fig. A5. The number of jobs per hours.

Jobs

(×10

3 )

Job A Job B

Mon Tue Wed Thu Fri Sat SunDays

12010080604020

0

Fig. A3. Mean number of jobs per day during a week.

Jobs

(×10

3 )

7 6 5 4 3210

Mean

Mon Tue Wed Thu Fri Sat SunDays

Fig. A2. Mean number of jobs per hour during a day.

Jobs

(×10

3 )

7 6 5 4 3210

0 2 4 6 8 10 12 Hours

14 16 18 20 22

Mean

Fig. A1. Total number of jobs per hour.

200

987654321

4 6 8 10 12 14 16Hours

18 2220

MonFri

Tue ThuSat

WedSun

1

1

2

2

3

3

5

5

6

6

7

7

4

4

Jobs

(×10

3 )



7. Gao, Y., Wang, Y., Gupta, S.K., and Pedram, M., Anenergy and deadline aware resource provisioning,scheduling and optimization framework for Cloud sys-tems,” Proceedings of the Ninth IEEE/ACM/IFIP Inter-national Conference on Hardware/Software Codesign andSystem Synthesis, Piscataway, NJ, USA, 2013, pp. 31:1–31:10.

8. Luo, L., Wu, W., Tsai, W.T., Di, D., and Zhang, F.,Simulation of power consumption of cloud data cen-ters, Simul. Model. Pract. Theory, 2013, vol. 39,pp. 152–171.

9. Liu, Z., Ma, R., Zhou, F., Yang, Y., Qi, Z., and Guan, H.,Power-aware I/O-intensive and CPU-intensive appli-cations hybrid deployment within virtualization envi-ronments, 2010 IEEE International Conference on Prog-ress in Informatics and Computing (PIC), 2010, vol. 1,pp. 509–513.

10. Lezama, A., Tchernykh, A., and Yahyapour, R., Per-formance evaluation of infrastructure as a ServiceClouds with SLA constraints, Computacion y Sistemas,2013, vol. 17, no. 3, pp. 401–411.

11. Matthias Splieth, S.B., Analyzing the Effect of Load Dis-tribution Algorithms on Energy Consumption of Servers inCloud Data Centers, 2015.

12. Tchernykh, A., Lozano, L., Schwiegelshohn, U., Bou-vry, P., Pecero, J., and Nesmachnow, S., Energy-awareonline scheduling: Ensuring quality of service for IaaSClouds, International Conference on High PerformanceComputing Simulation (HPCS 2014), Bologna, Italy,2014, pp. 911–918.

13. Tchernykh, A., Schwiegelsohn, U., Yahyapour, R., andKuzjurin, N., Online hierarchical job scheduling ongrids with admissible allocation, J. Scheduling, 2010,vol. 13, no. 5, pp. 545–552.

14. Tchernykh, A., Ramirez, J., Avetisyan, A., Kuzjurin, N.,Grushin, D., and Zhuk, S., Two level job-schedulingstrategies for a computational grid, Parallel Processingand Applied Mathematics, 6th International Conferenceon Parallel Processing and Applied Mathematics, Poznan,Poland, 2005, Wyrzykowski, R. et al., Eds., LNCS 3911,Springer-Verlag, 2006, pp. 774–781.

15. Dorronsoro, B., Nesmachnow, S., Taheri, J., Zomaya, A.,Talbi, E-G., Bouvry, P., A hierarchical approach forenergy-efficient scheduling of large workloads in multi-core distributed systems, Sustainable Comput.: Inform.Systems, 2014, vol. 4, pp. 252–261.

16. Tchernykh, A., Pecero, J., Barrondo, A., and Schaef-fer, E., Adaptive energy efficient scheduling in peer-to-

peer desktop grids, Future Generation Comput. Systems,2014, vol. 36, pp. 209–220.

17. Ramirez, J.M., Tchernykh, A., Yahyapour, R., Schwie-gelshohn, U., Quezada, A., Gonzalez, J., and Hirales, A.,Job allocation strategies with user run time estimates foronline scheduling in hierarchical grids. J. Grid Comput.,2011, vol. 9, pp. 95–116.

18. Iturriaga, S., Nesmachnow, S., Dorronsoro, B., andBouvry, P., Energy efficient scheduling in heteroge-neous systems with a parallel multiobjective localsearch, Computing Informatics, 2013, vol. 32, no. 2,pp. 273–294.

19. Schwiegelshohn, U. and Tchernykh, A., Online sched-uling for cloud computing and different service levels,26th Int. Parallel and Distributed Processing Symposium,Los Alamitos, CA, 2012, pp. 1067–1074.

20. Tchernykh, A., Lozano, L., Schwiegelshohn, U., Bou-vry, P., Pecero, J., Nesmachnow, S., and Drozdov, A.,Online bi-objective scheduling for IaaS Clouds withensuring quality of service, J. Grid Computing,Springer-Verlag, 2015. doi doi 10.1007/s10723-015-9340-0

21. Parallel Workload Archive, 2014. http://www.cs.huji.ac.il/labs/parallel/ workload.

22. Grid Workloads Archive, 2014. http://gwa.ewi.tudelft.nl.

23. Zitzler, E., Evolutionary algorithms for multiobjectiveoptimization: Methods and applications, PhD Thesis,Zurich: Swiss Federal Institute of Technology, 1999.

24. Tsafrir, D., Etsion, Y., and Feitelson, D., Backfillingusing system-generated predictions rather than userruntime estimates, IEEE Trans. Parallel DistributedSystems, 2007, vol. 18, no. 6, pp. 789–803.

25. Armenta-Cano, F., Tchernykh, A., Cortes-Mendoza, J.M.,Yahyapour, R., Drozdov, A., Bouvry, P., Kliazovich, D.,and Avetisyan, A., Heterogeneous job consolidation forpower aware scheduling with quality of service, Pro-ceedings of the 1st Russian Conference on Supercomputing –Supercomputing Days, Moscow, Russia, 2015, Voevodin,V. and Sobolev, S., CEUR-WS, 2015, vol. 1482,pp. 687–697. http://ceur-ws.org/Vol-1482/. URN:urn:nbn:de:0074-1482-7.

26. Rodriguez, A., Tchernykh, A., and Ecker, K., Algo-rithms for dynamic scheduling of unit execution timetasks, Europ. J. Operational Res., Elsevier Science,North-Holland, 2003, vol. 146, no. 2, pp. 403–416.

SPELL; OK

min c: heterogeneous concentration policy for...

Documents