issue december 2011 in hpc data centers - cmcc · 2006. it is reported that power and cooling costs...

Research PapersIssue 2011December 2011

Scientific Computing andOperation (SCO)

Green Computing and power savingin HPC data centers

By Osvaldo MarraCMCC

University of Salento, [email protected]

Maria MirtoCMCC

[email protected]

Massimo CafaroCMCC


and Giovanni AloisioCMCC


SUMMARY Data centers now play an important role in modern ITinfrastructures. Much research effort has been made in the field of greendata center computing in order to save energy, thus making data centersenvironmentally friendly, for example by reducing CO2 emission. TheScientific Computing and Operations (SCO) division of the CMCC hasstarted a research activity exploiting proactive and reactive monitoringtechniques in order to reduce the power consumption of data centers byusing Green Computing technologies. The goal of this research paper is toprovide an overview about the main issues and challenges to improve HPCresources management in terms of power consumption. A feasibility studyon the possibility to introduce a system to measure the power consumptionfor the CMCC data center, and in particular for the Calypso cluster, isdescribed.

02

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i

CMCC Research Papers

INTRODUCTION

High performance computing (HPC) - us-ing commodity clusters, special-purpose mul-tiprocessors, massive data stores and high-bandwidth interconnection networks - is rapidlybecoming an indispensable research tool inscience and engineering. In studies on cli-mate changes, HPC is fundamental for mod-eling and predicting climate changes and com-putation is taking an increasingly important rolebeside theory and experimentation as a toolfor inquiry and discovery. Beyond science andengineering, large compute clusters, massivestorage, and high-bandwidth connectivity havealso become mission-critical infrastructure inindustrial and academic settings. While thecosts of computers, storage, and networkinghave fallen dramatically over time, the costs ofthe buildings, power, and cooling that supportthis equipment have risen dramatically. Indeed,the costs for power and infrastructure exceedthe cost of the computing equipment they sup-port. The environmental costs of informationand communication technologies are also in-creasing; recent studies estimate that comput-ing and communication technology sectors areresponsible for 2% of global carbon emissionsand these emissions are increasing at 6% an-nually [1]. Green computing (also known assustainable computing) can be broadly definedas the problem of reducing the overall carbonfootprint (emissions) of computing and com-munication infrastructure, such as data cen-ters, by using energy-efficient design and op-erations. The area has garnered increasing at-tention in recent years, both from a technologyand societal standpoint, due to the increasedfocus on environmental and climate change is-sues. Hence, there is a need to balance thedramatic growth of high-performance comput-ing clusters and data centers in the computa-tional sciences with green design and use so

as to reduce the environmental impact. Techni-cal issues in high-performance green comput-ing span the spectrum from green infrastruc-ture (energy-efficient buildings, intelligent cool-ing systems, green/renewable power sources)to green hardware (multi-core computing sys-tems, energy-efficient server design, energy-efficient solid-state storage) to green softwareand applications (parallelizing computationalscience algorithms to run on modern energy-efficient multi-core clusters, intelligent load dis-tribution and CPU switch-off). The SCO divi-sion of the CMCC has started a research activ-ity about the use of proactive and reactive mon-itoring techniques in order to reduce the powerconsumption of data centers by using GreenComputing technologies. The goal of this re-search paper is to provide an overview aboutthe main issues and challenges to improve theHPC resources management in terms of powerconsumption. In particular, a feasibility studyon the possibility to introduce a system to mea-sure the power consumption for the CMCC datacenter, and in particular for the Calypso clusteris given. The research paper is articled as fol-lows. Starting from an overview of data centers,several issues related to their management arediscussed. In particular several metrics havebeen introduced in order to measure the powerconsumption (e.g., PUE). Moreover, several al-gorithms and techniques are described in orderto balance the workload on the cluster. Henceseveral benchmarking tools are introduced forevaluating the system performances by mea-suring the power consumption when varying theworkload. Finally, a feasibility study on the pos-sibility to introduce a system to measure thepower consumption for the CMCC data center,and in particular for the Calypso cluster, is pro-vided.

Green Computing and power saving in HPC data centers

03

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i

TOWARDS “GREEN” DATA CENTERS

A data center hosts computational power, stor-age and applications required to support an en-terprise business. A data center is central tomodern IT infrastructure, as all enterprise con-tent is sourced from or passes through it. Datacenters can be broadly classified, on the basisof power and cooling layout, into one of the 4tiers:

Tier 1: Single path for power and cooling;no redundant components;

Tier 2: Redundancy added to Tier 1,thereby improving availability;

Tier 3: Multiple power and cooling distri-bution paths, of which one is active;

Tier 4: Two active power and coolingpaths, and redundant components oneach path.

This classification however, is not precise andcommercial data centers typically fall betweenTiers 3 and 4 [21]. A higher tier implies animprovement in resource availability and reli-ability, but it comes at the expense of an in-crease in power consumption. Data centershost services that require high availability, closeto 99.99%. Fault tolerance, therefore, becomesimperative. The loss of one or more compo-nents must not cause the data center to termi-nate its services to clients. Consequently, datacenters feature hardware redundancy. Further-more, data centers are designed for a peakload, which might be observed only occasion-ally, and for short bursts of time. This con-servative design results in over-provisioning ofhardware in a data center. All these factorscombined together contribute to the high powerconsumption of data centers. So the electric-ity usage is the most expensive portion of adata center’s operational costs. In fact, the

U.S. Environmental Protection Agency (EPA)reported that 61 billion KWh, 1.5% of US elec-tricity consumption, is used for data center com-puting [18]. Additionally, the energy consump-tion in data centers doubled between 2000 and2006. It is reported that power and coolingcosts are the most dominant costs in data cen-ters [7]. Other studies ([4], [40]) have reportedthe global electricity costs for data centers run-ning into billions of dollars. Thus there is agrowing pressure from the general public forthe IT society to offer greendata center infras-tructures and services [17]. There are two ma-jor and complementary methods [38] to build agreen data center: (1) utilize green elementsin the design and building process of a datacenter, (2) greenifythe process of running andoperating a data center in everyday usage. Var-ious research activities have been carried outto manage data centers in a green mode (thelatter method), such as reducing data centertemperature [36], [42], [43], increasing serverutilization [33], [35], [41], and decreasing powerconsumption of computing resources [22], [23],[27], [32]. A fundamental research topic for theabove study is how we can define performancemetrics to identify how green a data center is.Green performance metrics, for data centers,are a set of measurements that can qualita-tively or quantitatively evaluate the environmen-tal impact of a data center. Nowadays, the termgreen computingrequires clear scientific defini-tions and it is more or less a marketing concept.Research on developing green data center met-rics brings solid specifications and definitionsfor green data centers in the following aspects:

identify and specify clearly how green adata center is, for example by calculatingits energy efficiency or greenhouse gasemission per time unit;

evaluate data center products and com-pare similar data centers;

04

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i


track green performance to increase adata center’s green efficiency, and pro-vide guidance to engineers, manufactur-ers, and service providers;

research and development on futuregreen data center technologies.

In the power/energy metrics Section, severalperformance metrics are introduced.

ENERGY USE IN A DATA CENTER

Making data centers more energy efficient isa multidimensional challenge that requires aconcerted effort to optimize power distribution,cooling infrastructure, IT equipment and IT out-put. A data center is composed of severalcomponents involved in the power consumption(Figure 1):

cooling equipment (e.g., computer roomair-conditioner units);

power equipment (e.g., uninterruptiblepower supplies and power distributionsunits);

IT equipment (e.g., rack optimized andnon-rack optimized enterprise servers,blade servers, storage and networkingequipment);

miscellaneous equipment (e.g., lighting).

The cooling system occupies a significant part,in terms of energy consumption, in a data cen-ter. The IT components present inside a datacenter when running produce a high heat. Toavoid overheating and consequent damage ofIT components, it is necessary to cool them andkeep them continuously at a constant tempera-ture. For this purpose, a cooling system whichincludes chillers, pumps and fans is used. Fig-ure 2 shows a typical cooling system, with mod-ular conditioning units and a raised floor, which

Figure 1:Components of a Data Center


05

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i

Figure 2:A cooling system

allows the circulation of cold air. The size andpower of the system is based on the expectedload of the servers and is chosen during thedesign phase of the data center.

The air flow is evenly distributed throughthe CRAC (Computer Room Air Conditioning),which controls the temperature through a sen-sor typically set at 20°C, which measures theinternal temperature. The cold air is distributedthrough the floor of the room with the use offans and pumps, cooling the servers from be-low; the hot air, which goes up, goes back inthe air conditioner to be cooled again and dis-tributed in the data center. The combination ofthese components ensures optimal longevity ofthe equipment. At the same time, however, thisecosystem has a significant cost in the econ-omy of the data center. The power equipmentincludes an energy supply system for the datacenter. Typically, the infrastructure includes theconnection to the electricity grid, generators,backup batteries and the energy for the cool-ing system. The energy is recovered exter-nally from the power supply. In the event of apower failure occurrence within the data center,a generator will supply the necessary energy tothe system. Electrical device backup batter-ies are recharged, with the aim of maintaining

a constant power even in case of interruptionsboth from the mains and the internal genera-tor and to compensate for fluctuations of thenetwork or temporary loss of power. The en-ergy is distributed to the computing resourcesof the data center through the UPS (Uninter-ruptible Power Supply), it is properly adaptedto the right voltage of the IT components, usingthe PDU (Power Distribution Unit). A doubleconnection to two different PDUs may also beavailable, to improve the reliability of the sys-tem. The degree of redundancy in the system,therefore, is a very important consideration inthe infrastructure. The data center should notrun out of energy so for this there are a numberof sources of supply and battery backup; themore the system is redundant, the greater thefault tolerance, but at the same time, the costsof installation and maintenance of this systemwill be higher. The IT equipment is the mainpart of a data center, which makes possible itsbasic operations, and contains the electronicsused for computing (servers), storing data (datastorage) and communicating (network). Col-lectively, these components process, store andtransmit digital information. They require a suit-able environment, where they are positionedand arranged in an optimal manner, to reduceoverheating and to prevent accidents to the ma-chines and the support staff. The servers, typ-ically, are located in 1U or blade units, rack-mounted and interconnected via local Ethernetswitches, these switches can be used in rack-level connections at 1 or 10 Gbps and have anumber of active links to one or more Ether-net switch at cluster level (or data center), thissecond level of switching can potentially covermore than ten thousand of individual servers.The IT equipment must be decided at designtime. It is very important to consider the ini-tial power and predict the future need, to makeroom for new updates or inclusion of new com-ponents and at the same time, to avoid exces-

06

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i


sive or unnecessary spending. This is just onethe things that must be taken into account, thegreatest impact on data center costs regardingmaintenance is the electricity consumption, in-cluding the one due to the cooling system: thebigger the size of the IT equipment, the greaterthe power required to cool the components. Apower and cooling analysis, also referred to asa thermal assessment, measures the relativetemperatures in specific areas of a data cen-ter, as well as the ability of the data center totolerate specific temperatures. Among otherthings, a power and cooling analysis can helpidentify hot spots, over-cooled areas that canhandle greater power use density, the break-point of equipment loading, the effectiveness ofa raised-floor strategy, and optimal equipmentpositioning (such as AC units) to balance tem-peratures across the data center. The energyconsumption of IT equipment, which performuseful work in a computer center, representon average only 50% of the total consumption,since the remainder is divided between the con-ditioning system (30%) and all the power supply(Figure 3). For energy is therefore critical thatthe share of consumption related to power andconditioning is the lowest possible, since theactivity of the computer center is representedby electronic devices. It is very important to un-derstand the values and distribution of energyflows within the computer center. The mea-sured values of the power required for variousequipments are essential for assessing the en-ergy efficiency of data centers.

POWER/ENERGY METRICS

As discussed in the previous section, currentdata centers consume a tremendous amount ofpower. Power/energy metrics of various scalesand various components for data center com-puting have been proposed and applied in datacenter computing practice. Figure 4 shows a

Figure 3:Power Consumption in a data center

summary of popular power related performancemetrics. The Data Center Infrastructure Effi-ciency, DCiE or DCE [5], [6], is an industry ac-cepted metric. It is defined as:

DCiE =ITEquipmentPowerTotalFacilityPower

where IT Equipment Powerincludes the loadassociated with all of the IT equipment, i.e.,computation, storage and network equipment,along with supplemental equipment such asswitches, monitors, and workstations/laptopsused to monitor or otherwise control the datacenter. Total Facility Powerincludes all IT Equip-ment power as described above plus everythingthat supports the IT equipment load such as:power delivery components i.e., UPS, switchgear, generators, PDUs, batteries and distribu-tion losses external to the IT equipment; Cool-ing system components i.e., chillers, computerroom air conditioning units, direct expansionair handler units, pumps, and cooling towers.Other miscellaneous component loads such asdata center lighting are also included.

A DCiE value of about 0.5 is considered typicalpractice and 0.7 and above is better practice.Some data centers are capable of achieving 0.9


07

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i

Figure 4:Power consumption metrics taxonomy

or higher [29]. An organization for green com-puting provides the following standard metricfor calculating the energy efficiency of the datacenters, Power Usage Effectiveness (PUE) [5]:

PUE =1

DCiE=

TotalFacilityPowerITEquipmentPower

PUE shows the relation between the energyused by IT equipments and energy used byother facilities, such as cooling needed for oper-ating the IT equipment. For example, a PUE of2.0 indicates that for every watt of IT power,an additional watt is consumed to cool anddistribute power to the IT equipment. At thepresent time, the PUE of a typical enterprisedata center falls between the range [1 - 3].

The HVAC (Heating, Ventilation, and Air Con-ditioning) system of a data center typically in-cludes the computer room air conditioning andventilation, a large central cooling plant, andlighting and other minor loads. The HVAC Sys-tem Effectiveness is the fraction of the IT equip-ment energy to the HVAC system energy. TheHVAC system energy is the sum of the electri-cal energy for cooling, fan movement, and anyother HVAC energy use like steam or chilledwater. The HVAC System Effectiveness is cal-culated as follows:

HV AC Effectiveness =

ITHV AC+(Fuel+Steam+ChilledWater)∗293

where IT is the annual IT Electrical Energy Use,HVAC is the annual HVAC Electrical EnergyUse, Fuel is the annual Fuel Energy Use, Steam

is the annual District Steam Energy Use and

Chilled Wateris the annual District Chilled Wa-ter Energy Use.

The HVAC System Effectiveness denotes theoverall efficiency potential for HVAC systems.A higher value of this metric means higher po-tential to reduce HVAC energy use [13].

The metric of SWaP (Space, Watts and Perfor-mance) [20] characterizes a data center’s en-ergy efficiency by introducing three parametersof space, energy and performance together.The SWaP is calculated as follows:

SWaP =Performance

(Space∗PowerConsumption)

where Performanceis computed using industry-standard benchmarks, Space measures theheight of the server in rack units (RUs) andPower Determines the watts consumed by thesystem, using data from actual benchmark runsor vendor site planning guides.

The SWaP metric gives users an effectivecross-comparison and total view of a server’soverall efficiency. Users are able to accuratelycompare the performance of different serversand determine which ones deliver the optimumperformance for their needs. The SWaP canhelp users better plan for current and futureneeds and control their data center costs.

The DCeP [14], [10] is proposed to characterizethe resource consumed for useful work in a datacenter. The DCeP is calculated as follows:

DCeP =Useful work produced

Total energy consumed to produce that work

Useful work is the tasks performed by the hard-ware within an assessment window. The calcu-lation of total energy consumed is the kWh ofthe hardware times the PUE of the facility.

MEASURING POWER CONSUMPTION

There are a variety of computer power con-sumption benchmarking techniques in practice

08

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i


today. Even though many techniques are avail-able, there is no universal one size fits alltech-nique that is appropriate for every benchmark-ing situation. Different methods of benchmark-ing must be completed for different usage pat-terns. Additionally, measuring computer powerconsumption is different than general computerbenchmarking because a tool is usually re-quired to measure how much electricity is be-ing consumed by a running machine. Generalcomputer benchmarks such as CPU or videobenchmarks do not require any special tools.Since power consumption benchmarks requireelectricity measuring devices, it makes it harderfor people to participate in the power consump-tion benchmarking area. The requirement andcost of a tool adds a burden to those who wishto run their own computer power consumptionbenchmarks. Here two electricity consump-tion meters are described, the affordable KILLA WATT meter and the more expensive Wattsup? meter. There are two fundamental waysof measuring power consumption: measuringpower consumption at one moment in time andmeasuring power consumption over time. Eachmethod has its pros and cons. Measuringpower consumption over one moment in timeis useful when measuring a device that is us-ing a constant amount of power. Less time isneeded to take a sample when measuring adevice just in a time instant, rather than whenmeasuring a device over some period of time.A disadvantage of measuring over one momentin time is that if the device being measured haspower fluctuations, it is not possible to get anaccurate average of how much power the de-vice uses. For example a refrigerator may usea lot of power when the compressor is activatedbut it may use very little power when the com-pressor is turned off. It may use a little bit morepower when the refrigerator door is opened andthe light turns on. The power consumption ofa refrigerator fluctuates over the day. In or-

der to get a better idea of how much power arefrigerator consumes, it is important to mea-sure the load over some time period that rep-resents typical refrigerator use patterns. In thecase of a refrigerator it may be necessary tomeasure a weeks worth of use in order to geta clear picture of how much power the refrig-erator consumes on average. Similarly com-puters fluctuate in their power consumption asthey carry out various tasks. When comparinga computer’s overall power usage, it is crucialto compare power usage over time [9]. Sim-ply measuring a computer’s power consump-tion when it is idling will not factor in how muchpower may be consumed while it has a loadplaced on it. A good way of comparing twosystems is to compare real tasks on each sys-tem identically from start until finish [9]. Thisreveals actual power requirements for tasks.It also exposes differences between systems.Here is an example: “Let’s say you are com-paring system A that uses 45 watts of powerunder load and system B that uses 55 wattsof power under load. If system B finishes thesame task as system A in half time (becauseit has a better performing CPU), system B isactually a more power efficient system for thebenchmarked task than system A even thoughit uses more power under load. Since it takesa lot less time to finish, less power is spent[9]”. Although Idle benchmarks can be com-pared between different reviewers and mag-azines for identical systems, the load bench-marks depend on the specific tests conductedby the reviewers. If two reviewers have identicalsystems but have different benchmarking andmeasuring schemes, the load benchmarks can-not be compared. Luckily Idle should be com-parable because in order to reach an idle state,the system has to be left doing effectively noth-ing or very little. The systems being comparedshould come installed with the same hardware,operating system, service packs, patches, soft-


09

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i

Figure 5:P3 International KILL A WATT

ware, and background processes. This wouldensure that comparisons could be made be-tween benchmarks performed on two differentsystems.

KILL A WATT Power Consumption Meter

KILL A WATT (Figure 5) is a simple meter madeby P3 International [11]. In order to use KILLA WATT, it is required to plug it into the walland then plug a device into KILL A WATT. Itcan be connected directly to the wall plug orcan be connected to an extension cable orpower strip which is then connected to the wallplug. It is easier to work with KILL A WATTwith an AC extension cable so that it could bepositioned closer to the experimenter for eas-ier viewing. KILL A WATT has several modesincluding Volt (voltage), Amp (amperes), Watt(wattage), VA (volt-amperes), Hz (hertz), PF(power factor), KWH (kilowatt hours), and Hour(the length of time the device has been con-nected to a power source). The voltage tendsto approximately 120 volts for most devices and

appliances. Some heavy appliances like elec-tric stoves and electric dryers use 240 volts ofalternating current (VAC). KILL A WATT is notable to be used with 240 VAC wall plugs. TheWatt mode displays how many watts are beingconsumed through the device at the moment oftime the wattage value is recorded. This valuetends to fluctuate with most devices as the elec-trical loads vary. The wattage is what we weremostly concerned with in this work. The Hzmode measures hertz or how many times thealternating current cycles per second (usually60 times). The PF mode displays Power Factor.The KWH mode displays the amount of kilowatthours consumed since the KILL A WATT meterwas connected to a power source. This modeis very useful in determining how much powera device or appliance uses over some time pe-riod. The last mode is the “Hour” mode, whichdisplays the amount of time the KILL A WATTmeter has been connected to a power source.Measuring power use over time is preferred be-cause this type of measurement will capture allthe fluctuations of power usage as the electri-cal load varies. Measuring power consump-tion by looking at the amounts of watts used atany given moment in time is not representativeof true power consumption of devices whoseloads vary, such as computers.

Watts Up? Power Consumption Meter TheWatts up? power consumption meter (Figure6) features the same functionality as the KILLA WATT meter and more. In addition to theKILL A WATT features, the Watts up? metercan record power measurements over time inwatt hours sensitively enough to be used forcomputer power consumption measurements.In addition to recording the amount of powerconsumed over time, the meter comes withmemory which saves the data that is recordedduring measurements. The data can then beaccessed by loading software from the watts

10

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i


Figure 6:Watts up? Power Consumption Meter

up? website [12] and connecting the meter toa Windows computer with an included USB ca-ble. There are several Watts up? meter modelsavailable. Among them the Watts up? PRO esmodel, which includes extra memory for savingmeasurements that are taken over long periodsof time. The Watts up? meter has several ad-vantages over the KILL A WATT meter. It hasmore complex features than the KILL A WATTmeter such as more sensitive measurementsand memory for data storage. It can connectto a Windows machine with a USB cable andtransfer data which can then be graphed andused for analysis. It has an extension cablethat gives the user a lot of leeway to positionthe meter in a viewable spot. The watts up?power consumption meter’s functionality madeit a great tool for professional power consump-tion benchmarking. There are also several im-portant disadvantages to note about the Wattsup? meter. The Watts up? meter costs severaltimes more money than the KILL A WATT me-

ter, around 90-220 dollars as of the time of thiswriting. The Watts Up? meters are more com-plicated and might be harder for some people tolearn how to use. Despite these drawbacks thewatts up? meter proved to be a better tool forour research in order to carry out serious com-puter power consumption benchmarks. With-out a watt hour measuring option on a powermeter, it is not possible to accurately measurehow much power a computer uses when itspower fluctuates unless the benchmarks lastsfor hours or days.

COMPUTER POWER CONSUMPTIONBENCHMARKING

One area of the computing sciences that is be-coming more and more important is computerpower consumption benchmarking. Bench-marking is a general and widely known ap-proach where a standard load, or benchmark,is used to measure some desired aspects, be-havior or other characteristics of the thing beingobserved. In computing, benchmarks are typi-cally computer programs that are run on a sys-tem, enabling accurate and repeatable mea-surement of some specific characteristic of in-terest. A performance metric is typically as-sociated with a benchmark or a workload, aswell as a well-defined execution environment. Abenchmark is used to emulate real-world com-puting tasks. A data center benchmark is theact of running a set of programs, or other op-erations, in order to assess the relative per-formance of data center performance metrics.Typically data center benchmarking is carriedout under a certain workload, which is artifi-cially generated or produced in real life, accord-ing to the following workflow: real or artificialworkload → benchmark → performance met-rics. Although some benchmark workloads areavailable, for example, server-side Java undervarious loads for SPECpower [34], JouleSort


11

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i

[39], data center benchmarking is generallytaken under normal data center practice andworkload. In this section, two types of bench-marks are described: server level benchmarksand data center level benchmarks. Server levelbenchmarks, like GCPI, Green500 and Joule-Sort, normally measure energy consumption ina compute server or a cluster. Data center levelbenchmarks, such as, HVAC Effectiveness In-dex and Computer Power Consumption Indexfrom LBNL, present a measurement for the en-tire data center.

Computer server level benchmark This sec-tion introduces several performance metrics forcomputer server level performance evaluation:GCPI, SPECpower, Green500 project, andJouleSort. SiCortex [19] proposes an inclu-sive metric for comparing energy efficiency inthe High-Productivity Computing segment. TheGreen Computing Performance Index (GCPI)[15] is an inclusive metric for comparing energyefficiency in High-Productivity Computing seg-ment proposed by SiCortex [19]. The GCPIanalyzes computing performance per kWattacross a spectrum of industry-standard bench-marks, providing organizations with much-needed guidance in the era of out-of-controldata center energy consumption. It has beendeclared that the GCPI is the first index tomeasure, analyze and rank computers on abroad range of performance metrics relative toenergy consumed. In 2006, the SPEC com-munity started to establish SPECpower [34],an initiative to augment existing SPEC bench-marks with a power/energy measurements.SPECpower’s workload is a Java applicationthat generates and completes a mix of trans-actions; the reported throughput is the num-ber of transactions completed per second overa fixed period. SPECpower reports the en-ergy efficiency in terms of overall operationsper watt. This metric represents the sum of

the performance measured at each target loadlevel divided by the sum of the average power(in watts) at each target load. The Green500project [26] aims at increasing awareness aboutthe environmental impact and long-term sus-tainability of high-end supercomputing by pro-viding a ranking of the most energy-efficient su-percomputers in the world. The Green500 listuses performance per watt(PPW) as its met-ric to rank the energy efficiency of supercom-puters. The performance is defined as theachieved maximum GFLOPS (Giga FLoating-point OPerations per Second) performance, bythe Linpack [16] benchmark on the entire sys-tem. The power is defined as the average sys-tem power consumption during the executionof Linpack with a defined problem size. Joule-Sort [39] is an I/O-centric benchmark that mea-sures the energy efficiency of systems at peakuse. It is an extension of the sort benchmark,which is used to measure the performance andcost-performance of computer systems. TheJouleSort benchmark measures the cost of do-ing some amount of work, which reflects somemeasure of power use, e.g., average power,peak power, total energy, and energy-delay.

Data center level benchmarking Data centerlevel benchmarks are used to evaluate howgreen a data center is and are also used to com-pare similar data centers. Lawrence BerkeleyNational Laboratory (LBNL) releases a set of ef-forts and practices for data center benchmark-ing [3]. These practices include improved airmanagement, emphasizing control and isola-tion of hot and cold air streams; rightsizing cen-tral plants and ventilation systems to operateefficiently both at inception and as the data cen-ter load increases over time; optimizing centralchiller plants designed and controlled to maxi-mize overall cooling plant efficiency, central air-handling units in lieu of distributed units. Figure7 shows the formal process for benchmarking

12

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i


Figure 7:Data center benchmarking process

a data center [13]. Main software packagesfor benchmarking and designing energy effi-cient buildings and data centers are: Calarch,Comis, DoE-2, EnergyPlus, Genopt, home en-ergy saver.

CLUSTER POWER MANAGEMENTALGORITHMS

Two basic power management mechanismsare dynamic voltage scaling and node vary-on/vary-off. In the following these mechanismsare briefly described. Also hybrid approachesthat combine these techniques have been in-vestigated.

Dynamic Voltage and Frequency ScalingOnetechnique being explored is the use of DynamicVoltage and Frequency Scaling (DVFS) withinClusters and Supercomputers [30], [31]. Byusing DVFS one can lower the operating fre-quency and voltage, which results in decreasedpower consumption of a given computing re-source considerably. High-end computing com-munities such as cluster computing and super-computing in large data centers, have appliedDVFS techniques to reduce power consumptionand achieve high reliability and availability [28],[24], [8]. A power-aware cluster is defined asa compute cluster where compute nodes sup-port multiple power/performance modes, for ex-ample, processors with frequencies that canbe turned up or down. This technique wasoriginally used in portable and laptop systemsto conserve battery power, and has since mi-grated to the latest server chipsets. Current

technologies exist within the CPU market suchas Intel’s SpeedStep and AMD’s PowerNow!technologies. These dynamically raise andlower both frequency and CPU voltage usingACPI P-states [2]. In [25], DVFS techniques areused to scale down the frequency by 400Mhzwhile sustaining only a 5% performance loss,resulting in a 20% reduction in power. A power-aware cluster supports multiple power and per-formance modes, allowing for the creation ofan efficient scheduling system that minimizespower consumption of a system while attempt-ing to maximize performance. When lookingto create a DVFS scheduling system for a datacenter, there are a few rules of thumb to builda scheduling algorithm which schedules vir-tual machines in a cluster while minimizing thepower consumption:

1. Minimize the processor supply voltage byscaling down the processor frequency;

2. Schedule virtual machines to processingelements with low voltages and try not toscale Processing Elements (PEs) to highvoltages.

Based on the performance model definedabove, Rule 1 is obvious as the power con-sumption could be reduced when supplied volt-ages are minimized. Then Rule 2 is applied:by scheduling virtual machines to processingelements with low voltages and trying not tooperate PEs with high voltages to support vir-tual machines. A scheduling algorithm for vir-tual machines in a DVFS-enabled cluster [45]is shown in Figure 8 where incoming virtualmachine requests arrive at the cluster and areplaced in a sorted queue. This system hasbeen modeled, created and described in [44];in this Section we will explain in detail thatscheduling mechanism. The scheduling algo-rithm runs as a daemon in a cluster with a pre-defined schedule interval, INTERVAL. During


13

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i

Figure 8:Working scenario of a DVFS-enabled cluster scheduling

the scheduling interval, incoming virtual ma-chines arrive at the scheduler and will be sched-uled at the next schedule round. For each vir-tual machine (VM), the algorithm checks the PEoperating point set from low voltage level to highvoltage level. The PE with lowest possible volt-age level is found; if this PE can fulfill the virtualmachine requirement, the VM can be sched-uled on this PE. If no PE can schedule the VM,a PE must be selected to operate with highervoltage. This is done using the processor withthe highest potential processor speed, which isthen adjusted to run at the lowest speed thatfulfills the VM requirement, and the VM is thenscheduled on that PE. After one schedule inter-val elapses, some virtual machines may havefinished their execution; thus this algorithm at-tempts to reduce a number of PE’s supply volt-ages if they are not fully utilized.

Vary-on/Vary-off Node vary-on/vary-off(VOVO) takes whole nodes offline when theworkload can be adequately served by asubset of the nodes in the cluster. Machinesthat are taken off-line may be powered off orplaced in a low power state. Machines thatare off-lined are placed back online should theworkload increase. Node varyon/vary-off is aninter-node power management mechanism.

This policy, originally proposed by Pinheiro et.al. [37], turns off server nodes so that onlythe minimum number of servers required tosupport the workload are kept active. Nodesare brought online as and when required.VOVO does not use any intra-node voltagescaling, and can therefore be implemented ina cluster that uses standard high-performanceprocessors without dynamic voltage scaling.However, some hardware support, such as aWake-On-LAN network interface, is neededto signal a server to transition from inactiveto active state. Node vary-on/vary-off can beimplemented as a software service runningon one of the cluster nodes (or a separatesupport server) to determine whether nodesmust be taken offline or brought online andrequires software probes running on each ofthe cluster nodes to provide information on theutilization of that node. The load distributionmechanism must be aware of the state of theservers in the cluster so that it does not directrequests to inactive nodes, or to nodes thathave been selected for vary-off but have notyet completed all received requests.

Given the features of the Euro-MediterraneanCentre for Climate Change (CMCC) clusters,at the moment it is not possible to apply DVFSalgorithms. So, several Vary-on/Vary-off mech-anisms will be investigated.

A PROPOSAL FOR A METERINGSYSTEM FOR THE POWERCONSUMPTION OF CALYPSO

The Supercomputing infrastructure provided byCMCC consists of two cluster systems, whichtogether reach a theoretical peak performanceof more than 30 TFlops. These systems arecomplemented by an high-capacity and high-performance storage infrastructure that, over-all, provides a useful storage space, whichamounts to more than 450 Tbytes. The two

14

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i


supercomputing clusters differ in the type of ar-chitecture: the first one is a vector system con-sisting of NEC SX-8R series nodes and SX-9nodes, while the second one is a scalar systemconsisting of 30 IBM POWER 6 nodes dividedinto 3 frames. Each node contains 16 Power 6dual-core processors, operating at a frequencyof 4.7 GHz. Therefore, the number of physicalcores amounts to 960 cores, capable of provid-ing a total computing power (theoretical peakperformance) of about 18 TFLOPS. To ensurethe continuity of the power to such systems -and the consequent preservation of data - theCMCC has a 500KVA Static UPS power sys-tem.

The power panel is already prepared for a sec-ond parallel UPS able to provide total protec-tion for 1000kVA for future additions (Figure 9).The CMCC has a dedicated transformer sub-station from which three lines of service reachthe CMCC. A 30 kW line is dedicated to theordinary user, another of 350 kW is dedicatedto users of air conditioning, and the third one -the most important - is the preferred line ded-icated to computer systems. The presence ofa single supply line and the presence of somecomponents of redundant cooling allow classi-fying the CMCC Supercomputing Center as aTier II data center. Inside the CMCC there aretwo main areas (Figure 10) well separated andwith well defined roles.

In the first environment called Control Room, allthe hosted electrical panels supply power tovarious power lines of computer systems, stor-age and services. The system cooling is lo-cated in the second environment (greater than200 square meters), called Computing Room. In-side the Control Room there are several electri-cal panels whose main task is to distribute elec-trical power to computer systems and storage.There are three main power supply lines: thepower supply line of NEC systems, the power

Figure 9:Diagram of the electrical connection between the ENEL

cabinet and CMCC through UPS

supply line of IBM systems and, finally, a sup-ply line dedicated to services. The main controlpanels of each line are equipped with meterswhich record the total consumption. RegardingCalypso the power of each individual frame ismanaged by four three-phase lines (12 in total)(Figure 10).

Since we intend to analyze the energy con-sumption of Calypso, a possibility is to performthe metering of the consumption of each IBM(frame) rack by measuring the three-phase sup-ply. Every IBM frame provides 4-phase electri-cal connections that feed to 12 PDUs. ThesePDUs in turn feed the IBM P575 computingnodes in addition to two control units. Hence, itis possible to place a power panel in the Com-puting Room for measuring the energy con-sumption of each of the 4 lines per frame, thusobtaining a measurement of the total consump-tion of each IBM Calypso rack (Figure 11).


15

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i

Uninterruptible

Power Supply IBM Panel

Figure 10:Scheme of electrical connection between the control

room and Calypso

Uninterruptible

Power SupplyIBM Panel

Figure 11:Scheme of electrical connection between the control

room and Calypso with insertion of a panel dedicated tomeasure the power consumption of each frame

16

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i


Bibliography

[1] High Performance Green Comput-ing, A Proposal for New Investmentsin Faculty Hiring in the Departmentsof Astronomy and Computer Science.http://www.umass.edu/oapa/oapa/proposals/high performance green computing.pdf.

[2] Data Center Power Management and Ben-efits to Modular Computing, 2003. IntelDeveloper Forum.

[3] Best practices for data centers: lessonslearned from benchmarking 22 data cen-ters, 2006. Technical report, LawrenceBerkeley National Laboratory.

[4] Estimating total power consumption byservers in the US and the world, 2007.Tech. rep., Lawrence Berkeley NationalLaboratory.

[5] The green grid data center efficiency met-rics: PUE and DCIE, 2007. Technical re-port, The Green Grid.

[6] The Green Grid metrics: data centerinfrastructure efficiency (DCiE) detailedanalysis, 2007. Technical report, TheGreen Grid.

[7] The green grids opportunity: decreasingdatacenter and other IT energy usage pat-terns, 2007. Technical report, the greengrid.

[8] The Green500 List: Encouraging Sustain-able Supercomputing, 2007. IEEE Com-puter Society.

[9] The Truth About PC Power Consumption,2007. Tom’s Hardware, Bestofmedia Net-work.

[10] A framework for data center energy pro-ductivity, 2008. Technical report, TheGreenGrid.

[11] P3 - Kill A Watt, 2008.

[12] Watts Up? Products: Meters, ElectronicEducational Devices, 2008.

[13] Self-benchmarking guide for data centers:metrics, benchmarks, actions, 2009. Tech-nical report, Lawrence Berkeley NationalLaboratory, Berkeley, California.

[14] 42U DCeP: Data Center energy Produc-tivity, 2011.

[15] Green Computing Per-formance Index, 2011.http://www.sicortex.com/green index.

[16] Green Computing Performance Index,2011. http://www.netlib.org/linpack/.

[17] IEEE TCSC technical area of green com-puting, 2011.

[18] Report to Congress on Server andData Center Energy Efficiency, 2011.www.energystar.gov/ia/partners/prod development/downloads/EPA Datacenter Report Congress Final1.pdf.

[19] SiCortex, 2011. http://www.sicortex.com/.

[20] SWaP (Space, Watts and Performance)Metric, 2011.

[21] L. A. Barroso and U. Holzle. The Dat-acenter as a Computer: An Introductionto the Design of Warehouse-Scale Ma-chines. Synthesis Lectures on Computer Ar-

chitecture, 1(1):1–108, 2009.

[22] Y Chen, A Das, W Qin, A Sivasubrama-niam, Q Wang, and N Gautam. Manag-ing server energy and operational costs inhosting centers. In SIGMETRICS’05: pro-

ceedings of the 2005 ACM SIGMETRICS inter-

national conference on measurement and mod-

eling of computer systems, pages 303–314,2005.

[23] R Das, JO Kephart, C Lefurgy, G Tesauro,DW Levine, and H Chan. Autonomic multi-agent management of power and perfor-mance in data centers. In AAMASO08:


17

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i

proceedings of the 7th international joint con-

ference on autonomous agents and multiagent

systems, pages 107–114, 2008.

[24] W. Feng, A. Ching, C.H. Hsu, and V. Tech.Green Supercomputing in a Desktop Box.In IIn IEEE International Parallel and Dis-

tributed Processing Symposium, pages 1–8,2007.

[25] W. Feng, X. Feng, and R. Ge. Green Su-percomputing Comes of Age. IT PROFES-

SIONA, 10:17, 2008.

[26] WC Feng and KW Cameron. TheGreen500 list: encouraging sustainablesupercomputing. Computer, 40:50–55,2007.

[27] VW Freeh and DK Lowenthal. Using mul-tiple energy gears in mpi programs on apower-scalable cluster. In Pingali K, Yelick

KA, Grimshaw AS (eds) PPOPP. ACM Press,

New York, pages 164–173, 2005.

[28] I. Gorton, P Greeeld, A Szalay, andR. Williams. Data-Intensive Computingin the 21st Century. Computer, 41:30–32,2008.

[29] S Greenberg, A. Khanna, and W Tschudi.High performance computing with high ef-ficiency. AmSoc Heat Refrig Air-cond Eng

(ASHRAE) Trans, page 179, 2009.

[30] Chung hsing Hsu and Wu chun Feng. Afeasibility analysis of power awareness incommodity-based high-performance clus-ters. In In Proceedings of IEEE International

Conference on Cluster Computing, pages 1–10, 2005.

[31] C. Hsu and W. Feng. A power-aware run-time system for high-performance comput-ing. In In Proceedings of the 2005 ACM/IEEE

conference on Supercomputing, 2005.

[32] N Jiang and M Parashar. Enabling au-tonomic power-aware management of in-strumented data centers. In IPDPSO09:

proceedings of the 2009 IEEE international

symposium on parallel & distributed process-

ing, pages 1–8, 2009.

[33] G Jung, KR Joshi, MA Hiltunen,RD Schlichting, and C Pu. A cost-sensitive adaptation engine for server con-solidation of multitier applications. InACM/IFIP/USENIX 10th international mid-

dleware conference, pages 163–183, 2009.

[34] KD Lange. Identifying shades of green:the specpower benchmarks. Computer,42:95–97, 2009.

[35] S Mehta and A Neogi. Recon: a toolto recommend dynamic server consoli-dation in multi-cluster data centers. InIEEE/IFIP network operations and manage-

ment symposium: pervasive management for

ubioquitous networks and services, pages363–370, 2008.

[36] CD Patel, R Sharma, CE Bash, and A Beit-elmal. Thermal considerations in coolinglarge scale high compute density data cen-ters. In Proceedins of Thermal and thermo-

mechanical phenomena in electronic systems -

ITHERM 2002, pages 767–776, 2002.

[37] E. Pinheiro, R. Bianchini, E.V. Carrera, andT. Heath. Load Balancing and Unbalanc-ing for Power and Performance in Cluster-Based Systems. In In Workshop on Com-

pilers and Operating Systems for Low Power,pages 180–195, 2001.

[38] S. Rivoire, MA Shah, P Ranganathan, andJ Meza C. Kozyrakis. Models and metricsto enable energy-efficiency optimizations.Computer, 40(12):39–48, 2007.

[39] S Rivoire, MA Shah, P Ranganathan,and C Kozyrakisr. Joulesort: a bal-anced energy-efficiency benchmark. InSIGMOD’07: proceedings of the ACM SIG-

MOD international conference on manage-

ment of data, pages 365–376, 2007.

18

Cen

troE

uro-

Med

iterr

aneo

per

iCam

biam

enti

Clim

atic

i


[40] A Shah and N Krishnan. Optimizationof global data center thermal manage-ment workload for minimal environmentaland economic burden. IEEE Transactions

on Components and Packaging Technologies,31(1):39–45, 2008.

[41] A Singh, MR Korupolu, and A Mohapa-tra. Server-storage virtualization: integra-tion and load balancing in data centers. InProceedings of the ACM/IEEE conference on

high performance computing, page 53, 2008.

[42] Q Tang, SKS Gupta, and G Varsamopou-los. Energy-efficient thermal-awaretask scheduling for homogeneous high-performance computing data centers: acyber-physical approach. IEEE Trans Par-

allel Distrib Syst, 19(11):1458–1472, 2008.

[43] DC Vanderster, A Baniasadi, and NJ Di-mopoulos. Exploiting task tempera-ture profiling in temperature-aware taskscheduling for computational clusters. InAsia-Pacific computer systems architecture

conference, pages 175–185, 2007.

[44] Gregor von Laszewski, Lizhe Wang, An-drew J. Younge, and Xi He. Power-AwareScheduling of Virtual Machines in DVFS-enabled Clusters. In IEEE Cluster 2009,2009.

[45] Andrew J Younge, Gregor Von Laszewski,Lizhe Wang, and Geoffrey C Fox. Provid-ing a Green Framework for Cloud BasedData Centers. In The Handbook of Ener-

gyAware Green Computing, 2011.

© Centro Euro-Mediterraneo per i Cambiamenti Climatici 2012

Visit www.cmcc.it for information on our activities and publications.

The Euro-Mediteranean Centre for Climate Change is a Ltd Company with its registered office andadministration in Lecce and local units in Bologna, Venice, Capua, Sassari and Milan. The societydoesn’t pursue profitable ends and aims to realize and manage the Centre, its promotion, and researchcoordination and different scientific and applied activities in the field of climate change study.

issue december 2011 in hpc data centers - cmcc · 2006. it is reported that power and cooling costs...

Documents