green computing - jyväskylän yliopistousers.jyu.fi/~riesta/green_computing.pdf · introduction...

Green Computing

Johan Lilius

January 19, 2012

Johan Lilius Green Computing 1/67

Introduction

Contents

1 Introduction

2 The economics of Data-Centers: Why Power Matters

3 Energy-proportional computing

4 Servers built on mobile processors

5 Summary

6 Bibliography


Introduction

What is Green-Computing (Green-ICT)?

Green Computing refer to 2 different things1 Reducing energy consumption of ICT2 Using ICT to reduce energy consumption

Goal: reduce carbon footprintThis presentation : Reducing energy consumption of ICT


Introduction

Green-Computing aspects [11]

Start1992 EPA Energy star rating

5 issues (2009)1 E-Waste2 Data-centers and Servers3 PCs, Monitors and Workstations4 Software5 Telecommuting


Introduction

E-Waste

Recycling issueestimates that over 25 billion computers, televisions, cellphones, printers, gaming systems, and other deviceshave been sold since 1980,2 million tons of unwanted electronic devices in 2005alone,with only 15 to 20 percent being recycled

Material is transfered to developing countriesClear environmental hazards


Introduction

Data centers & Servers

Power consumption is an issue

Energy efficiency

•  Power draw of computing clusters is becoming an increasing fraction of their cost1

•  The density of the datacenters that house them is in turn limited by their ability to supply and cool 10–20 kW of power per rack and up to 10–20 MW per datacenter

•  Future datacenters may require as much as 200 MW, and datacenters are being constructed today with dedicated electrical substations to feed them.

•  1Kenneth G. Brill: “The Invisible Crisis in the Data Center: The Economic Meltdown of Moore's Law” Uptime Institute, 2009

9/3/10& 2&

Driver money, not environmental aspects :-(


Introduction

PCs, Monitors, Workstations

5th upgarde Energy Star requirements for monitors (1April 2009)Requires a 20 percent increase in electrical efficiency.Estimate if all monitors comply,

saving of roughly $1 billion per year in energy expensesandavoid GHG emissions equivalent to 1.5 million cars


Introduction

Software

Cost of Spam (McAfee, 2008)62 trillion spam messages in 20080.3 grams of carbon dioxide (CO2) per messageannual spam energy use 33 terawatt hours (tWh)equivalent to the electricity used in 2.4 million homesevery year, with thesame GHG emissions as 3.1 million automobiles usingtwo billion US gallons of gasoline

How we use the computing systems influences powerconsumption


Introduction

Telecommuting

An ITIF report found that, if only 14 percent of existingAmerican office jobs were converted to work-from-home jobs,the savings would be dramatic: estimated at 136 billionvehicle travel miles annually in the US by 2020 and 171billion miles by 2030


Introduction

This presentation

How to reduce energy consumption in datacentersCentral concept: Energy Proportionality

Use only as much computation power that is needed forthe task at hand

Still a lot of R&D to do!


The economics of Data-Centers: Why Power Matters

Contents

1 Introduction




5 Summary

6 Bibliography



Energy cost in data-centers I

Power draw of computing clusters is becoming anincreasing fraction of their cost

Energy efficiency

•  Power draw of computing clusters is becoming an increasing fraction of their cost1

•  The density of the datacenters that house them is in turn limited by their ability to supply and cool 10–20 kW of power per rack and up to 10–20 MW per datacenter

•  Future datacenters may require as much as 200 MW, and datacenters are being constructed today with dedicated electrical substations to feed them.

•  1Kenneth G. Brill: “The Invisible Crisis in the Data Center: The Economic Meltdown of Moore's Law” Uptime Institute, 2009

9/3/10& 2&

The density of the datacenters that house them is inturn limited by their ability to supply and cool 10–20 kWof power per rack and up to 10–20 MW per datacenterFuture datacenters require as much as >100 MW, anddatacenters are being constructed today with dedicatedelectrical substations to feed them.

Facebook, Luleå, budget 120MW



Energy cost in data-centers II

Exascale computing roadmap, budgets of >100MW forsupercomputing facilities

Nuclear powerplants:Loviisa reactors 488MW nominal powerOlkiluoto new reactor 1600 MW nominal power



Cost breakdown for data-centers I

A $200M facility capable of delivering 15MW of criticalload (server power)

Facility: ˜$200M for 15MW DC (15 yr Amortization)Servers: ˜$2k/each, roughly 50,000 (3 yr Amortization)Commercial Power: ˜$0.07/kWh5% cost of money



Cost breakdown for data-centers II

2

reports suggest that people costs dominate [5]. People costs often are dominant in enterprise data centers, however, in high-scale facilities with tens of thousands of servers, server administration is heavily automated [10] and, once it has been, administration costs fall below 10% and often below 5%.

In order of magnitude from largest first, the most significant costs are 1) server acquisition, 2) cooling, 3) power distribution, and 4) power itself.

To understand this data in more detail, we model a $200M facility capable of delivering 15MW of critical load (server power) with the following assumptions:

• Facility: ~$200M for 15MW DC (15 yr Amortization) • Servers: ~$2k/each, roughly 50,000 (3 yr Amortization) • Commercial Power: ~$0.07/kWh • 5% cost of money

To compare these costs, we need to normalize long lived capital costs with 15-year amortization periods and short lived capital costs having 3-year amortization periods. In addition we need to compare monthly operational costs with these capital costs in order to be able to understand which are the most important. We normalize by assuming a 5% annual cost of money with monthly payments and essentially borrow the money for capital expenses and pay back monthly over the assumed life and amortization period of the equipment. This converts the capital expenses to effective cost per month. And, by considering amortization periods, we normalize long lived and short lived capital and recognize each appropriately. In this model, land, taxes, security and administration are not included due to their relatively small contribution to overall costs.

Figure 1: Monthly Server, Power, and Infrastructure Costs Figure 1 shows that power costs are much lower than infrastructure costs, and also much less than the servers themselves. Servers are the dominant cost, but, before we conclude that power is only 23% of the total, it’s worth looking more closely. Infrastructure includes the building, power distribution, and cooling. Power distribution and cooling make up 82% of the costs of infrastructure [2] with the building itself down in the 12-15% range. Power distribution is functionally related to the power consumed in that sufficient power distribution equipment is required to distribute the maximum amount of power consumed. Cooling is also functionally related to power in that

the heat from all power that is dissipated in the building must be removed. The vast majority of the infrastructure cost is functionally related to power. We define the fully burdened cost of power to be the sum of power, power distribution, and cooling costs.

3. Where Does the Power Go? To get started, it helps to define a few terms. Total Facility Power is the power delivered by the utility to the property line of the data center. IT Equipment Power is the power delivered to the critical load, the servers in the data center. The difference between Total Facility Power and IT Equipment Power is the power lost in power distribution and in cooling the facility. Effectively, this difference is the facility infrastructure overhead.

The Green Grid defines two useful terms when looking at data center efficiency: Power Usage Effectiveness (PUE) and Data Center Infrastructure Effectiveness (DCiE) [9].

PUE = (Total Power) / (IT Equip. Power) DCiE = (IT Equip. Power) / (Total Power) * 100%

Power Usage Effectiveness is Total Facility Power over IT Equipment power. The PUE tells us how many watts must be delivered to the data center in order to get one watt to the critical load, the servers themselves. DCiE is the reciprocal of PUE and is defined as IT Equipment Power over Total facility power. DCiE tells us what percentage of the power delivered to the facility actually gets delivered to the servers.

These terms both have the same information content. A PUE of 1.7 states that for every watt delivered to the IT equipment (the servers), we dissipate 0.7W in power distribution and mechanical systems (air conditioning, pumps, fans, etc.). A PUE of 1.7 is the same as a DCiE of 59% which states that for every watt delivered to the facility, 59% is delivered to the IT equipment. The DCiE also tells us that 41% of the power delivered to the data center is lost in power distribution and cooling overhead.

PUEs vary greatly. Very inefficient enterprise facilities are often as low as 2.0 or even 3.0 [9] and unusual, industry-leading facilities are being advertised as better than 1.2 [8]. These latter reports, however, are difficult to corroborate.

In this exploration into power losses, we’ll consider a current-generation facility. This is one that would be built if current, well understood techniques are applied and good quality but widely available equipment is deployed. This test facility has a PUE of 1.7, putting it much better than most of the world’s data centers. But it is not using some of the latest, not yet well documented innovations. A PUE of 1.7 is far above average but lower than the best and forms a good baseline for us to look at to understand where the power is going and where the largest inefficiencies lie.

Looking more deeply at our PUE 1.7 facility, we know by the definition of PUE that we are delivering 59% of the data center power to the IT equipment. We need to understand where the remaining 41% is going.

To understand where the 41% lost to data center infrastructure is going, we look first to the power distribution equipment since it is both easier to inventory and these distribution loses are easier to track. Looking at figure 2, we can see every conversion and the

Infrastructure costs are mostly power related50% of costs are power related



Measures (PUE, DCiE) I

PUE: Power Usage Effectivenes

PUE =TotalPower

ITEquip.Power

DCie: Data Center Infrastructure Effectiveness

DCiE =ITEquip.Power

TotalPower

⇤ 100%

Google average Q3 2011: PUE 1.16CSC Kajaani data-center goal: PUE 1.15PUE is highly dependent on outside temperature



Measures (PUE, DCiE) II

Nordic countries are interestingGoogle: SummaFacebook: Luleå

Added advantage: green powerFacebook datacenter will be powered by hydropowerDiesel only as backup

Other ideas:Use feed excess heat into city-wide heating system(Gaudeamus datacenter, Helsinki)

NOTE PUE is a measure on the effectiveness of thecooling, not of the energy-efficiency of the computing!!!Quite a lot remains to be done to cut down the totalpower consumption of the system



Energy losses in Power Distribution

3

efficiency of each conversation from the power delivered by the utility at 115,000V through to deliver to the servers at 208V.

Starting at the upper left corner of Figure 2, we see the utility delivers us 115kV and we first step it down to 13.2kv. The 13.2kv feed is delivered to the Uninterruptable Power Supply (UPS). In this case we use a battery-based UPS system, but rotary systems are also common. This particular battery-based UPS is 94% efficient, taking all current through rectifiers to direct current and then inverting it all back to AC. Rotary designs are usually more efficient than the example shown here and bypass designs can exceed 97% efficiency. In this example, a non-bypass UPS installation, all power flowing to UPS protected equipment (the servers and most of the mechanical systems) is first rectified to DC and then inverted back to AC. All the power destined to the servers flows through these two conversions steps whether or not there is a power failure, and these two conversion steps contribute the bulk of the losses, bringing down the UPS efficiency to 94%. More efficient bypass UPSs avoid these losses by routing most power “around” the UPS in the common, non-power failure case.

Figure 2: Power Distribution For longer term power outages, there are usually generators to keep the facility operational. The generation system introduces essentially no additional losses when not being used but they greatly increase the capital expense with a 2.5MW generator

pricing out at more than $2M. Most facilities will have at least 1 extra generator (N+1) and many facilities will have 2 spares (N+2) allowing one to be in maintenance, one to fail on startup and still to be able to run the facility at full load during a power failure. A 2.5MW generator will burn just under 180 gallons/hour of diesel so environmentally conscious operators work hard to minimize their generator time. And the storage of well over 100,000 gallons of diesel at the facility brings additional cost, storage space, insurance risk, and maintenance issues.

After the UPS, we step down the 13.2kV voltage to 480V and then that is further stepped down to 208V for distribution to the critical load, the servers. In this facility, we are using very high quality transformers, so we experience losses of only 0.3% at each transformer. We estimate that we lose a further 1% in switch gear and conductor losses throughout the facility.

In summary, we have three 99.7% efficient transformers, a 94% efficient UPS and 1% losses in distribution for an overall power distribution loss of 8% (0.997^3*0.94*0.99 => 0.922).

We know we deliver 59% of the facility power to the critical load and, from the electrical distribution system analysis above, we know we lose 8% of total power to power distribution losses. By subtraction, we have 33% lost to mechanical systems responsible for data center cooling.



AC vs DC?Ishikari Datacenter in Ishikari City, Hokkaido JapanUse High-Voltage (400V) DC power as much as possible



Energy losses in Mechanical Systems

4

Figure 3: Mechanical Systems Several observations emerge from this summary. The first is that power distribution is already fairly efficient. Taking the 8% efficiency number down to 4% to 5% by using a 97% efficient UPS and eliminating 1 layer of power conversion is an easy improvement. Further reductions in power distribution losses are possible but the positive impact can’t exceed 8% so we’re better rewarded looking to improvements in the mechanical systems, where we are spending 33% of the power, and in the servers, where we are dissipating 59% of the power. The CEMS project focused on the latter, increasing the efficiency of the servers themselves.

4. CEMS Introduction From the previous section, we understand that 59% of the power dissipated in a high-scale data center is delivered to the critical load. Generally that is a good thing in that power delivered to the servers is success from a data center infrastructure perspective. All power delivered to the server has a chance of actually getting work done. However, with the bulk of the power going to servers, server efficiency and utilization clearly will have a substantial impact on overall data center power efficiency. In this work, we focus on the former, server efficiency. The CEMS project originated from two core observations: 1) nearly 60% of the power delivered to a high-scale data center is delivered to servers, so server efficiency has a dominant impact on overall system (data center and server) efficiency, and 2) newer servers design are increasingly out of balance as CPU performance increases without matching improvements in memory and storage subsystems. Let’s look first at the balance issue in more detail and then come back to how to leverage these two observations to deliver a reliable service while substantial lowering costs and increasing power utilization efficiency.

4.1 System Balance Looking back 25 years, we have experienced steady improvement in CPU performance and, for a given algorithm, increased performance generally requires increased data rates. In the high performance computing world, this is reported in bytes/FLOP but it’s just as relevant in the commercial processing world. More CPU performance requires more memory bandwidth to get value from that increase in performance. Otherwise, the faster processor just spends more time in memory stalls and doesn’t actually get more work done. For the bulk of the last 25 years, CPU performance improvements have been driven by design improvements and clock frequency increases. Having hit the power wall, we’re now less reliant on clock frequency improvements than in the past and more dependent upon increases in core counts. But the net is that processor performance continues to grow unabated and this is expected to continue. Looking at the first row of Table 1, from Dave Patterson’s “Latency Lags Bandwidth” paper [16], we can quantify the argument above. The data in Table 1 are extracted from leading commodity components over the last 25 years and what is reported is the multiplicative performance increase per year. Looking at this chart, we see that CPU bandwidth is growing at 1.5x per year whereas memory bandwidth, LAN bandwidth, and disk bandwidth are all growing more slowly. Table 1. Annual Bandwidth and Latency Improvements

Annual Improvement CPU DRAM LAN Disk

Bandwidth 1.5 1.3 1.4 1.3

Latency 1.2 1.1 1.1 1.1



Power losses in Servers

Electricity consumption in a typical data center

20%

25%

30%

35%

40%

45%

50%

Po

we

r p

er

co

mp

on

en

t

40%

50%

60%

70%

80%

90%

100%

Cu

mu

lati

ve

po

we

r

0%

5%

10%

15%

Load PSU Chiller UPS VRs Server

fans

CRAC fan PDU CW pump Total

baseline

Po

we

r p

er

co

mp

on

en

t

0%

10%

20%

30% Cu

mu

lati

ve

po

we

r

Source: Intel Systems Technology Lab 2008



Heat density I

Power dissipated / per unit areaChip-level issueBuilding-level issue

Chip level 40W/cm2

Shuttle re-entry 100W/cm2

Building levelBlade server > 5kVAFinnish sauna ˜6kVAMost dense datacom products > 9kVA

One cannot pack servers into a small spaceCooling & Air Handling Gains

2009/4/1

• Tighter control of air-flow increased delta-T• Container takes one step further with very little air in motion, variable speed fans, & tight feedback between CRAC and load• Sealed enclosure allows elimination of small, inefficient (6 to 9W each) server fans 14

Intel

Intel

Verari

http://perspectives.mvdirona.com



Processor

Issues:1 Where does processor power go?2 What can we do about it?



Processor Power I

22% yearly increase



Processor Power II

Concrete values for AMD and Intel



Processor Power III

Cooler in 1993 and in 2005



Processor Power IV

How far can you go?AMD FX Processor Takes Guiness World Record

http://bit.ly/ux5ORA



Power budget of processor

Exact numbers difficult if impossible to obtainIntel Penryn (2007) uses 50% of silicon for L2/L3 cachesMeasurement on ARM shows that memory subsystemuses 50% of total power



Processor power

Power as function of voltage and frequency

P = c ⇤ V 2 ⇤ f

c is activity factorWhat can you do?

1 Use techniques to shutdown unused parts of the chipAdds complexity to chip, shutdown circuits add toenergy consumption

2 Run the chip at lower frequencies and/or lower voltageDoes not scale linearly downwards: running at 20% ofcapacity may still use 50% of energy

Dead-end?


Energy-proportional computing

Contents

1 Introduction




5 Summary

6 Bibliography



IntroductionEnergy-proportional computing introdcued by Barrosoand Hölzle at Google˜[2]

34 Computer

voltage-frequency scaling. Mobile devices require highperformance for short periods while the user awaits aresponse, followed by relatively long idle intervals ofseconds or minutes. Many embedded computers, suchas sensor network agents, present a similar bimodalusage model.4

This kind of activity pattern steers designers to empha-size high energy efficiency at peak performance levelsand in idle mode, supporting inactive low-energy states,such as sleep or standby, that consume near-zero energy.However, the usage model for servers, especially thoseused in large-scale Internet services, has very differentcharacteristics.

Figure 1 shows the distribution of CPU utilization lev-els for thousands of servers during a six-month inter-val.5 Although the actual shape of the distribution variessignificantly across services, two key observations fromFigure 1 can be generalized: Servers are rarely com-pletely idle and seldom operate near their maximum uti-lization. Instead, servers operate most of the time atbetween 10 and 50 percent of their maximum utiliza-tion levels. Such behavior is not accidental, but resultsfrom observing sound service provisioning and distrib-uted systems design principles.

An Internet service provisioned such that the averageload approaches 100 percent will likely have difficultymeeting throughput and latency service-level agree-

ments because minor traffic fluc-tuations or any internal disrup-tion, such as hardware orsoftware faults, could tip it overthe edge. Moreover, the lack of areasonable amount of slackmakes regular operationsexceedingly complex becauseany maintenance task has thepotential to cause serious servicedisruptions. Similarly, well-pro-visioned services are unlikely tospend significant amounts oftime completely idle becausedoing so would represent a sub-stantial waste of capital.

Even during periods of low ser-vice demand, servers are unlikelyto be fully idle. Large-scale ser-vices usually require hundreds ofservers and distribute the loadover these machines. In somecases, it might be possible tocompletely idle a subset of serversduring low-activity periods by,for example, shrinking the num-ber of active front ends. Often,though, this is hard to accom-plish because data, not just com-

putation, is distributed among machines. For example,common practice calls for spreading user data acrossmany databases to eliminate the bottleneck that a cen-tral database holding all users poses.

Spreading data across multiple machines improvesdata availability as well because it reduces the likeli-hood that a crash will cause data loss. It can also helphasten recovery from crashes by spreading the recov-ery load across a greater number of nodes, as is donein the Google File System.6 As a result, all servers mustbe available, even during low-load periods. In addition,networked servers frequently perform many small back-ground tasks that make it impossible for them to entera sleep state.

With few windows of complete idleness, servers can-not take advantage of the existing inactive energy-savings modes that mobile devices otherwise find soeffective. Although developers can sometimes restruc-ture applications to create useful idle intervals duringperiods of reduced load, in practice this is often difficultand even harder to maintain. The Tickless kernel7 exem-plifies some of the challenges involved in creating andmaintaining idleness. Moreover, the most attractive inac-tive energy-savings modes tend to be those with the high-est wake-up penalties, such as disk spin-up time, andthus their use complicates application deployment andgreatly reduces their practicality.

1.00

0.01

0.005

0.02

0.03

0.015

0.025

CPU utilization

Frac

tion

of ti

me

0.90.80.70.60.50.40.30.20.10

Figure 1. Average CPU utilization of more than 5,000 servers during a six-month period.Servers are rarely completely idle and seldom operate near their maximum utilization,instead operating most of the time at between 10 and 50 percent of their maximum utilization levels.

Observation: Servers are idling a lot of the timeWhere does idling come from:

There is not enough computationWaiting for I/OWaiting for memory



Energy-proportional computing I

Barroso and Hölzle suggest dynamic power control ofnodes in the datacenterThe datacenter would only keep a needed number ofnodes online to match the computational requirementsGoal: given a load curve match it with computationalpower as closely as possible

Energy Waste

Energy Waste

Computation Requirement

Energy Waste

Energy Waste

Computation Requirement



Energy-proportional computing II

Big servers, large granularity => lowenergy-proportionalitySmall servers, small granularity => betterenergy-proportionalitySmall servers will have less computational power, is thisa problem?



Memory Wall

big discrepancy between processor speed and memoryaccess speeds

processor speed has been growing with 50-100%annuallymemory speed has been growing with 7% annually

speed gap that is growing all the timeto avoid wait times, and keep deep pipelines working,large caches are required, which use a lot of powercomplex memory subsystems DDR1-3 are developedwhich also consume power



Amdahl blades˜[12] I

A balanced computer system needs:1 sequential I/O per sec - Amdahl number2 memory with Mbyte/MIPS ratio of close to 1 - Amdahl

memory ratio3 performs on I/O operation per 50k instructions -

Amdahl IOPS ratioGraywulf system (2008, state of the art architecture)Amdahl number 0.56, Amdahl ratio 1.12, Amdahl IOPSratio 0.014

Data cannot be fed fast enough into the system



Amdahl blades˜[12] II

Compare Graywulf with COTS systemsTable 2: Performance, power, and cost characteristics of various data-intensive architectures.

CPU Mem SeqIO RandIO Disk Power Cost Relative Amdahl numbers[GHz] [GB] [GB/s] [kIOPS] [TB] [W] [$] Power Seq Mem Rand

GrayWulf 21.3 24 1.500 6.0 22.5 1,150 19,253 1.000 0.56 1.13 0.014ASUS 1.6 2 0.124 4.6 0.25 19 820 0.017 0.62 1.25 0.144Intel 3.2 2 0.500 10.4 0.50 28 1,177 0.024 1.25 0.63 0.156Zotac 3.2 4 0.500 10.4 0.50 30 1,189 0.026 1.25 1.25 0.163AxiomTek 1.6 2 0.120 4.0 0.25 15 995 0.013 0.60 1.25 0.125Alix 3C2 0.5 0.5 0.025 N/A 0.008 4 225 0.003 0.40 1.00

2008 (essentially the same today). We note that for theSSD-based systems the cost and disk size columns inTable 2 represent projections for a 250 GB drive withthe same performance and a projected cost of $400 atthe end of 2009, in line with historic SSD price trends.

Power consumption varies between 15W-30W depend-ing on the chipset used (945GSE, USW15, ION) andgenerally agrees with the values reported in the moth-erboards’ specifications. The current university rate forelectric power at JHU is $0.15/kWh. The total cost ofpower should include the cost for cold water and air con-ditioning, thus we multiply the electricity cost by 1.6[7]. Table 2 presents these cumulative costs.

Lastly, we present the different Amdahl numbers andratios for the various node types. It is clear that, com-pared to the GrayWulf and Alix, the Atom systems, es-pecially with dual cores, are better balanced across allthree dimensions.

Scaling Properties. Table 3 illustrates what happenswhen we scale the other systems to match the Gray-Wulf’s sequential I/O, power consumption, and diskspace. We present the number of nodes necessary tomatch the GW’s performance in the selected dimension,while the remaining columns show the aggregate perfor-mance across all these nodes.

We note that a cluster of only three Intel or Zotacnodes will match the sequential read I/O of the Gray-Wulf and deliver five times faster IOPS, while consum-ing 90W, compared to 1150W for the GW. The onlyshortcoming of this alternative is that the total storagecapacity is 15 times smaller. At the same time, thepower for a single GrayWulf node can support 41 Inteland 38 Zotac nodes, respectively, and offer more thanten times higher sequential read I/O throughput.

Table 3 also shows that one needs to strike a balancebetween low power consumption and high performance.For example, while the sequential read I/O performanceof the Alix system matches that of the GrayWulf at aconstant price, it falls behind that of the Amdahl blades.Furthermore, one needs 60 Alix boards to match thesequential rate of a GW node which consume approxi-

mately three times more power than the equivalent Intelsystem (240 W vs. 84 W).

5. DISCUSSIONThe nature of scientific computing is changing – it is

becoming more and more data-centric while at the sametime datasets continue to double every year, surpass-ing petabytes. As a result, the computer architecturescurrently used in scientific applications are becomingincreasingly energy inefficient as they try to maintainsequential read I/O performance with growing datasetsizes. The scientific community therefore faces the fol-lowing dilemma: find a low-power alternative to exist-ing systems or stop growing computations on par withthe size of the data. We thus argue that it is unavoidableto build scaled-down and scaled-out systems compris-ing large numbers of compute nodes each with a muchlower relative power consumption at a given sequentialread I/O throughput.

We use Amdahl’s laws to guide the selection of thesmallest CPU throughput necessary to run data-intensiveworkloads dominated by sequential reads. Furthermore,we propose a new class of so-called Amdahl bladesthat combine energy-efficient processors and solid statedisks to offer significantly higher throughput and lowerenergy consumption. We find that today the dual-coreAmdahl blades represent a sweet spot in the energy-performance curve, while alternatives using lower powerCPUs (i.e., single-core Atom, Geode) and Compact Flashcards offer lower relative throughput. As technologytrends evolve, we believe that Amdahl’s laws can con-tinue to guide the design of servers in the future.

The only advantage of existing systems is their highertotal storage space. However, as SSD capacities are un-dergoing an unprecedented growth, this temporary ad-vantage will rapidly disappear: as soon as we have a750 GB SSD for $400, the storage built of low-powersystems will have a lower total cost of ownership thanregular hard drives. An intriguing alternative is usingnodes in which one SATA port will be connected to anSSD while the other port(s) will be connected to low-

It would seem that running at lower frequencies givesbetter power efficiency?


Servers built on mobile processors

Contents

1 Introduction




5 Summary

6 Bibliography



Introduction

Mobile processors are very energy efficientCould we used mobile processors (e.g ARM) toimplement energy proportional data centers?Research problems:

Are mobile processors more energy efficient compared toIA processors?How would one implement energy proportionality?

Our goal: build an energy proportional cluster based onARM processors



Commercial drivers

MoneyMobile processors are cheap

Optimized for low energyconsumptionArchitectureLow-power states

Heat-densityIt will be possible to place servers into spaces where itcurrently is not possible



Energy efficiency of ARM processors

Comparison of Energy per Instruction (i.e. what couldbe ideally achieved?)Challenges

9/3/10&5&

Processor EPI(nJ) NominalCPI Power consumption

ARM720T 0.22 2.2 65mW

ARM926EJ-S 0.46 1.6 95mW

ARM1136J-S 0.63 1.4 115mW

Processor EPI (nJ) Nominal CPI Power consumption

Pentium 4 48 2.59 5060mW

Pentium M 15 2320mW

Core Duo 11 1190mW

Energy efficiency: ARM vs Pentium

ARM720T is 220 times more energy efficient than a Pentium 4

Do 220 ARM720T’s have more computational power than a Pentium 4? For which workloads?



Hardware

Versatile Express Quad-core Cortex A9, 1GB DDR2,

400MhzTegra 250 Dual-core Cortex-A9, 1GB DDR2, 1G



Benchmarks

Autobench and Apache 2 HTTP serverstatic web pages

SPECweb2005more demanding web services

Erlangmicro benchmarksreal world SIP proxy



Autobench and Apache



SPECWeb2005



Erlang



Results

The performance of 2 ARMv7 based ARM cortex-A9was measured and evaluated and compared to XeonprocessorsMeasurements show that the Cortex A9 can be up to

11 times more efficient with the Apache server3.6 times more efficient with Erlang base SIP proxy2.9 times more efficient with the SPECweb2005



Implementation of energy proportianality

Since the computational capacatiy of an ARM processoris lower that an IA processer we need more of themThis enables a more fine-grained control of thecomputational power



Evaluation Platform

Few fast CPUs dissipate much power and have roughpower granularityInstead use slow but many mobile CPUs to increase thepower granularityEvaluation done on a cluster using ARM Cortex-A8(Beagleboard)Low energy consumption and low price tagRunning Ubuntu 11.04 with Linux 2.6.34



The problem with DVFS

The CPU is the main energy consumer in the serverPower managers have been used to scale theperformance of the CPU according to the demandDVFS scales the voltage and frequencyDVFS does not scale the power dissipation linearly !



Power Manager using Sleep StatesThe power manager is used to dynamically adjust thesystem capacity to the workloadThe manager wakes up cores when the capacity is toolow and shuts down cores when capacity is unnecessaryhighGoals:

Good power-to-workload proportionality resulting inlittle energy wasteShow a minimal performance degradation to usersScale in large clusters



System level power manager

Switches on/off CPU cores in the clusterOperates on systems level, meaning that it controls thewhole cluster as one entityMaster core controlling the other workersWorkers taking orders: Sleep/Wake and workload



Simulation framework



Simulation framework

Quality of Service (QoS) is our current measurement ofperformanceQoS shows how many % of the incoming requests arehandled in a time frameThe unhandled requests are moved to the next timeframe and result in a QoS dropQoS drop is a result of latency that triggers a deadlinemissSimulation framework can select static cores that arenot altered by the power manager, i.e. always running



CPU characteristics

Data based on ARM Cortex-A8 benchmarksThe power dissipation of the CPU is 1.4W at full speedWaking up a core takes about 650 msThe load capacity of a CPU was benchmarked with thetool Autobench resulting in 5 requests/second for a 248KB fileAn arbitrary amount of CPUs can be simulated with theframework assumed that the CPU specs are given



Comparison with DVFS



Power savings vs QoS

DVFS has the highest QoS but wastes much energyUp to 60% in energy can be saved with only 4%degradation in QoS20% energy can still be saved if less than 1% QoSdegradation is requested



Other results and future work

Implementation of a real-life demo based on clusterconsisting of 8 XM BeagleBoards

This is done and we have validated the resultsImplementation of PID power manager into LinuxScheduler

Validated the approach using LinSchedCurrently implementing the framework into Linux kernel

Video transcoding “over the cloud”Evaluation of different control mechanisms

State based control theory vs. PIDEvaluation of more complex settings:

Is the approach applicable with VMs running on thecluster?


Summary

Contents

1 Introduction




5 Summary

6 Bibliography


Summary

Summary

1 PUE measures the overall power loss in the datacenter,it does not measure the total power consumption

2 ARM based processors are more energy efficient that IAbased processors for typical datacenter loads

3 IA based processors are more energy efficient incomputations with large computational kernels

4 Energy Proportionality is the key to lower total powerconsumption in datacenters

5 It is feasible to implement Energy Proportional PowerManagers into modern OS kernels

6 Many interesting research challenges remain!


Summary

Acknowledgements

The Lean Server TeamProf. Johan LiliusDr. Sebastien LafondM.Sc. Simon HolmbackaM.Sc. Fareed JohkioM.Sc. Tewodros DenekeM.Sc Fredric HällisAlumni

M.Sc. Jens SmedsM.Sc. Olle Swanfeldt-WinterM.Sc. Joachim SjölundM.Sc. Joakim Nylund


Summary

Our Papers I

Simon Holmbacka, Sébastien Lafond, Johan Lilius, APID-Controlled Power Manager for Energy Efficient WebClusters. In: International Conference on Cloud andGreen Computing (CGC2011), 721-728, IEEE ComputerSociety, 2011.Simon Holmbacka, Jens Smeds, Sébastien Lafond,Johan Lilius, System Level Power Management forMany-Core Systems. In: (Ed.), Workshop on MicroPower Management for Macro Systems on Chip, 2011.Sébastien Lafond, Simon Holmbacka, Johan Lilius, ASystem Level Power Management for Web Clusters. In:COST Action IC0804 on Energy Efficiency in Large ScaleDistributed Systems, 2nd Year, 127-131, IRIT, 2011.


Summary

Our Papers II

Sébastien Lafond Johan Lilius Simon Holmbacka, PowerProportional Characteristics of an Energy Manager forWeb Clusters. In: Embedded Computer Systems:Architecture, Modeling and Simulation (SAMOS) 2011,8, IEEE pres, 2011.Olle Svanfeldt-winter, Sébastien Lafond, Johan Lilius,Cost and Energy Reduction Evaluation for ARM BasedWeb Servers. In: International Conference on Cloud andGreen Computing (CGC2011), 480-487, IEEE ComputerSociety, 2011.


Bibliography

Contents

1 Introduction




5 Summary

6 Bibliography


Bibliography

References I

David Andersen, Jason Franklin, Michael Kaminsky,Amar Phanishayee, Lawrence Tan, and Vijay Vasudevan.FAWN: a fast array of wimpy nodes.SOSP ’09: Proceedings of the ACM SIGOPS 22ndsymposium on Operating systems principles, October2009.LA Barroso and U Holzle.The case for energy-proportional computing.Computer, 40(12):33–37, 2007.

BG Chun, G Iannaccone, G Iannaccone, R Katz, G Lee,and L Niccolini.An energy case for hybrid datacenters.ACM SIGOPS Operating Systems Review, 44(1):76–80,2010.


Bibliography

References II

James Hamilton.Cooperative Expendable Micro-Slice Servers (CEMS):Low Cost, Low Power Servers for Internet-Scale Services.

pages 1–8, December 2008.

James Hamilton.Data Center Efficiency Best Practices.pages 1–29, April 2009.

Urs Holzle.Brawny cores still beat wimpy cores, most of the time.IEEE Micro, pages 1–2, June 2010.


Bibliography

References IIIW Lang, JM Patel, and S Shankar.Wimpy Node Clusters: What About Non-WimpyWorkloads?2010.Jacob Leverich and Christos Kozyrakis.On the Energy (In)efficiency of Hadoop Clusters.pages 1–5, July 2009.

Reijo Maihaniemi.Energy Efficient ICT.Presentation, pages 1–27, September 2009.

J Mankoff, R Kravets, and E Blevis.Some computer science issues in creating a sustainableworld.Computer, 41(8):102–105, 2008.


Bibliography

References IV

S Ruth.Green IT More Than a Three Percent Solution?IEEE Internet Computing, 2009.

Alexander Szalay, Gordon Bell, H Huang, Andreas Terzis,and Alainna White.Low-power amdahl-balanced blades for data intensivecomputing.SIGOPS Operating Systems Review, 44(1), March 2010.


green computing - jyväskylän yliopistousers.jyu.fi/~riesta/green_computing.pdf · introduction...

Documents