Data Center Energy Efficiency

SC07 Birds of a FeatherNovember, 2007Bill Tschudiwftschudi@lbl.gov

Benchmark results helped to find Benchmark results helped to find best practicesbest practices

The ratio of IT equipment power to the total The ratio of IT equipment power to the total (or its inverse) is an indicator of relative (or its inverse) is an indicator of relative overall efficiency. Examination of individual overall efficiency. Examination of individual systems and components in the centers that systems and components in the centers that performed well helped to identify best performed well helped to identify best practices.practices.

Percentage of electricity delivered Percentage of electricity delivered to IT equipmentto IT equipment

IT Power to Total Data Center Power

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Data Center Number

Rat

io

Average .57

Higher is better

Source: LBNL Benchmarking

Performance varies

Data Center Server Load

51%

Data Center CRAC Units

25%

Cooling Tower Plant4%

Electrical Room Cooling

4%

Office Space Conditioning

1%

Lighting2%

Other13%

Computer Loads67%

HVAC - Air Movement

7%

Lighting2%

HVAC - Chiller and

Pumps24%

The relative percentages of the energy actually doing computing varies considerably.

Typical power useTypical power use

Courtesy of Michael Patterson, Intel Corporation

Computer/SystemArchitecture

System Architecture /Power Engineering

Facility Design /Mechanical Design

Server Load/ Computing Operations

Cooling Equipment

Power Conversion & Distribution

Alternative Power

Generation

• High voltage distribution• Use of DC power• Highly efficient UPS systems• Efficient redundancy

strategies

• Load management• Server innovation

Energy efficiency opportunities are Energy efficiency opportunities are everywhereeverywhere

• Better air management• Move to liquid cooling• Optimized chilled-water plants• Use of free cooling

• On-site generation• Waste heat for cooling• Use of renewable energy/fuel cells

Best practices topics identified Best practices topics identified through benchmarkingthrough benchmarking

HVAC – Air Delivery – Water Systems

Facility Electrical Systems

IT Equipment

Cross-cutting / misc. issues

Air management

Cooling plant optimization

UPS systems

Power Supply efficiency

Motor efficiency

Air economizers

Free cooling Self generation

Sleep/standby loads

Right sizing

Humidificationcontrols alternatives

Variable speed pumping

AC-DC Distribution

IT equip fans Variable speed drives

Centralized air handlers

Variable speed Chillers

Standby generation

Lighting

Direct liquid cooling

Maintenance

Low pressure drop air distribution

Commissioning/continuous benchmarking

Fan efficiency Heat recovery Redundancies Method of charging for

space and power Building envelope

A word about appropriate A word about appropriate environmental conditionsenvironmental conditions……

ASHRAE published thermal guidelinesASHRAE published thermal guidelines–– All IT suppliers participated and agreedAll IT suppliers participated and agreed–– Guidelines allow most centers to relax setpointsGuidelines allow most centers to relax setpoints–– HPC community agreesHPC community agrees

Recommended and allowable ranges of Recommended and allowable ranges of temperature and humidity are provided temperature and humidity are provided –– at the at the inlet to the IT equipmentinlet to the IT equipment

High temperatures in the High temperatures in the ““hot aisleshot aisles”” and return to and return to air conditioners are desirable.air conditioners are desirable.

Humidity guidelines Humidity guidelines –– at the inlet to IT equipmentat the inlet to IT equipment

ASHRAE HUMIDITY GUIDELINES

0

10

20

30

40

50

60

70

80

90

100

% R

elat

ive

Hum

idity

ASHRAE Allowable Maximum

ASHRAE Allowable Minimum

ASHRAE Recommended Maximum

ASHRAE Recommended Minimum

Temperature guidelines Temperature guidelines –– at the inlet to IT equipmentat the inlet to IT equipment

ASHRAE TEMPERATURE GUIDELINES

40

50

60

70

80

90

100

Deg

rees

F

ASHRAE Allowable Maximum

ASHRAE Allowable Minimum

ASHRAE Recommended Maximum

ASHRAE Recommended Minimum

Air management best scenario Air management best scenario –– isolate cold and hotisolate cold and hot

70-75º

95-100º

Another isolation schemeAnother isolation scheme

Measured fan energy savingsMeasured fan energy savings –– 75%75%

If mixing of cold supply air If mixing of cold supply air with hot return air can be with hot return air can be eliminatedeliminated--fan speed can be reducedfan speed can be reduced

Better temperature control can Better temperature control can allow raising the temperature in allow raising the temperature in the entire data center!the entire data center!

Cold Aisle NW - PGE12813

40

45

50

55

60

65

70

75

80

85

90

6/13/2006 12:00 6/14/2006 0:00 6/14/2006 12:00 6/15/2006 0:00 6/15/2006 12:00 6/16/2006 0:00 6/16/2006 12:00

Time

Tem

pera

ture

(deg

F)

LowMedHigh

Baseline Alternate 1

Setup

Setup

Alternate 2

ASHRAE Recommended Range

Ranges during demonstration

See the problem areasSee the problem areas

Infrared thermography and CFD modeling can be used as visualization tools

Best practices Best practices –– Free cooling with Free cooling with air economizersair economizers

HVAC – Air Delivery – Water Systems

Facility Electrical Systems

IT Equipment

Cross-cutting / misc. issues

Air management

Cooling plant optimization

UPS systems

Power Supply efficiency

Motor efficiency

Air economizers

Free cooling Self generation

Sleep/standby loads

Right sizing

Humidificationcontrols alternatives

Variable speed pumping

AC-DC Distribution

IT equip fans Variable speed drives

Centralized air handlers

Variable speed Chillers

Standby generation

Lighting

Direct liquid cooling

Maintenance

Low pressure drop air distribution

Commissioning/continuous benchmarking

Fan efficiency Heat recovery Redundancies Method of charging for

space and power Building envelope

Encouraging outside air economizersEncouraging outside air economizers

Issue: Issue: –– Many are reluctant to use air economizersMany are reluctant to use air economizers–– Outdoor pollutants and humidity control considered Outdoor pollutants and humidity control considered

equipment riskequipment risk

Goal: Goal: –– Encourage use of outside air economizers where Encourage use of outside air economizers where

climate is appropriateclimate is appropriate

Strategy: Strategy: –– Address concerns: contamination/humidity controlAddress concerns: contamination/humidity control–– Quantify energy savings benefitsQuantify energy savings benefits

Outdoor MeasurmentsFine Particulate Matter

0

20

40

60

80

100

120

140

160

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

Part

icle

Con

c. (

μg/

m3 ) LBNL

NERSC

Center 3

Center 4

Center 5

Center 6

Center 7

Center 8

Outdoor measurementsOutdoor measurements

IBM Standard

EPA Annual Health Standard

EPA 24-Hour Health Standard

and ASHRAE Standard

Indoor Measurments Fine Particulate Matter

0

20

40

60

80

100

120

140

160

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

6:00AM

12:00PM

6:00PM

12:00AM

Part

icle

Con

c. (

μg/

m3 ) LBNL

NERSC

Center 3

Center 4

Center 5

Center 6

Center 7

Center 8

Measurements inside the centersMeasurements inside the centers

IBM Standard

EPA Annual Health Standard

EPA 24-Hour Health Standard

and ASHRAE Standard

Data center w/economizerData center w/economizer

Center 8w/economizer

0.3-5 Particulate Matter

0

10

20

30

40

50

60

70

80

90

100

8/18/060:00

8/18/0612:00

8/19/060:00

8/19/0612:00

8/20/060:00

8/20/0612:00

8/21/060:00

8/21/0612:00

8/22/060:00

8/22/0612:00

8/23/060:00

8/23/0612:00

8/24/060:00

8/24/0612:00

8/25/060:00

8/25/0612:00

8/26/060:00

Part

icle

Con

c. ( μ

g/m

3 )

OutsideOutside (PostFilter)PreServerRmAmb

FindingsFindings

Water soluble salts in combination with high Water soluble salts in combination with high humidity can cause failureshumidity can cause failuresNew ASHRAE particle limits drastically lower than New ASHRAE particle limits drastically lower than one manufacturerone manufacturer’’s recommendations recommendationParticle concentration typically an order of Particle concentration typically an order of magnitude lower than new ASHRAE limits (no magnitude lower than new ASHRAE limits (no economizer) economizer) Economizers, without other mitigation, can cause Economizers, without other mitigation, can cause particle concentration to approach new ASHRAE particle concentration to approach new ASHRAE limits but filtration can mitigate thislimits but filtration can mitigate thisLarge energy savingsLarge energy savings

050

100150

200250300

350400

450500

Hrs/YR

28 35 42 49 56 63 70 77 84 91 98 105 112Temperature (F)

Berkeley Weather

hr/yr0 10 0

11 20 021 30 331 40 50441 50 232551 60 315361 70 154271 80 84681 90 29791 100 75

101 110 15111 120 0

Range F

Set Inlet TemperatureTo upper end of ASHRAE

NERSC/LBNL CRT Building Conceptual DesignDRAFT: Subject to change Howard Walter

Schematic Design ConceptsSchematic Design ConceptsDRAFT: Subject to change; Do not re-distribute

Howard Walter

Best practices Best practices –– power conversionpower conversion

HVAC – Air Delivery – Water Systems

Facility Electrical Systems

IT Equipment

Cross-cutting / misc. issues

Air management

Cooling plant optimization

UPS systems

Power Supply efficiency

Motor efficiency

Air economizers

Free cooling Self generation

Sleep/standby loads

Right sizing

Humidificationcontrols alternatives

Variable speed pumping

AC-DC Distribution

IT equip fans Variable speed drives

Centralized air handlers

Variable speed Chillers

Standby generation

Lighting

Direct liquid cooling

Maintenance

Low pressure drop air distribution

Commissioning/continuous benchmarking

Fan efficiency Heat recovery Redundancies Method of charging for

space and power Building envelope

Inverter

In Out

Bypass

Battery/ChargerRectifier

Internal Drive

External Drive

I/O

Memory Controller

μ Processor

SDRAM

Graphics Controller

DC/DCAC/DC

DC/DC

AC/DC Multi output PS

Voltage Regulator Modules

5V

12V

3.3V

12V 1.5/2. 5V

1.1V- 1.85V

3.3V

3.3V

12V

PWM/PFCSwitcher

Unregulated DCTo Multi Output Regulated DC

Voltages

Data center power conversions

AC voltage conversions

Prior research illustrated large Prior research illustrated large losses in power conversionlosses in power conversion

45%

50%

55%

60%

65%

70%

75%

80%

85%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

% of Nameplate Power Output

% E

ffici

ency

Average of All Servers

Uninterruptible Power Supplies (UPS)

Power Supplies in IT equipment

Factory Measurements of UPS Efficiency

70%

75%

80%

85%

90%

95%

100%

0% 20% 40% 60% 80% 100%

Percent of Rated Active Power Load

Effic

ienc

y

Flywheel UPS

Double-Conversion UPS

Delta-Conversion UPS

(tested using linear loads)

With over 25 industry partners direct DC powering of servers was demonstrated

Typical AC distribution todayTypical AC distribution today

DC/ACAC/DC480 VACBulk Power

Supply

UPS PDU

AC/DC DC/DC VRM

VRM

VRM

VRM

VRM

VRM

12 V

Loadsusing

LegacyVoltages

Loadsusing

SiliconVoltages

12 V

5 V

3.3 V

1.2 V

1.8 V

0.8 VServer

PSU

480 Volt AC

FacilityFacility--level DC distributionlevel DC distribution

AC/DC480 VACBulk Power

SupplyDC UPS

orRectifier

DC/DC VRM

VRM

VRM

VRM

VRM

VRM

12 V

Loadsusing

LegacyVoltages

Loadsusing

SiliconVoltages

12 V

5 V

3.3 V

1.2 V

1.8 V

0.8 VServer

PSU

380 VDC380V.DC480 Volt AC

AC system loss compared to DCAC system loss compared to DC

DC/ACAC/DC480 VACBulk Power

Supply

UPS PDU

AC/DC DC/DC VRM

VRM

VRM

VRM

VRM

VRM

12 V

Loadsusing

LegacyVoltages

Loadsusing

SiliconVoltages

12 V

5 V

3.3 V

1.2 V

1.8 V

0.8 VServer

PSU

AC/DC480 VACBulk Power

SupplyDC UPS

orRectifier

DC/DC VRM

VRM

VRM

VRM

VRM

VRM

12 V

Loadsusing

LegacyVoltages

Loadsusing

SiliconVoltages

12 V

5 V

3.3 V

1.2 V

1.8 V

0.8 VServer

PSU

380 VDC

7-7.3% measured improvement

2-5% measured improvement

Rotary UPS

websites: websites:

http://http://hightech.lbl.govhightech.lbl.gov/datacenters//datacenters/

How soon will we get to liquid cooling? How soon will we get to liquid cooling? Why not doing it today?Why not doing it today?Why isnWhy isn’’t efficient distribution power more t efficient distribution power more widely utilized?widely utilized?–– High efficiency UPS or onHigh efficiency UPS or on--site generationsite generation–– High voltage distribution to RackHigh voltage distribution to Rack–– Direct DC Direct DC –– eliminating conversions eliminating conversions –– ease of ease of

incorporating renewable sources incorporating renewable sources –– available available todaytoday