alternative financing of data center energy projects · 2020-01-03 · 1. data center energy: a....

1U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY

Alternative Financing of Data Center Energy Projects

Dale Sartor, Lawrence Berkeley National Laboratory

November 9, 2018

Version 110118


1. Data center energy:

a. Trends

b. Start with metrics and benchmarking

2. Drivers for Federal data center optimization.

3. Opportunities in data centers.

4. How can ESPCs/UESCs help?

5. Alternative financing examples.

6. Lessons learned/resources/discussion.

Agenda


Data Center Energy

Data centers are energy intensive facilities

• 10 to 100+ times more energy intensive than an office

• Server racks now designed for more than 30 kW

• Surging demand for data storage

• 2% of US electricity consumption

• Power and cooling constraints in existing facilities

• Perverse incentives –

IT and facilities costs separate

Potential Benefits of Energy Efficiency

• 20-40% savings & high ROI typical

• Aggressive strategies can yield 50+% savings

• Extend life and capacity of infrastructures


US Data Center Energy Usage Reports (2007 & 2016)

~1.8% U.S. Electricity

45% Reduction Possible with Best Practices and greater shift to hyper-scale


Energy Use Projections and Counterfactual

Savings: 620 billion

kWh


Energy Use Estimates by Data Center Type

• Hyperscale is a growing percentage of data center

energy use


2050 Projections


In Conclusion…

• Data center energy use has approximately

plateaued since 2008

• Expected to continue through 2020

• Further efficiency improvements possible, but will

eventually run out

• Next-generation computing technologies and

innovative data center business models will be

needed to keep energy consumption down over

the next 20-30 years


Data Center

Server Load

51%

Data Center

CRAC Units

25%

Cooling Tower

Plant

4%

Electrical Room

Cooling

4%

Office Space

Conditioning

1%

Lighting

2%

Other

13%

First Step: Benchmark Energy Performance

• Compare to peers – Wide variation

• Identify best practices

• ID opportunities

• Track performance – Can’t manage what isn’t measured

• The relative percentage of energy actually doing computing varies Computer

Loads

67%

HVAC - Air

Movement

7%

Lighting

2%

HVAC -

Chiller and

Pumps

24%


High Level Metric: PUE

Power Utilization Effectiveness (PUE) = Total Power/IT Power


Data centers (big energy usage sector and growing)

Account for >2% of US electricity consumption

Federal Information Technology Acquisition Reform Act (FITARA) (2014)

Agencies must submit annual reports (inventories, strategies, timeline, savings)

OMB must set targets and publish agency cost savings and optimization

improvements

Data Center Optimization Initiative (DCOI) (2016)

Sets framework for agencies to meet FITARA consolidation and optimization

requirements - Defines metrics and sets goals

Requires Installing and monitoring advanced energy meters in all data centers (by

FY2018?)

Establishing a Power Usage Effectiveness (PUE) target of 1.2 to 1.4 for new data

centers and less than 1.5 for existing data centers

Update expected soon

Modernizing Government Technology (MGT) Act (Dec 2017)

Establishes $100M Technology Modernization Fund (TMF)

What drives Federal Data Center change?


What is the opportunity in data centers?

How is a typical Federal data center configured? IT

• Not virtualized (software/application tied to specific hardware).

• Not consolidated (small data centers spread across campus/installation).

• Under utilized (10-20% vs 45-60%).

• Power management features disabled (e.g. ‘sleep mode’).

• Many servers (10-20%) on but doing no work.

Air Management• No cold/hot air isolation, air pathways congested, server exhaust recirculation

• CRAC controlled by return air temperature, constant speed fans

Cooling• CRAC-cooled (65°F, 60RH), (direct expansion coil, refrigerant compressor)

• Over-spec’d: 2N (twice as many as needed, for backup)

• Equipment “fighting” - e.g. simultaneous humidifying and dehumidifying

Electrical• Uninterruptible power supply (UPS) over-spec’d: 2N (instead of N+1) with all units running in

parallel at very low loads (and efficiency)

• Many AC->DC->AC->DC & voltage conversions.

Management• Little or no information systems (e.g. Data Center Infrastructure Management (DCIM) system)

to monitor and track performance

• Little or no systems integration (e.g. controls)


100 Units

Source

Energy

IT Load

Cooling

Equipment

Power

Conversions

& Distribution

33 Units

Delivered

35 Units

Power Generation

Typical Data Center Energy Efficiency ~ 15%


Energy Efficiency Opportunities

IT Load/

Computing

Operations

Cooling

Equipment

Power Conversion & Distribution

AlternativePower

Generation

• High-voltage distribution

• High-efficiency UPS

• Efficient redundancy strategies

• Use of DC power

• IT innovation

• Virtualization

• High-efficiency

power supplies

• Load management

• Better air management

• Move to liquid cooling

• Optimized chilled-water plants

• Use of free cooling

• Heat recovery

• On-site generation

Including fuel cells and

renewable sources

• CHP applications

(waste heat for cooling)


1. Measure and Benchmark Energy Use

2. Identify IT Equipment and Software Opportunities

3. Use IT to Monitor and Control IT

4. Optimize Environmental Conditions

5. Manage Airflow

6. Evaluate Cooling Options

7. Reconsider Humidity Control

8. Improve Electrical Efficiency

9. Implement Energy Efficient O&M

Best Practices


IT Load Can Be Controlled

Computations per Watt is improving

Opportunities:• Consolidation• Server efficiency (Use ENERGY STAR)

Flops per Watt Efficient power supplies and less redundancy.

• Software efficiency Virtualize for higher utilization Data storage management.

• Enable power management (e.g., sleep mode)

• Reducing IT load has a multiplier effect Savings in infrastructure energy depends on

PUE

Core Integer Performance Over Time*

Pentium® Pro Processor

Pentium® -II Processor

Pentium® -IIi Processor

Pentium® 4 Processor

Pentium® 4 Processor EE

Pentium® -D Processor

Core™ 2 Duo Processor X6800

Core™ 2 Duo Extreme QX6700

i486DX2

i486

386

Pentium® Processor

1986 20081988 1990199219941996199820002002200420061

Single Core Moore’s Law


Virtualize and Consolidate Servers and Storage

• Run many “virtual” machines

on a single “physical” machine

• Consolidate underutilized

physical machines, increasing

utilization

• Energy saved by shutting down

underutilized machines


Server Consolidation

10:1 in many cases

HWHW

HW

VMM

Disaster Recovery

HW

VMM

HW

VMM

• Upholding high-levels of business continuity

• One Standby for many production servers

…OS

App

OS

App

OS

App …OS

App

HWVMM

HW

VMM

Balancing utilization with head room

Dynamic Load Balancing

OS

App1

OS

App2

OS

App3

OS

App4

CPU Usage

30%

CPU Usage

90%

CPU Usage CPU Usage

Enables rapid deployment,

reducing number of idle, staged servers

R&D Production

HW

VMM

OS

App

Virtualize and Consolidate Servers and Storage


Using IT to Save Energy in IT

• Operators lack visibility into data center environment

• Provide same level of monitoring and visualization of the physical space as we have for the IT environment

• Measure and track performance

• Spot problems early

• Example: 800 point SynapSense system– Temperature, humidity, under-floor pressure, current

source: SynapSenseLBNL Wireless Monitoring System


Visualization getting much better

SynapSense™


Real-time PUE Display


Environmental Conditions: Safe Temperature Limits

~65C

~75C

CPUs

GPUs

~85C

Memory

(149F)

(185F)

(167F)

CPU, GPU & Memory, represent ~75-90% of heat load …

So why do we

need jackets in

data centers?

Slide courtesy of NREL


Provides common

understanding between IT

and facility staff.

Developed with IT

manufacturers

Recommends temperature range

up to 80.6°F with “allowable”

much higher.

ASHRAE Thermal Guidelines

The defacto standard in the industry

Six classes of equipment

identified with wider

allowable ranges to 45°C

(113°F).

Provides more justification

for operating above the

recommended limitsProvides wider

humidity

ranges


Air Management: The Early Days at LBNL

Fans were used to redirect air

High flow tiles reduced air pressure

It was cold but hot spots were everywhere


Typically, more air circulated than required

Air mixing and short circuiting leads to:

- Low supply temperature

- Low Delta T

Use hot and cold aisles Improve isolation of hot

and cold aisles

- Reduce fan energy

- Improve air-conditioning

efficiency

- Increase cooling

capacity

Hot aisle / cold aisle

configuration decreases

mixing of intake & exhaust

air, promoting efficiency.

Air Management


top of rack

middle of rack

SynapSense™

SynapSense™

Results: Blanking Panels

One 12 inch blanking panel

reduced temperature ~20°F

Equip.

Rack

Recirculation

Air


Results: Tune Floor Tiles

• Too many permeable floor tiles

• if airflow is optimized

– under-floor pressure

– rack-top temperatures

– data center capacity increases

• Measurement and visualization

assisted tuning process

under-floor pressures

rack-top temperatures

SynapSense™

SynapSense™


Improve Air Management

• Overhead plenum

converted to hot-

air return

• Return registers

placed over hot

aisle

• CRAC intakes

extended to

overhead

Before

After


Adding Air Curtains for Hot/Cold Isolation


Isolate Cold and Hot Aisles

95-105°vs. 60-70°

70-80°vs. 45-55°


Use Free Cooling

Cooling without Compressors

• Water-side Economizers

• Outside-Air Economizers

• Let’s get rid of chillers in data centers


Liquid Based Cooling

• Liquid is much more efficient than air

for heat transfer

• Efficiency improves the closer the liquid

comes to the heat source (e.g. CPU)

• Most efficient data centers often don’t

have raised floors!


LBNL Example: Rear Door Cooling

• Used instead of adding

CRAC units

• Cooling with tower-only

or chiller assisted

– Both options

significantly more

efficient than existing

direct expansion (DX)

CRAC units.


“Chill-Off 2” Evaluation of Liquid Cooling Solutions

Data Center Opportunity: Getting Liquid Closer


• Eliminate inadvertent dehumidification

– Computer load is sensible only

• Use ASHRAE allowable RH and temperature

– Many manufacturers allow even wider humidity range

• Eliminate equipment fighting

– Coordinate controls

– Turn off

Improve Humidity Control


Temp RH Tdp Temp RH Tdp Mode

AC 005 84.0 27.5 47.0 76 32.0 44.1 Cooling

AC 006 81.8 28.5 46.1 55 51.0 37.2 Cooling & Dehumidification

AC 007 72.8 38.5 46.1 70 47.0 48.9 Cooling

AC 008 80.0 31.5 47.2 74 43.0 50.2 Cooling & Humidification

AC 010 77.5 32.8 46.1 68 45.0 45.9 Cooling

AC 011 78.9 31.4 46.1 70 43.0 46.6 Cooling & Humidification

Min 72.8 27.5 46.1 55.0 32.0 37.2

Max 84.0 38.5 47.2 76.0 51.0 50.2

Avg 79.2 31.7 46.4 68.8 43.5 45.5

Visalia Probe CRAC Unit Panel

The Cost of Unnecessary Humidification

Humidity down 2%

CRAC power down 28%


Power Chain Conversions Waste Energy

local distribution lines

to the building, 480 V

HVAC system

lights, office space, etc.

UPS PDU computer racks

backup diesel

generators

Electricity Flows in Data CentersElectricity Flows in Data Centers

computer

equipment

uninterruptible

load

UPS = Uninterruptible Power Supply

PDU = Power Distribution Unit;


45%

50%

55%

60%

65%

70%

75%

80%

85%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

% of Nameplate Power Output

% E

ffic

ien

cy

Average of All Servers

Improving the Power Chain

• Increase distribution voltage

– NERSC going to 480 volts to the racks

• Improve equipment power supplies

– Avoid redundancy unless needed

• Improve UPS

– LBNL uses minimal UPS

– Selected to minimize losses


Redundant

Operation

Measured UPS Efficiency


Redundancy

• Understand what redundancy costs – is it worth it?

• Different strategies have different energy penalties (e.g. 2N vs. N+1)

• Redundancy in electrical distribution puts you down the efficiency curve

• Does everything need the same level?

• Redundancy in the network rather than in the data center


• Get IT and Facilities people working together

• Use life-cycle total cost of ownership analysis

• Document design intent and provide training

• Benchmark and track existing facilities

• Eat your spinach (blanking panels, leaks, CRAC maintenance)

• Re-commission regularly as part of maintenance

• Keep an eye on emerging technologies (e.g. rack-level

cooling, DC power)

Improve M&O Processes


1. Measure and Benchmark Energy Use

2. Identify IT Equipment and Software Opportunities

3. Use IT to Monitor and Control IT

4. Optimize Environmental Conditions

5. Manage Airflow

6. Evaluate Cooling Options

7. Reconsider Humidity Control

8. Improve Electrical Efficiency

9. Implement Energy Efficient O&M

Best Practices Summary


Most importantly:

Get IT and Facilities people talking and working together as a team!!!

• Ensure they know what each other is doing

• Consider impact of each on other, including energy costs

Data Center Best Practices


What are the barriers to success?1. Organizational Inertia and conflicting goals, risk aversion

2. Lack of funding

3. Lack of manpower / expertise

ESPCs can solve or help overcome barriers – at least 2 & 3!

2. Inflexible, flat funding stunts modernization of IT Lowest up-front cost vs lifecycle cost of IT and infrastructure

TMF/WCF offer flexible means to invest/innovate (supplement ESPCs?)

3. Optimization and the Cloud require new skills, processes

Virtualization, consolidation, push to the cloud are big lifts ($ and time)

Pay-as-you-go can be cheaper than in-house data centers

Hybrid / Public cloud adoption fosters innovation / further modernization

Barrier 1 and 3 require very strong and credible team

How can ESPCs help?


OverviewReplace existing constant speed fan motors on data center’s

computer room air handlers (CRAH) with variable speed fans.

Upgrade controls and increase temperature set point to 78°F.

Benefits to the ArmyDecrease cooling energy

Reduce fan energy

Improve data center efficiency and reduce PUE

Projected Year One Savings: $102K

Project Example – Army Ft Knox


Overview: $24M implementationConsolidate to two data centers.

Buildout and upgrades to existing data center.

Install a scalable Modular Data Center (MDC) to allow for geographic separation of IT assets.

Benefits to NASAHelp facilitate NASA JPL’s data center consolidation efforts.

Reduce NASA JPL’s data center-related costs, including utility & IT costs (e.g. reduce IT refresh and O&M costs).

Provide lower PUE data centers with more efficient cooling infrastructure.

Projected Annual Savings: $2.7 MillionEnergy savings (annual): $0.6M

O&M / IT savings (annual): $2.0M

Project Example – NASA Jet Propulsion Laboratory (JPL)


Critical Goals: reliability, sustainability, resiliency, and efficiency 95% of the ESPC is in a mission

critical data center with

comprehensive ECMs

Task Order awarded February, 2016

with a value of $114 Million.

Performance guarantee is

structured around ESCO

guaranteeing temperatures

on the server floor, uptime of critical equipment, and full O&M,

in addition to energy savings.

Guaranteed savings are $4.4 million/year

Project Example – Naval Base Coronado


• Overview: UESC

– Replace interior and exterior lighting, new controls installed

– Refurbish HVAC with new variable frequency drives on fans and pumps;

building rebalanced and major HVAC systems recommissioned

– Repair and balance economizer damper and VAV boxes for optimal cooling

• Benefits to the Army

– Optimized the site’s data center for efficiency and simplified future flexibility

– Achieved over 50% savings in natural gas

– Improved power usage effectiveness (PUE) by 33% from 1.9 to 1.6 – Actually

achieved even better – 1.52 PUE

– Exterior lighting improved by 70% with bi-level dimming LED parking lot lights

– Addressed maintenance, airflow, and comfort issues through upgrades and

retro-commissioning of systems.

• Projected Annual Savings: $8 million over 10 years

Project Example – Army Presidio of Monterey


• Separate agency organization (Facilities & IT) – ESPCs are traditionally employed by Building and Public Works departments,

while IT managers are the key decision maker.

– Split Incentive (Who pays utility bill? Who sets IT acquisition/ops policy?).

– The data center also has customers / stakeholders that want control.

– Performance should be enhanced, security increased (non-energy benefits

will often “sell” the project)

• Integrity of IT – Criticality of performance raises concerns with risk adverse staff that any

change could compromise IT integrity.

– Technical solution must be developed by data center experts and remain in

the control of the agency IT departments

• Unique implementation challenges– Keeping system current under program designed for long-term components

(IT refresh 3-5 years with lots of unknowns)

But there are few IT/Data Center ESPCs


• High cost of consolidation and optimization strategies can be too high to be financed by energy savings alone:

– O&M savings and/or appropriations may be needed to make a project

economic.

– Technology Modernization Fund (TMF) and IT Working Capital Funds

(WCF) are opportunities for leverage

Financing IT/Data Center ESPCs


• IT/data center ESPC projects can stand alone or be part of a comprehensive project including other building systems.

• IT projects can save a high percentage of energy

• IT projects can save a very high percentage of energy

• For VDI (Virtual Desktop Infrastructure), the savings can be over 90%.

• If cloud solution is preferred, energy savings can be a very high percentage.

– ESPC can help finance the move of equipment for colocation of

equipment/cloud service.

ESPC and IT/Data Center


DOE’s Center of Expertise

Featured

Resources

Navigation

Search

Featured

Activities

datacenters.lbl.gov

• FEMP provides

technical

resources and

assistance

through the

Center of

Expertise:

Datacenters.lbl.gov

datacenters.lbl.gov


Dale Sartor, P.E.Lawrence Berkeley National Laboratory

MS 90-3111

University of California

Berkeley, CA 94720

[email protected]

(510) 486-5988

http://datacenters.lbl.gov/

FEMP’s Data Center Team is ready to Help

http://datacenters.lbl.gov/

alternative financing of data center energy projects · 2020-01-03 · 1. data center energy: a....

Documents