energy efficient strategies for high density telecom applications alan m. lyons, david t. neilson...

Energy Efficient Strategies for High Density Telecom Applications

Alan M. Lyons, David T. Neilson and Todd R. SalamonBell Laboratories

All Rights Reserved © Alcatel-Lucent 2008

Why do we care about energy efficiency?

Greener Networks Overall power dissipation Network operator power dissipation is 1% of many countries energy

consumption Pressure to reduce network power consumption while still growing

network capacity and functionality

Scalability Power density in racks of communications equipment is reaching

practical limits. Makes cost-efficient scaling of telecom networks difficult

Increasing Thermal Energy

Transistor Package Circuit Pack Shelf Cabinet Central Office Environment


Thermal Management: Motivation

How can cooling technologies be extended to meet future power

density needs?

Source: ASHRAE Society Handbook, 2005

1. Increase equipment functionality and density

Meet future power density needs – exceeding 15,000 watts/cabinet

Maintain high reliability and meet acoustic noise limits.

2. Reduce carbon footprint and OPEX for our customers

Telecom consumes 3% of U.S. electricity Major Telcos spend > $1 Billion USD

annually on electricity

3. Meet or exceed regulations NEBS, ETSI & guidelines for energy

consumption and reduction

4. Differentiate ALU’s products

Telecom equipment vendors face the toughest thermal challenge and must lead the electronics industry.Telecom equipment vendors face the toughest thermal challenge and must lead the electronics industry.


What are we doing in Thermal Management Research?

Thermal Manageme

nt Research

Thermal Manageme

nt Research

Liquid Cooling Cabinet Architectures (with

CTO)

Liquid Cooling Cabinet Architectures (with

CTO)Waste Heat Recovery

Thermo-Electric Modules

Waste Heat Recovery Thermo-Electric

Modules

Increasing Cooling Efficiency

Increasing Cooling Efficiency

Extending the Limits of Air

Cooling

Extending the Limits of Air

CoolingThermal Interface

Materials Thermal Interface

Materials

Heat Sinks & Vortex GeneratorsHeat Sinks & Vortex Generators

Fan ReliabilityFan Reliability

Vapor ChambersVapor Chambers


Liquid Cooling Architectures for the Central Office

Motivation:

Thermal Density: Heat capacity of air limits total heat that can be removed at air flow rates that

meet acoustic noise specifications. Liquids have 103 higher heat capacity compared to gases

Energy Efficiency: Pumping Power required for air is >> than that for water Hot air leaks into cold aisles. Air must be chilled below ambient temperature to

insure sufficient cooling capacity

Challenges:

Insure system reliability, minimize costs, increase thermal capacity from 15 to 50kW

To: transferring waste heat into a liquid coolant and piping outside the

CO

From: dispersing waste heat into Central Office air


Design Example: Heat Exchangers Above Each Shelf

Approach: Heat-pipe based heat exchangers above

electronics extract heat from air Vapor in heat pipe transports heat

through cabinet walls into water side heat exchanger

Chilled water pumps heat to outdoor cooling tower.

Advantages:

Cabinet Level: Insures that cooling air is at ambient

temperature Reduces number of fans Isolates water from electronics

Room Level: Minimizes room air handlers – eliminates

large blower motors Eliminates need to chill air below ambient

temperature. No opportunity to intake hot room air

Ambient Airflow

Equipment shelves

Cooling LiquidSupply

Heat Exchangers

Heated waterreturn

Front ViewFront View Perspective Cut-AwayPerspective Cut-Away


Sealed Cabinet Advantages

Cold Airflow

Hot Airflow

Cooling WaterSupply

FinnedHeat Pipes

Heat Pipe Architecture(Open System)

Heat Pipe Architecture(Closed Recirculating

Air System)

Higher thermal densities can be accommodated both in the cabinet and Central Office/Data Center

Improved real-estate utilization Closer spacing of cabinets Eliminates need for raised floors

Higher reliability - ambient pollutants & dust not continuously drawn across equipment

Quieter operation at higher fan speeds. Closed cabinet reflects noise and enables acoustic foam installation.

Energy savings

• Eliminates need to under-cool air. Cooling

is provided directly to the cabinet

• Large air volumes need not be moved across large distances.

• Enables use of low-cost cooling towers and potential off-peak cooling.

Higher thermal densities can be accommodated both in the cabinet and Central Office/Data Center

Improved real-estate utilization Closer spacing of cabinets Eliminates need for raised floors

Higher reliability - ambient pollutants & dust not continuously drawn across equipment

Quieter operation at higher fan speeds. Closed cabinet reflects noise and enables acoustic foam installation.

Energy savings

• Eliminates need to under-cool air. Cooling

is provided directly to the cabinet

• Large air volumes need not be moved across large distances.

• Enables use of low-cost cooling towers and potential off-peak cooling.

(cut-away view)


Enhanced Cooling in a Sealed Cabinet Using an Evaporating-Condensing Dielectric Mist

Fan

Shelf 1

Shelf 2

Co

ld w

ate

r

Acoustic foam

Chiller unit

Ho

t wate

r

∞ ∞

Heat pipe

Heat pipe

Recirculating air

Fan

Shelf 1

Shelf 2

Co

ld w

ate

r

Acoustic foam

Chiller unit

Ho

t wate

r

∞ ∞

Heat pipe

Heat pipe

Recirculating air

Pump

Atomizer

Mist condenses on

heat pipes and falls by

gravity into collector

Mist from collectors

pumped to atomizer

Large droplets of mist from

atomizer directed into circuit packs

Pump

Atomizer

Mist condenses on

heat pipes and falls by

gravity into collector

Mist from collectors

pumped to atomizer

Large droplets of mist from

atomizer directed into circuit packs

Cabinet Level Circuit Pack Level Heat Sink Level

Objective: Limit temperature rise of air flowing through circuit packs by injecting atomized mist to increase its effective sensible heat capacity.

Approach: Inject atomized HFE7000 (environmentally benign dielectric fluid) upstream of circuit packs and/or high power component’s heat sinks. Condense vapor on finned heat pipes between shelves and recirculate it.


Calculated Optimum Droplet Diameter for Complete Evaporation at Outlet of a 32 mm Wide x 32 mm Long x 13 mm High Heat Sink and Cooling Capacity and Associated Parameters as Function of Mist Loading

Parameter Mist loading = 0.01 % by

volume

Mist loading = 0.1 % by volume

Total mass flow rate (gm/s) 0.435 0.871

Length of side of unit cell (in terms of droplet diameter)

17.3 8.04

Optimum droplet diameter (μ m) 83.2 74.9

RH after complete evaporation (constant volume, constant pressure) (%)

(2.8, 2.75) (28.1, 24)

Residence time in heat sink (msec) 31.5 22.5

Acceleration pressure drop (Pa) 0.0355 0.67

Evaporative cooling (W) 6.87 68.7

Parameter Mist loading = 0.01 % by

volume

Mist loading = 0.1 % by volume

Total mass flow rate (gm/s) 0.435 0.871

Length of side of unit cell (in terms of droplet diameter)

17.3 8.04

Optimum droplet diameter (μ m) 83.2 74.9

RH after complete evaporation (constant volume, constant pressure) (%)

(2.8, 2.75) (28.1, 24)

Residence time in heat sink (msec) 31.5 22.5

Acceleration pressure drop (Pa) 0.0355 0.67

Evaporative cooling (W) 6.87 68.7

Enormous potential for cooling enhancement (about 70 W) for modest pressure drop (< 1 Pa) and relative humidity (<25%) increases


Extending the Limits of Air CoolingInitial approach is to extend the limits of air cooling while maintaining

full NEBS/ETSI compliance…

…to allow thermal density increases without exceeding noise standards

Initial approach is to extend the limits of air cooling while maintaining full NEBS/ETSI compliance…

…to allow thermal density increases without exceeding noise standards

Heat Sink

TIM2

Silicon Die

IHS

Chip Carrier

TIM1

0.36 K/W

0.15 K/W

0.34 K/W

Heat Sink

TIM2

Silicon Die

IHS

Chip Carrier

TIM1

0.36 K/W

0.15 K/W

0.34 K/W

Schematic of Microprocessor

Cooling

Thermal Resistance

Stack

Thermal Resistance = (Tjunction – Tambient)/Power

60%

Improvement Targets: IC Package to heat

sink interface Fin to air interface Heat Spreading in

the base


Thermal Interface Materials (TIMs)Conventional TIMs: polymer-metal

composites Limits:

Thermal path limited by particle-particle point contacts: typical values 2-10 W/mK

Voids reduce reliability and achievable thermal properties

TIMs research program Measurements:

Designed and built a world-class test rig to verify and compare commercial materials

Materials: Developing new micro-textured TIM with

higher thermal conductivity, compliance and reliability

Maximize conductivity while minimizing assembly force.

First Results Thermal conductivity >4.5 W/mK with >60%

compression (>1.5mm compliance) Large experimental/modeling space to

explore

Pre

ssu

re (

MPa)

Compression (mm)Compression (mm)

Eff

ect

ive T

herm

al C

on

du

ctiv

ity (

W/m

K)

TIM test rig

conventional TIM

void

Effective conductivity and loading pressure versus

compression


Energy Harvesting Using Thermoelectrics

die

Thermoelectric Module

dieHot

Cold

VElectricity generated

by Thermo-electric Module

Vapor ChamberVapor Chamber

Goals: Reduce power required to cool

equipment by > 20% Convert waste heat into

electricityApproach:

Carnot Efficiency is inherently too low for CMOS devices - Target IC’s with junction temperatures > 300oC for adequate efficiency examples: SiC, GaN

Direct approach: Thermoelectric Modules to convert waste heat directly to electricity

Challenge: Efficiency of Thermoelectric

Module is low – research required for new materials and designs.

Thermoelectric Module approach


Network routers are responsible for much of ICT energy growth

Overall Network power growth In 2006 in Japan, IT and Communications equipment consumed about

45TWh or 4% of the total electricity generated or 1% of the country total energy consumption.

About ¼ of this i.e. 1% of power consumption is network operators

Japan's Ministry of Internal Affairs and

Communications Study Group on ICT System and

Network for Reducing Environmental Impacts,

March 2007

Packet switching responsible for much of growth

rate

Year

Energyusage[TWh]

Information and Communications Technologies Energy Usage


Scalability: Electronic Router Power Density Historical trend of x2 capacity every 18

months Not sustainable because of thermal

density Tb/s router today with 10-15kW is at

the limit Future growth thermal density

limited Shortfall of 30 fold in capacity by

2015 compared to historical trend for single rack router

Where the power dissipation is in core routers 2/3 of power associated with layer 3

function Packet Forwarding Engine dominates :

–Processing IP headers for destination and quality of service queuing

Eliminating L3 function in core would allow more scalability and lower overall power

22%

62%

16%

L1+L2

L3

Switch

Core Router Power

Consumption by Layer


Current Core Network - IP over DWDM

DWDM terminalor fixed OADM

Core IP router

Core IP routers connected by point to point DWDM All switching at a node uses router

Router capacity scales as node capacity

Traffic has multi hop routing Requires total router capacity in network to be larger than traffic by the average

number of hops

Legend


Network Evolution to Transport switching

OChDWDM terminal, ROADM Layer 1 Switch

Core IP router Layer 3

TMPLS Layer 2 Switch OCh

OCh

OCh

OCh

Packet Transport switch

Transparent flexible optical switching

Reduce switching requirement for Layer 3 IP routers Layer 1 ROADM: Wavelength channels bypass

nodes reduce number of hops for electronic switches

Layer 2 TMPLS: Packet switching network for transporting packets 1/3 of the power of Layer 3 switching

Layer 3 Routers only handle service layer not transport switching Primarily edge/service layer function not core switching

Does not require rapid scaling of core IP routers Uses lower-power switching elements

Architecture change is already happening: Qwest runs it’s Juniper T640’s as MPLS routers in the core (Poll: OFC 2008 plenary)

However they are still using core

routers


Summary

Why do we care about energy efficiency? Financial and environmental cost of energy usage Practicality of cooling high-energy-density equipment

Thermal Management Solutions Liquid cooling solutions for cabinet architectures Extending the limits of air cooling

Thermal interface materials (TIMs)

Energy harvesting Thermoelectric modules

Network Evolution to Reduce Load on Layer 3 Routers Combine Layer 1 ROADM and Layer 2 TMPLS to

reduce energy usage of Layer 3 Routers Expected reduction of 30 to 50 percent

OCh

OCh

OCh

OCh

Energy-efficient switching architecture

Enhanced cooling using a dielectric mist


Acknowledgements

Vaihbav Bahadur

Martin Cleary

Cormac Eason

Ryan Enright

Marc Hodes

Domhnaill Hernon

Roger Kempers

Paul Kolodner

Shankar Krishnan

Wei Ling

Sal Messana

John Mullins

Paul Rominski

Patricia Scanlon


Additional Material


2 GPM water at 10oC

adiabatic wall

•Fin gap

•FIN THICKNESS

Air sideheat fluxinto eachof three

heat pipes

Fixed 76

mm Stack

of 3 Heat Pipes

HEIGHT

Assumptions for liquid heat transfer Flow rate of water at 2 GPM and 10oC Fin gap of >3 mm (maximize number of fins) Area for water flow: 50mm x 50mm

Governing equations for steady laminar flow Mass conservation (incompressible fluid)

Momentum conservation

Energy conservation

Example optimal design problem: Determine heat exchanger fin height and thickness

0 v

TkTvC p

Tvv

pvv

where

,


Physics-based optimization is applicable to a range of other heat transfer problems, e.g., Transient enhancement of heat transfer Three-dimensional heat sinks

Very large potential design space

Essential design tool to minimize time and expense

Characteristics of optimal geometry:Minimizing heat pipe temperature rise

Temperature contours along middle fin for optimal geometry

EC

HIP

1.0

1.5

2.0

2.5

3.0

finth

ick

80 90 100 110 120 130 140 150

finheight

delt_fixedpitch

Value Low Limit High Limit 10.32 9.87 10.76

finheigh=135.61 finthick=1.63

Contours of Average Heat Pipe Temperature Rise (oC)

fin height [mm]

fin

thic

kness

[m

m] local minimum

in design space(20% improvement

relative to otheradmissible

design choices)


Current Core Network Example Example of a core network

800Gb/s capacity packet network from Paris to Barcelona

Power Consumption 800Gb/s Optical transport on protected ring

between 8kW Terabit class routers in Paris and Barcelona

26kW Cooling and power supply equipment x2 =

68KW.

Assume 30% link utilization this gives power consumption of ~300W per Gb/s

Of the core power 25% is transport, 75% switching (33% layer 1 and layer 2, 66% is layer 3 IP).

Half of the network power consumption is because of Layer 3 functionality

Paris Barcelona

energy efficient strategies for high density telecom applications alan m. lyons, david t. neilson...

Documents

cooling air

heat exchangers

heat pipe transports

higher heat capacity

air vapor

thermal density

dispersing waste heat

thermal capacity