review of cooling technologies for computer products: past...

Systems and Technology Group

© 2003 IBM Corporation

CMRR SeminarUniversity of California at San Diego

May 31, 2005

Review of Cooling Technologies for Computer Products:Past, Present and Future

Richard C. ChuIBM FellowPoughkeepsie, NY



Overview

Thermal interfacesHeat spreadingAir coolingIndirect and direct water coolingImmersion coolingMicro-scale cooling Refrigeration coolingSolid State Cooling Data Center Cooling

Cooling will play a pivotal role in the coming decade for all types of electronic products. Increased heat fluxes at all levels of packaging from chip to system to facility pose a major cooling challenge. To meet the challenge significant cooling technology enhancements will be needed in each of the following areas:



OutlinePower Trends

Chip levelModule levelRack level

Module Level CoolingInternalExternalImmersion

System Level CoolingHybrid Air/LiquidLiquid CoolingRefrigeration

Data Center Cooling

Future Cooling Needs

Conclusion



OutlinePower Trends

Chip levelModule levelRack level

Module Level CoolingModule Level CoolingModule Level CoolingInternalInternalInternalExternalExternalExternalImmersionImmersionImmersion

System Level CoolingSystem Level CoolingSystem Level CoolingHybrid Air/LiquidHybrid Air/LiquidHybrid Air/LiquidLiquid CoolingLiquid CoolingLiquid CoolingRefrigerationRefrigerationRefrigeration

Data Center CoolingData Center CoolingData Center Cooling

Future Cooling NeedsFuture Cooling NeedsFuture Cooling Needs

ConclusionConclusionConclusion



Chip Heat Flux Trend

Year of Announcement1996 1998 2000 2002 2004 20060

10

20

30

40

50

60

Freeway

GP (1.3 GHz)Pentium 4

Pentium 3

McKinley zSeries

i & pSeries

SunUltraSPARC

Air Cooling LimitAir Cooling Limitfor processorsfor processors

Intel

Chi

p H

eat F

lux

(Wat

ts/s

q cm

)



General Availability - Year2010200019901980197019601950

15

10

5

0

Mod

ule

Hea

t Flu

x (

Wat

ts/c

m )2

Bipolar

IBM ES9000

VacuumTube IBM 360

IBM 370 IBM 3033

Honeywell DPS-88Hitachi S-810

NEC LCM IBM RX3

IBM RX4

IBM RY6

IBMRY5

CMOS

Fujitsu M380

IBM 3081 TCMIBM 4381

CDC Cyber 205

IBM 3090 TCM

Fujitsu M780

NTT

IBM 3090S TCM

Fujitsu VP 2000

Pentium 4 Xeon Dp

Itanium 2

Pulsar

IBM RY7

Module Heat Flux Trend

Steam Iron

Module Powers Shifted ~12 years from Bipolar to CMOS



Rack Level Heat Load TrendRack Level Heat Loads Rising at an Exponential Rate

Year of Announcement1994 1996 1998 2000 2002 2004 2006

5

10

15

20

25

30

35R

ack

Leve

l Hea

t Loa

d (k

W)



Rack Level Heat Load Density (ASHRAE)

Shark Silvertip

xSeries Blades(Nacona)

pSeries p5-575

Megamouth

FastT 700

Shark Silvertip

xSeries Blades(Nacona)

pSeries p5-575

Megamouth

FastT 700



OutlinePower TrendsPower TrendsPower Trends

Chip levelChip levelChip levelModule levelModule levelModule levelRack levelRack levelRack level

Module Level CoolingInternalExternalImmersion



Future NeedsFuture NeedsFuture Needs




Module Level Cooling

Generic Module

External

Internal

Heat Sink

External Interface(TIM2)

Lid or CapSubstrate

ChipInternal Interface (TIM1)



Module Level Cooling – Internal

Micro-FinInsert

Substrate

Module CapSpring

Hitachi M-880 Module

IBM Thermal Conduction Module

1980’s

Substrate

Piston

Module Cap

Gas or OilChip

Spring




1990’s

IBM Flat Plate Cooling

Module Cooling Hat

Substrate

C-Ring Seal

Semiconductor Chip

Decoupling Capacitor

Advanced Thermal Compound External Heat Removal

Surface

CLBase Plate

Note: Not to scale.




SiC Spreader(240 W/m-K)

Glass Ceramic Substrate

2000’s

Substrate

Cap

AdhesiveThermalInterface

PasteInterface

Chip

Heat Spreader



Module Level Cooling – External

Extruded aluminum

Higher aspect ratio extruded aluminum

Folded aluminum fin on aluminum base- Soldered fins

Folded aluminum fin on copper base

Heat sinks with enhanced thermal spreading- Embedded heat pipes- Vapor chamber base

All-copper heat sinks w/ and w/o thermallyenhanced base

Air Cooled Heat Sink Progression



Module Level Cooling – Liquid Cold Plates

Separable Cold Plate Integral Cold Plate

Individual Chip-Scale Cold Plate



Future Module Cooling

Direct Water Jet Impingement

Silicon Microchannel Cold Plate



Module Level Cooling – Immersion Cooling

Liquid Encapsulated Module(LEM)

Boiling on Si SurfaceIn Fluorocarbon Liquid



Module Level Cooling – Immersion Cooling

InletOutlet

Substrate

chip chip

InletOutlet

Coolingcap

Substrate Chip

Board

InletOutlet

CoolingcapCoolingcap

Substrate Chip

Board

Jet Impingement Evaporative Spray Cooling






System Level CoolingHybrid Air/LiquidLiquid CoolingRefrigeration






System Level Cooling – Hybrid Air/Liquid

Interboard Heat Exchanger

1960’s



System Level Cooling – Hybrid Air/Liquid

Air / LiquidHeat Exchanger

Air Moving Device

Centrifugal Pump

ColdPlate

ColdPlate

Reservoir

Schematic

Apple G5 2.5 GHz Liquid Cooling System

2000’s



System Level Cooling – External Liquid

Coolant Distributionto TCM Cold Plates

Coolant DistributionUnit (CDU)



System Level Cooling – RefrigerationCryogenerator

SCRControlPanel

MainControlPanel

Storrage andDistribution Skid

Computer Room

Mechanical RoomCPUCabinet

CPUCabinet

Cryostat

Cryostat

V-J Lines

ValveBox

Control Data / ETA-10Cryogenic System Configuration

Processors immersed in liquid N2 (77K)Peak nucleate heat flux limit ~ 12 W/cm2

1980’s



System Level Cooling – Refrigeration

Power

Modular Cooling Units(MRUs)

Front Side Back Side

Refrigerant(R134A)

Lines

I/O Cage

Evaporator(MCM behind)

Memory Cards

Blowers

IBM Z990 System

1990’s

IBM Z900 System







Data Center Cooling





Data Center Cooling – How the Data Center Works

CRAC CRAC

Subfloor

Raised FloorPerforated Floor Tiles (can be 6, 11, 25, 40, 60% open)

The key requirement maintain rack inlet temperatures within manufacturers specifications

2 separate complex thermo-fluid problems- (1) Underfloor air distribution (2) Above floor air/temperature distribution

CI CIHI HIHI



Data Center Cooling – How the Rack Works

Front Cold

BackHot

Cable Opening

Subfloor Underfloor Chilled Air

Air flow

Perf tile Tile floor

What are the requirements for cooling our servers?

Inlet air temperatures maintained at or below 35 oC temperature for elevation below 3k feet, or 32 oC for elevation between 3k and 7k feet

Humidity 8-80%

Front to back cooling- the most efficient method available

Plenty of space in front and rear as well as over head to handle flow

Enough CFM (cubic feet per minute-volume) of air to supply the server-this varies by server

Cold Aisle Hot Aisle



The Problem in the Data Center Chilled air does not reach top of racks

ColdAisleColdAisle

ColdAisle

ColdAisle Hot

Aisle



Rack / Data Center Water CoolingRack / Data Center Water Cooling

Chilled Water

Data Center Processor Chilled Water

Water cooledCold Plates within Node

Cover Heat Exchanger

Side Car Heat Exchanger

CommonCRWC

Cold Plate

Dual loop

Building utility or local chiller

Top ViewsFront

Rear

CRWC CRWCCRWC CRWC

CRWC

Base Products Remain Air Cooled

Air Flow

ChilledWater

CRWC: Computer RoomWater Conditioning Unit



BaseRack

Top View

Programs that applyAny x,i,p,z > 12 kW

Heat Exchanger in Door

Heat Exchanger

Rear Door Heat ExchangerRear Door Heat Exchanger



HeatExchanger

Side Car Heat ExchangerSide Car Heat Exchanger

REGATTA FRAME

HEAT EXCHANGER FRAME

HEAT EXCHANGER

SHIPPING FRAME

COVERS

Base Rack

Closed system also provides reduced acoustics and EMC emissions



Vendor Data Center Solutions

Apple Supercomputer cooled with pumped refrigerant Overhead

Water Cooled Coilin Bottom of Server Rack

Water Cooled CoilInside of Server Rack



IBM Sponsored Heat Transfer Research

Optimal PCB spacing- Natural and forced

convection

Duke UniversityA. Bejan

Channel flow- CFD code verification

University ofArizona

A. Ortega

Channel flow- Entrance, periodic

and fully developed

University ofRhode Island

M. Faghri

Channel flow- Module height effects

Upstream heat transferDownstream heat transfer

Clarkson UniversityUNV - Reno

R. Wirtz

Channel flow- Adiabatic heat

transfer coefficient

Stanford UniversityR. Moffat

Natural convectionChannel flow- Effect of barriers- Effect of missing modulesImpinging flow- Pin fins- Straight fins

University ofMinnesota

E. Sparrow

Area of ResearchUniversityInvestigator

Pool boiling- Enhancement

University ofMinnesota

T. Simon andA. Bar-Cohen

Flow boilingThermosyphon cooling

University ofCalifornia atBerkeley

V. Carey andC.L. Tien

Liquid jet impingement- Single phase and boiling- Single and multiple jets

City College ofCity UniversityOf New York

L. Jiji andZ. Dagan

Falling liquid filmPool boilingFlow boiling

Purdue UniversityI. Mudawar

Forced convection- Single phase- Extended surfaces- BoilingLiquid jet impingement- w/o Enhancement- w/ Enhancement

Purdue UniversityF. IncroperaandS. Ramadhyani

Natural convectionForced convectionPool and flow boilingBoiling enhancement- Ultrasonic- Surface treatment- Extended surfaces

MIT, Iowa StateUniversity, andRPI

A. Bergles

Area of ResearchUniversityInvestigator

Air Cooling

Liquid Immersion Cooling




Thermal SpreadersInexpensive, high thermal conductivity with closer TCE match to siliconAlgorithms for optimizing thermal/mechanical design of thermal spreadersImproved thermal spreading within a chip – alleviate hot spotsCorrelations and Analytical models of dryout and rewetting of micro-channelsand micro-porous structures to facilitate design of micro-heat pipes

Thermal InterfacesThermal pastes, epoxies, and elastomers with high thermal conductivity nanoparticlesNew interface materials based on carbon nanotubes and other materialsNovel techniques/materials to minimize bonded interfacial stresses Correlations and analytical relations to predict fatigue life of bonded interface materialsStandardized method to characterize thermal interface performanceSelf contained solid-to-solid phase change materials or micro-encapsulated materials as suitable interface materials for a range of applications including harsh environments



Future Cooling Needs (continued)

Heat Pipes

Flexible heat pipes

Heat pipes that handle high heat fluxesLow cost heat pipes that can transport heat effectively over large distances (>0.5 m)Designs to reduce the gravitational orientation impact on heat pipe efficiency, especially for avionics applicationsHeat pipe technology capable of withstanding harsh environmentsSound numerical models and optimization tools for predicting the performance andoperational limits, including dry-out, in heat pipesCorrelations and algorithms for thermosyphon (i.e. wickless heat pipe) designs




Air Cooling

Models and correlations to predict heat transfer in transition and low Reynolds numberflow over packages and in heat sink passages

Low Reynolds number turbulence models for use in CFD codes

Heat sink design and optimization procedures for the minimization of heat sinkthermal resistance subject to mass and volume constraints

Advanced manufacturing techniques for metal and composite material heat sinks

Concepts for higher head-moderate flow, low noise, compact fans

Novel, low power consumption, low acoustic emission micro-fans for forced convectioncooling in notebook computers and handheld electronics including low-frequency andultrasonic piezoelectric fans

High pressure/high flow blowers with low acoustical power




Water CoolingMiniaturized components that have high reliability and provide enhanced performance(e.g. pumps and heat exchangers)MEMS and meso-scale components to create low cost, low noise, water-to-airheat exchangersMEMS and meso-scale components to create low cost, package-size cold platesMicrochannel heat sinks with novel integrated micropumps to minimize packagevolume for high heat flux applications Methods to enable direct water cooling of chips or chip packages

Direct Liquid ImmersionSingle + two-phase heat transfer correlations for new families of dielectric coolantsNanofluids: nanoparticles in dielectric coolants to enhance heat transfer characteristicsConvective + phase change correlations that account for highly non-uniform HF conditions CHF models to account for highly non-uniform heat fluxesCharacterization of boiling and two-phase flow in narrow passages and 3D structuresMEMS/meso-scale components to enhance convective, pool + flow boiling heat transfer Correlations and models for evaporative spray cooling heat transfer




Sub-Ambient and Refrigeration CoolingHighly reliable miniaturized components such as compressors, condensers, and evaporatorsMEMS / meso-scale components to create low-cost, low noise refrigerators usingsolid-state, vapor compression, or absorption cyclesMEMS / meso-scale components to create low-cost, pacakge-size cold plates New thermoelectric materials and fabrication methods that can improve theperformance of thermoelectric coolers

Low Temperature RefrigerationApplication of auto-refrigerating cascade (ARC) systems to provide low temperaturecooling of electronics packagesApplication of mechanically cascaded (2-stage) refrigeration systems to provide Low temperature cooling for electronic packages








Future NeedsFuture NeedsFuture Needs

Conclusion



Future Cooling Technologies and Strategy

Strategy for the Future- Explore all options- Establish a closer working relationship with vendors- Pool resources to fund cooling technology development- Get university/research labs involved

Enhanced Air Cooling Technology and System- High performance heat sink- Mini air movers for local enhancement- Higher pressure air movers and higher volume air flow systems- Highly parallel flow distribution system- Active redundancy with control

Other Candidate Cooling Technologies- Direct liquid cooling technology - for high performance applications- Heat pipe and vapor chamber cooling technology- Thermoelectric cooling technology - for special situations

- Low temperature cooling technology - for performance enhancement- Self-contained, low cost liquid cooling technology- Thermal interface enhancement technology



Low cost, high performance, direct immersion cooling technology

Low cost, high performance thermal interface (10X) technology

Low cost, high performance cold plate (5X) technology

Low cost, high performance heat sink (5X) technology

Low cost and low noise (2X), high performance (2X) air/liquid moving device technology

Low cost, high performance, scalable cooling system

Low cost, high performance, future data center cooling concept

Significant Challenges for Electronic CoolingTechnology in the Coming Decade



“Imagination is more important than knowledge.”Albert Einstein

“Everyone is trying to accomplish something big,Not realizing that life is made up of little things.”

Frank A. ClarkCowles Syndicate

“Perseverance – There is no substitute for hard work.”Thomas A. Edison

“The harder you work the luckier you get.”Gary Player



Career = f ( EQ, IQ, AQ, WQ, …)

Where, EQ = Emotional quotientIQ = Intelligence quotientAQ = Adversity quotientWQ = Work quotient



Do not look back to the goodold days; instead, look ahead

to the better new days.

review of cooling technologies for computer products: past...

Documents