energy dissipation is a severe limit to push further the ... · pdf file– drive...

© 2011 IBM Corporation

Energy Dissipation is a Severe Limit to Push Further the Computation Performance of ICT Devices: Towards Zero-Emission Datacenters

Dr. Bruno Michel, IBM Research – Zurich, Manager Advanced Thermal Packaging May 06, 2011

© 2010 IBM Corporation2 IBM Research Zurich, Advanced Thermal Packaging

Green Datacenter Market Drivers and Trends

Increased green consciousness, and rising cost of power

IT demand outpaces technology improvements – Server energy use doubled 2003-2008; temporary slowdown due to

economic crisis; resumed growth is not sustainable – Koomey Study: Server use 1.2% of U.S. energy

ICT industries consume 2% world wide energy– Carbon dioxide emission like global aviation

Real Actions NeededBrouillard, APC, 2006

Future datacenters dominated by energy cost; half energy spent on coolingSource IDC, 2009


From Individual Transistors to the GlobeMorteratsch Glacier, Switzerland, 1985

Morteratsch Glacier, Switzerland, 2007

- Thermal issues propagate up to the world climate- Global length- and decade time-scales involved - A holistic view is required to solve these problems

IT to become part of the solution to climate challenge!

100

kmDa

y

5’00

0 km

Year

40’0

00 k

m10

0 ye

ars

Glacier recession 1985-2007: 400 meters!

Pow

er

Stat

ion

Cont

inen

tal

Pow

erNe

ts

Glo

bal

Tem

pera

ture


Motivation for Chip Liquid Cooling

Superior thermal properties of liquids compared to airDisadvantage: Increased complexity

Biological vascular systems optimized for mass transport at low pressure

>700 W/cm2 cooling capacity: ΔT = 65K, Δp <0.25 Bar, and branching manifold <1.5 mm

Impinging liquid jets for high heat transfer from hot chip

Thermal conductivity [W/(m*K)]

Volumetric heat capacity [kJ/(m3*K)]

Air 0.0245 1.27

H2 O 0.6 4176


Zero-Emission Data Centers

High-performance chip-level cooling improves energy efficiency AND reduces carbon emission:– Cool chip with ΔT = 20ºC instead of 75ºC– Save chiller energy: Cool with T > 60ºC hot water– Re-use: Heat 700 homes with 10 MW datacenter

Need carbon footprint reduction– EU, IPCC, Stern report targets– Chillers use ~50% of datacenter energy – Space heating ~30% of carbon footprint

Zero-emission concept valuable in all climates– Cold and moderate climates:

energy savings and energy re-use– Hot climates: Free cooling, desalination

Europe: 5000 district heating systems– Distribute 6% of total thermal demand– Thermal energy from datacenters absorbed


Aquasar Hot-Water Cooled HPC Cluster for ETH

Aquasar connected to space heating at ETH

Experimental validation: Air vs. cold, vs. hot water cooling

World record in facility-level Efficiency ERE: 7.9 TFLOP/gCO23x smaller datacenter energy cost (saved energy and sold heat)Operative since May 6th, 2010 (Anniversary today)

– Two chassis liquid cooled (green) and one air cooled (red)– Storage server and Infiniband switch (cyan) – Closed cooling loop with 20 liters water and 30 liters per min. flow– Recover approximately 80% of waste heat @ 60ºC

33 QS22 and 9 HS22 Servers


SuperMUC 3 Petaflop Supercomputer at LRZ In 2012, LRZ (Leibnitz Rechenzentrum) builds a supercomputer based on IBM System x iDataPlex with 19476 CPUs / 157464 Cores, 324 TB Memory, and 10 PetaByte File Space

“SuperMUC” provides previously unattainable energy efficiency by exploiting new Intel processors and innovative IBM hot water cooling.

– Hot Water Cooling technology invented for Aquasar

First general available Hot Water Cooled iDataPlex clusterMost Energy Efficient high End HPC System:

– PUE 1.1 – Green IT– 40% less energy consumption compared to air-cooling– 90% of waste heat will be reused

The system is part of the Partnership for Advanced Computing in Europe (PRACE) HPC infrastructure

Cooling at Rack Level with Rear Door Heat Exchanger

Cooling at Component Level with Hot Water Cooling Technology


“Value” of Heat in Different Climates

0

10

20

30

40

50

60

70

80

90

-5 20 45

savi

ngs

[%]

Electricity consumptionCO2-emissionEnergy costNet energy cost

Montreal / Anchorage Dubai / Las VegasBerlin / Washington

Basic concept – Moderate climates: energy savings and energy re-use – Hot climates: free cooling on hottest day ever recorded (57ºC)

Longer term developments– Drive desalination with datacenter heat– Drive adsorption chiller with hot water cooled part – Best in cold climate: Space heating– Best in hot and dry climate: Desalination– Best in hot climate: Sorption cooling

Heating degree days (red) and cooling degree days (blue) for different cities

PUE and reduction of energy cost – a) reference (red), direct heat reuse (green),

and with cooling (blue) – b) Energy cost reduction with heating (green)

and cooling (blue) Emission reduction (light green / blue)

0.0

0.5

1.0

1.5

2.0

2.5

New York Washington Texas

PUE

eff [

-]

0

10

20

30

40

50

60

70

80

New York Washington Texas

redu

ctio

n [%

]


Paradigm Change: Vertical Integration

Global wire lengths reduction

Meindl 05 et al.

System on Chip

3D Integration

Multi-Chip Design

Benefits:High core-cache bandwidth

Separation of technologies

Reduction in wire length

Equivalent to two generations of scaling

No impact on software development

Brain: synapse network


Limits of Traditional Back-Side Heat Removal

Microchannel back-side heat removal

Heat removal limit constrains electrical design


Scalable Heat Removal by Interlayer Cooling3D integration requires interlayer cooling for stacked logic chips Bonding scheme to isolate electrical interconnects from coolant

Through silicon via electrical bonding and water insulation scheme

A large fraction of energy in computers is spent for data transportShrinking computers saves energy

cross-section through fluid port and cavities

Test vehicle with fluid manifold and connection

MicrochannelPin fin


Key Messages and Outlook

Scale up to full size data centersVision of a Zero-Emission DatacenterIT driven solutions for the climate challenge

Reduce emission up to 85% through heat re-useSave 40% energy and reduce energy cost by 2-3 x Roadmap for large efficiency increase in next 15 years

– Interlayer cooling of 3D chips

(149 F)

(140 F)


Summary

Paradigm changes– Energy will cost more than servers– Coolers are million fold larger than transistors

Moore’s law goes 3D– Single layer scaling slows down– Stacking of layers allows extension of Moore’s law– Reach functional density and connectivity of Human brain

Future computers to look different– Liquid cooling and heat re-use: Aquasar / SuperMUC– Interlayer cooled 3D stacks– Bionic designs Smart Energy

Energy aspects and communication are key– Cooling - power delivery – photonics– Shrink a rack to a “sugar cube”: 50x efficiency– Synergy with solar power Smart Planet


Acknowledgment

Advanced Thermal Packaging: Thomas Brunschwiler, Ingmar Meijer, Stephan Paredes, and Werner Escher, Javier Goicochea, Patrick Ruch, and Brian SmithAquasar FOAK: Bruno Battaglia, Peter Buhler, Marcel Bürge, Ralph Heller, Daniel Jehle, Markus Kirschner, Urs Kloter, Thorsten Kramp, Marcus Oestreicher, Walter Riess, Daniel Ronzani, Sandra Rufener, Hansruedi Steinauer, Erich Rütsche, Jörg Schanze, Gerhard Schoor, Jürgen Stumpp, HeinerTschopp, Kurt Wasser, and Martin WitzigIBM Boeblingen (Germany): Martin Bachmaier, Gottfried Goldrian, Michael Malms, Jürgen Marschall, Oliver Rettig, Manfred Ries, and Walter WeberCCEM and CMOSAIC Partners: Dimos Poulikakos, Peter Kasten, Fabio Alfieri, Ming Hu, Igor Zinovik, Manish Tiwari, Severin Zimmermann, Petros Komoutsakos, and Wendelin Stark (ETH), and John Richard Thome, Jonathan Olivier, Navid Borhani, and Yassir Madhour (EPFL) Microfabrication: Rene Beyeler, Daniele Caimi, Ute Drechsler, Urs Kloter, Richard Stutz, Kurt Wasser, and Martin WitzigIBM Research Yorktown (USA): Paul Andry, Evan Colgan, Claudius Feger, Winfried Haensch, HendrikHamann, Theodore vanKessel, Ken Marston, Yves Martin, John Maegerlein, and Thomas TheisIBM Server and Technology Group in East Fishkill and Poughkeepsie (USA): Pepe Bezama, Kamal Sikka, Michael Ellsworth, Roger Schmidt, and Madhu IyengarIBM other locations: Dave Frank, Vinod Kamath, and Hannes EngelstaedterFinancial Support: IBM Zurich Research Laboratory, IBM Research FOAK program (Aquasar Project), Swiss Government KTI Projects, Swiss Government CCEM Project, Swiss Government NanoTera Project, EU FP7 project NanoPack


Thank You for Your Attention!

Dr. Bruno Michel – Manager Advanced Thermal Packaging

May 06, 2011

energy dissipation is a severe limit to push further the ... · pdf file– drive...

Documents