!1

100%Warm-WaterCoolingwithCoolMUC-3Nochillers,nofans,noproblems…

AxelAuweter–MEGWARECTO WorkshoponEnergyEfficiencyinHPC

May30,2018,Ljubljana

• Providerofturn-keysolutionsforcompute,

storage,andacceleratorclusters

• Morethan1,200HPCdeploymentssince1990

• ̴50employees

• StrongfocusonR&D

• Morethan8yearsexperienceinliquidcooling

technologiesforenergyefficiency!

MEGWAREComputerVertriebundServiceGmbHCompanyOverview

!2

AJointVision...…forGreenDataCentersoftheFuture

!3

TechnologyPartners

EarlyAdopters

•

•

•

Chiller-FreeCooling…forimprovedenergyefficiencyratio(EER)

!4

Vapor-CompressionRefrigeration Chiller-FreeCooling

14ºC

10ºC

45ºC

40ºC

Compressor

ExpansionValve

CondenserEvaporator

HeatExchanger

DataCenter OutsideAir DataCenter OutsideAir

• ASHRAEspecifies

designpointsfor

coolingequipment

• Useofpercentiles:

specifiedvalueis

reachedor

exceededforatotal

of3hours(0.4%),7

hours(1%),or14

hours(2%)

TemperatureDesignPointChartsLjubljana,Slovenia

!5

0

10

20

30

40

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

0.4%Dry-Bulb 1%Dry-Bulb 2%Dry-Bulb

• ASHRAEspecifies

designpointsfor

coolingequipment

• Useofpercentiles:

specifiedvalueis

reachedor

exceededforatotal

of3hours(0.4%),7

hours(1%),or14

hours(2%)

TemperatureDesignPointChartsLjubljana,Slovenia

!6

0

10

20

30

40

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

0.4%Wet-Bulb 1%Wet-Bulb 2%Wet-Bulb

ThermalConductivitydependson:• ΔTbetweenheatsourceandsink

• Geometry(transfersurfaceA&distanced)

• Material

ThermalCapacitydependson:• Mass

• Material

TransferringHeat…intheoryandpractice!

!7

HeatSource

HeatSink

TransferMaterial

d

A

max. 7 K in Ljubljana (for air-cooling)

Air (0.0026 W/mK)

HeatSource

HeatStorage

Air (1.005 J/kgK)

Water (0.598 W/mK)

Water (4.128 J/kgK)

Copper (401 W/mK)

=1.2 kJ/m3K=4178 kJ/m3K

Aluminum (237 W/mK)

• Goingtowater-coolingsignificantly

facilitateswaste-heatreuse

• Highertemperaturesenablemoreoptions

WasteHeatReuse…you’vealreadypaidforthepower-useit!

!8

DistrictHeating >70ºC

HouseholdRadiator 60ºC

UnderfloorHeating 35ºC

AbsorptionChiller >70ºC

AdsorptionChiller >50ºC

MEGWARECooLMUCSupercomputer(2012)withSorTechACS08adsorptionchiller.

PhotocourtesyofLeibnizSupercomputingCentreoftheBavarianAcademyofSciencesandHumanities.

PowerBreakDownofaTypicalComputeNode…accordingtocomponentmaxTDPs

!9

Others

PSU

Interconnect

Mainboard

MemoryCPU&VRs

BasedonMegwareDualSocket“Skylake”6140,192GBRAM,Omni-PathHFI

EvolutionofMEGWAREColdCon®DirectLiquidCoolingsince2011

!10

ColdCon®2011

Liquidcoolingfor • CPUs • VRs

Maxinlettemp:40°C

ColdCon®2013

Liquidcoolingfor • CPUs • VRs • DIMMs

Maxinlettemp:40°C

ColdCon®2015

Liquidcoolingfor • CPUs • VRs • DIMMs • I/O

Maxinlettemp:50°C

ColdCon®2017

Liquidcoolingfor • CPUs, VRs • DIMMs • I/O • Powersupplies

• NetworkSwitches Maxinlettemp:55°C

• Upto10nodeswithyourchoiceof- Intel®Xeon®Scalable

- Intel®Xeon®Phi™x200

- Intel®Xeon®E5v4

• Highperformancenetwork- Intel®Omni-PathFabric

- IBFDR,EDR

• Integratedchassismanagement

• Hotpluggable,easymaintenance

MEGWARESlideSX-LCFullyintegratedchassissolutionfordirectliquidcooling!

!11

MEGWAREColdCon®…forComputeNodes

!12

• HighPerformanceHeatSink- e.g.CPUs

• DirectAttachCopper- e.g.DIMMCooling

• ColdPlate- e.g.NICs

• PassiveCooling- Yourmainboardconductsheataswell!

• 1620WperPSU

• Upto5PSUsperSlideSXChassis

• 80PlusPlatinumCertified

• LiquidCoolingwithhot-plugsupport

MEGWAREColdCon®…forPowerSupplyUnits

!13

• RefitofIntel®OPAswitch48-portfor

ColdCon®directliquidcooling

• 12VDCpowersupplyviaMEGWARE

liquidcooledpowerbank

MEGWAREColdCon®…forOmni-PathHighPerformanceNetworkSwitches

!14

• 15MEGWARESlideSX-LC®chassis

• 148computenodeswithIntel®Xeon®Phi™Processor7210F

• Dual100GBpslinkstoIntel®Omni-Pathfabric

• MEGWAREColdCon®forpowersupplies

• MEGWAREColdCon®forIntel®OPAswitch

• MEGWAREEnergyMeterforhigh-frequencypowerand

energymeasurements&Intel®NodeManagerintegration

• Thermalinsulationforracks

• Lessthan3%heatlosstocomputerroomair

TheNewColdCon®inActionCooLMUC-3Supercomputer@LeibnizSupercomputingCentre

!15

• Year-roundchiller-lesscoolingfor

theentiresystem

• Highoutlettemperaturesfor

optimalheat-reuse

• Insulatedracksforclosetozero

heatemissiontoroomair

• Additionalpowersavingsdueto

removaloffans

100%Warm-WaterCooling…whyyouneedit,too…!

!16

!17

100%Warm-WaterCoolingwithCoolMUC-3Nochillers,nofans,noproblems…

Thankyou!