LL Bean Presentation

Cooling:Best Practices and Economizers1Randall PoetA C Systems5/1/20121Server Load = CRAC CapacityServer Airflow = CRAC Airflow2Ideal Situation2The Conventional Approach Base CaseRaise the Return / Supply Air TempContain the Cold AisleAdd Intelligent ControlThe Set-upSummary3Agenda3Hot aisle / Cold aisleBlanking panels in racksBlanking panels between racksBlanking Panels between racks and floorCable cut-outs coveredRelatively clean underfloor areaProper location of CRAC unitsVapor barrier around space4Start with Best Practices 420 Racks8 kW each35F delta T 5Base Case 75F Return Air Control4 CRACs (N+1)54 kW each21F delta T design

520 Racks8 kW each35F delta T 6Base Case 75F Return Air Control3 CRACs Running54 kW each21F delta T design

6Unit Airflow 24,000 CFMRack Airflow 14,600 CFMBypass Airflow 9,400 CFM7Base Case Operating Scenario75F89F54F54FCFM 100%78The Airflow Pattern

89Typical Air Cooled DXEnergy ConsumptionEvap Fan MotorCompressorsCondenser Fan Motors9Compressors run at full capacityFans run at full speed10Typical Energy ConsumptionEvap Fan MotorCompressorsCondenser Fan MotorsSystem kW75F RACompressors35.4Evap Fans8.7Condenser Fans5.4Total 49.510The Conventional Approach Base CaseRaise the Return / Supply Air TempContain the Cold AisleAdd Intelligent ControlThe Set-upSummary11Agenda11Unit Airflow 24,000 CFMRack Airflow 14,600 CFMBypass Airflow 9,400 CFM12New Operating Scenario85F99F64F64FCFM 100%12

13Increased Capacity at Full Airflow5/1/201213

14Increased Capacity at Full Airflow5/1/201214Compressors run at reduced capacity or unloadedFan motors run at full speed15Operating Systems ComparisonSystem kW75F RA85F RACompressors35.424.9Evap Fans8.78.7Condenser Fans5.45.4Total 49.539.0Base78.8%15Sanity Check 100 dF OAT, 6400 ASL16

5/1/201216Sanity Check17

The Conventional Approach Base CaseRaise the Return / Supply Air TempContain the Cold AisleAdd Intelligent ControlThe Set-upSummary18Agenda18Unit Airflow 21,600 CFMRack Airflow 15,800 CFMBypass Airflow 5,800 CFM

19Contain the Cold Aisle85F94F62F62FCFM 90%19At the higher RA temperature, the contained system has very similar operating costs as the non-containedFan motors run at full speed but at a reduced CFM and HP due to the higher static pressureCompressors run at reduced capacity or unloaded but slightly higher than the non-contained

20Operating Systems ComparisonSystem kW75F RA85F RA85F RAContainedCompressors35.424.925.8Evap Fans8.78.77.5Condenser Fans5.45.45.4Total 49.539.038.7Base78.8%78.2%2021Why use Containment??

System operating costs are similarContainment partitions and doors cost $$

21

22Hot Spots in Racks due toWrap-Around22

23No Leakage into the Cold AIsle2324Higher Temperatures without Containment

2425Containment of Cold Aisle

25Issues to ConsiderFire Detection / SuppressionWide variation between municipalitiesIf local Fire Inspector involved early, typically goes wellCurtains usually eliminate this issueInstallationWill be site specificyIrregular row length/height, gaps, etc.Site conditions critical, one size does NOT fit all5/1/201226Issues To ConsiderADACAC space is for Service Personnel (Section 4.1.1)What about cooling for components in rest of room?Best solution today may be ducted return from hot aislePerf tiles near other equipment requiring cooling (eg. UPS)85 Room temperature?

5/1/201227HAC vs CACMain Purpose of Cooling in Data Center?Cool the equipmentData Centers commonly on raised floorCAC allows current investment to be usedHAC typically requires in row coolingSo, refrigerant or chilled water and condensate intermingled with IT equipment and racksWhich aisle does majority of work take place in?CAC hot aisle likely in 85F rangeCan use perf tiles in other spaceHAC hot aisle likely in 100F range5/1/201228The Conventional Approach Base CaseRaise the Return / Supply Air TempContain the Cold AisleAdd Intelligent ControlThe Set-upSummary29Agenda29Unit Airflow 16,800 CFMRack Airflow 14,600 CFMBypass Airflow 2,200 CFM30Closer to Matching the Load to the Cooling92F97F62F62FCFM 70%

Variable Capacity CompressorsVariable Speed FansIntelligent Control30The airflows and capacities/loads are more closely matchedFan motors run at reduced speed, CFM and HP based on the demand in the contained areaCompressors run at reduced capacity or unloaded but slightly higher than the non-contained

31Operating Systems ComparisonSystem kW75F RA85F RA85F RAContained92F RA Contained & ControlledCompressors35.424.925.825.5Evap Fans8.78.77.53.0Condenser Fans5.45.45.45.4Total 49.539.038.733.9Base78.8%78.2%68.5%31The Conventional Approach Base CaseRaise the Return / Supply Air TempContain the Cold AisleAdd Intelligent Control (Creating SmartAisle)The Set-upSummary32Agendaslide3233Sensor LocationServer centric solution, meaning that it focuses on the inlet temperature to the serversSelf adapting to environment changes due to server utilization, equipment location changes and outside variablesCan adapt to situations with no containment, end containment, and full containment

33 Rack Sensors

Rack Sensors without doors can be mounted on the frame of the racks.Temperature differences were within .5F

34Cold Aisle SensorsSensors can also be mounted at the top of cold aisles when rack mounting is unavailable

3536Supply CompensationCompensation is the magical link between controlling sensorsThe controller evaluates the Rack Sensors and Fan SpeedIf the fan is operating at 100% and remains above the cold aisle set pointThen the supply temperature set point will slowly lower to drive the correct cold aisle temperature

Lower SupplyFan Speed100%36

CW Valve Open %100%90%80%70%60%50%40%30%20%10%0%

Fan Speed %Controller and CRAC OperationIncrease in kWUnit ONIT Load IncreasesRack Temperature Sensor detects inlet rack temperatureFan Speed IncreasesSo that cold aisle temperature is maintained at customer temperature SetpointSupply TemperatureThe increase in fan speed will result in a warmer supply air temperature which is detected by the supply air sensor that will increase cooling to maintain supply air setpointCold Aisle ContainmentiCOM will automatically adjust to changes that result in a temperature increase or decrease3738Controller and CRAC OperationAdvanced freeze protection routineAllow all units to reduce fan speed to 60%Fan speed and compressor capacity (or CW valve) managed for best unit efficiency and performanceMultiple remote sensorsController can use averaged and maximum/minimum values to individually control multiple CRAC systemsCRAC systems work as a teamAll remote sensors usedIncrease capacity of other applicable adjacent units if one is at maximum and unable to handle the loadAutomatically adjust for units not in service

38The Conventional Approach Base CaseRaise the Return / Supply Air TempContain the Cold AisleAdd Intelligent ControlThe Set-upSummary39Agenda39Issues to ConsiderFire Detection / SuppressionWide variation between municipalitiesIf local Fire Inspector involved early, typically goes wellCurtain eliminates this issueInstallationWill be a local responsibilityYou need to develop relationship with local Cable Contractor or similar company for installationIrregular row length/heightSite Survey criticalMost expensive componentIssues To ConsiderADAConsultant raised issue at BAIS projectSuccessfully defended CAC space is for Service Personnel (Section 4.1.1)What about cooling for components in rest of room?Best solution today is ducted return from hot aisleperf tile by IT component85 Room temperature

Issues To ConsiderPurpose of Cooling in Data Center?Control equipment inlet tempsData Centers often on raised floorCAC allows current investment to be usedWhich aisle does majority of work take place in?CAC hot aisle likely in 85F rangeCould use perf tile in other spaceHAC hot aisle likely in 100F rangeBest practices are a must if improved efficiency is a goalRunning warmer temperatures in the space will improve the cooling system operating efficiencyContainment improves the availability of the servers by eliminating hot spotsIntelligently controlling fan speed and compressor capacity balances the system to operate at its most efficient level43Summary43SummaryEfficiency is the story, not containmentContainment can be done in several ways, none necessarily fit all situationsEach critical space is unique and merits individual planningDynamic Control is (currently )the final element5/1/201244Shifting on to Economizers and Equipment Considerations

4590.1 Economizer Map

Definitions of Cooling EfficiencyEER (Energy Efficiency Ratio)Total Cooling Capacity (BTUH)/Total power input (watts)Full-load value on 95F design dayCommercial return air conditions of 80F/50%

SCOP (Sensible Coefficient of Performance)Sensible Cooling Capacity (kW)/Total power input (kW)Full-load value on 95F design dayData Center return air conditions of 75F/45%

47These are two common terms to describe the energy efficiency of cooling units:

EER (energy efficiency ratio) is expressed at Total cooling BTUhs divided by the total power input (in watts). It is also based on one specific operating point, that the cooling unit is likely never to operate because of cooling redundancy that is usually designed into the data center. It is also based on 80F air temperature back to the cooling unit.

SCOP (sensible coefficient of performance) is a net cooling capacity, with fan motor heat removed, and is expressed as kW cooling divided by kW power input. Again, it is based on a full-load condition on a 95F day. Data center conditions of 75F are used. ASHRAE 90.1, reference in most state energy codes, now includes minimum requirements for data center coolin.47ASCOP Efficiency MetricAnnualized Sensible Coefficient of Performance(location specific)

Bin efficiency x bin hours/total annual hoursFactors hours at each efficiency operating pointFactors part-load efficiencyFactors economizer hours 48

Single System Outside Air Economizer49

Large or Multiple Systems Outside Air Economizer50

Single System withGlycol Economizer51

Precooler on Chilled Water System52

Chilled Water withCooling Tower Economizer53What if we could do a refrigerant based economizer cycle?54Energy Consuming ComponentsAir-Cooled System

55Compressor energy reduction is low-hanging fruit

55Digital Scroll TechnologyContinuous variable capacity compressor technology without inverter drive from 10% to 100% of capacityIn commercial use since 2004Available in single compressor and tandem compressor configurationsHigh reliability versus Inverter drive systemsNo electrical harmonics introduced56

EC fans in unit57

EC fans in raised floor58

EconoPhase Pumped Refrigerant Economizer59CondenserCheck ValveCompressors ONEvaporatorElectronic expansion valveSummer operationCheck ValveCheck ValveRefrigerantPumps OFFSolenoidValveEconoPhaseOutdoor Ambient 95FKW @ 70% Load 24.1Circuit 2Circuit 159EconoPhase Partial Economization60CondenserCheck ValveCompressorEvaporatorElectronic expansion valveFall / Evening operationCheck ValveCheck ValveRefrigerantPumpSolenoidValveEconoPhaseOutdoor Ambient 65FKW @ 70% Load 15.1Circuit 2Circuit 1ONOFFOFFONAs ambient temperatures get cooler in the evenings or with the change of seasons, the Econophase will act as a pre-cooler to reduce compressor operation. In this case one circuit runs on compressor operation and the second circuit runs on the refrigerant pump. This is where the staged A coil comes into play. It pre-cools the refrigerant to reduce the compressor workload.

Note in this case my system energy draw has decreased by 37% to 15 kW.60EconoPhase - Full Economization61CondenserCheck ValveCompressorEvaporatorElectronic expansion valveWinter operationCheck ValveCheck ValveRefrigerantPumpSolenoidValveEconoPhaseOutdoor Ambient 25FKW @ 70% Load 3.7Circuit 2Circuit 1ONOFFONOFFIn full economization mode when temperatures are below 35 degrees both compressors are off and both refrigerant pumps circulate the refrigerant through their respective circuits. Energy consumption is now only 3.7 kW for the entire system! And I get full capacity from the system. This is an 85% reduction in the total system energy draw and thats because the refrigerant pump draws about 1/20 the energy of a compressor!61Technology Electronic Expansion Valve EC Plug FansTandem compressors w/ Digital Scroll New Evaporator Coil design Staged A CoilGreater surface area Microchannel Condenser Indoor Unit Communicates to CondenserRefrigerant Economizer

The DSE is loaded with new technology and Liebert firsts.

Its features include electronic expansion valves, ec plug fans, and features new to the precision cooling market: tandem compressors in each circuit with digital scroll compressors, a larger, staged A coil optimized to the compressor, and a microchannel condenser that even communicates with the indoor unit for optimized performance.

62Advantages ofRefrigerant EconomizerNo additional heat exchangersNo dust or contamination concernsNo need for added ductwork or structural changes to building for economizerNo dampers to maintainQuick changeover between compressor mode and economizer modeNo water usageNo water freeze or coil freeze concerns

63Liebert DSE Full-Load Efficiency SCOP (kW/kW) @ 85F return airLiebert DSE DA125A @ 50% loadASHRAE 90.1 @ 75FEfficiency Plateau+92%Liebert DSE DA125A @ 100% load+45%+115%Traditional RefrigerationWith Refrigerant Economizer Maximum capacity, max airflow65Ambient, F5/1/201265Designing for LEEDEnergy efficiencyNo water use for coolingMERV 13 filtersNo outside air contaminationR-410A refrigerant/lower chargeUtility Rebates66Condenser Improvements67Air Cooled Condenser ImprovementsMicrochannel vs. Fin and TubeVariable Speed FansControl Based on Refrigerant Pressure68

Aluminum Fins & Tubes Developed & used in Automotive A/C for 25+yrs

69Microchannel Coil

6970Microchannel Coil Advantage1 Thin coil replaces 2 to 6 fin/tube coilLower airside PReduced maximum fan wattage for heat transferReduced refirgerant quantity

6170EC Axial Fan Technology EC (Electronically Commutated) MotorEvery fan/motor is designed for variable speed. Superior part load efficiency3-Phase AC power feed, but efficiency of a DC motor20% reduction of fan speed = 49% savings in energy

71

SoundQuiet fan blade design

72

72Sound Level Factors & OptionsSound varies with fan speedFan speed is proportional to % compressor load and outdoor ambient temperatureWill you ever hear a lightly loaded system condenser?Even more quiet optionsUpsizing condenser will keep fan speed in nearly inaudible range!73Why do they cool things with air, anyway??74

75Why indeed? How about pumped refrigerant?Refrigerant piping anifold suppliespumped refrigerant

Cam levers activates engagement and disengagement of server

Refrigerant connection points at top of the rack on left side. All plumbing runs along left side.75Cooling ArchitectureEach server is cooled by an individual cold plateCold plates and insertion mechanism is designed for use of standard 1U serversCirculating refrigerant is the cooling mediumRackServerHeat exchanger & circulation unitCold PlateTo/From Chiller

Cold PlateHighly flexibleBacked with pressure plateForms to the server lid76In-Rack Cooling withPumped RefrigerantThe rack is 45U tall by 800mm wide by 1200mm deep.36 1U server slots with cold plate Three non-cooled server slots for cool devices like switches, etc.Can be configured for a cooling capacity of 20kW or 40kW.

7778Thank You.Questions?5/1/201278Chart170020303847100

Typical DX UnitColumn1Return Air Temp (oF)% Capacity Increase

Sheet1Column1Column2707500802018.5853037903855954773100To resize chart data range, drag lower right corner of range.

Chart17070002018.5303738554773100100

Typical Chilled Water UnitTypical DX UnitColumn1Column2Return Air Temp (oF)% Capacity Increase

Sheet1Column1Column2707500802018.5853037903855954773100To resize chart data range, drag lower right corner of range.

Chart10.69930069930.17342657340.1272727273

Component

Sheet1ComponentCompressor70%Evap Fan17%Condenser fan13%