viacom data center optimization project executive briefing may 7, 2010 john wallerich,
TRANSCRIPT
VIACOMData Center Optimization Project
Executive Briefing
May 7, 2010
John Wallerich,
Dell Global Services2
Agenda
• Review Project Requirements and Objectives
• Dell’s Gated Project Methodology
• ASIS State Assessment
• TOBE State Analyses
• APC InRow Cooling Analysis
• Implementation Cost Estimates
Dell Global Services3
Project Objectives
IMPROVE
Cooling Infrastructure EffectivenessData Center CapacityIT Equipment TemperaturesCost of Operations
TOOLSComputation Fluid Dynamics (CFD)3-D Graphical Software to model airflow and thermal dynamics
Dell Global Services4
Cost Improvement Categories
• Improvements to room density and OPEX
o Cool load using less cold air volume
o Reduce Bypass air
o Reduce equipment Inlet air Delta-T
o Raise Supply Air Temperatures
o Reduce Fan Energy
o Reduce DX Compressor loads
• Extend useful life of this data center and avoid new build
Dell Global Services5
Services Engagement MethodologyAn Interface to Investment Decision-Making
An efficient “gated” engagement approach designed to arrive at solutions with reduced cost and improved efficiency
Dell Global Services6
Glossary of TermsBLANKING PANELA plastic or metal plate fitted to the front of a rack where no IT equipment is present. BYPASSThis refers to the percentage of cold air that is generated by the cooling system, yet for one reason or another never passes through IT equipment. CACComputer Room Air-Conditioner. This is a generic term that includes all of the cooling devices in the Viacom data center. CFDComputational Fluid Dynamics. This is the technical term for the software tool that we use to model both airflow movement and thermal dynamics for both the under-floor cold air supply plenum, the computer room itself, and if used, the hot air return plenum (above the false ceiling). CFMCubic Feet per Minute. This is a measurement of air volume. For this study, this term is used to describe the amount of air processed by CACs as well as the volume of air flowing into the computer room through a perforated tile. RETURN AIRThis is a term that refers to the air that enters to CAC cooling units to be cooled.
HOT AIR RETURN PLENUMThis represents an area above the false ceiling when it is used as part of an airflow management system. COLD AIR SUPPLY PLENUMThis refers to the area under the raised floor where cold air is injected by the CAC units. The cold air is distributed into the room via perforated tiles.
Dell Global Services7
Solution Options Summary
Solution Option #1: Deploy Hot Air Return Plenum
Solution Option #2: Install Airflow Management
Solution Option #3: Deploy Containment Strategy
Special Study: APC In-Row Containment Design
Dell Global Services
CFD Modeling
Computational Fluid Dynamics allows for the analysis of airflow and thermal dynamics with the detail and accuracy required for fine tuning of the data center cooling infrastructure
8
Dell Global Services9
Low Pressure resulting
from high tile count
Low Pressure resulting from high velocity
Low Pressure resulting from high velocity
Under-Floor PRESSURE DynamicsHigh
Pressure
High Pressure
Dell Global Services10
ASIS: Thermal Slice at 6ft Elevation
Exhaust air contaminatio
n
Exhaust air contaminatio
n
Exhaust air contaminatio
n
Dell Global Services11
ASIS: Max Inlet Temps and CAC % USEDASHRAE Temperature Guidelines
Dell Global Services12
ASIS: Cooling Challenges
Hot air contamination
SAN RoomExhaust air is flowing over tops of racks contaminating surrounding racks.
AS400’sRacks are facing same direction so
exhaust air contaminates rows behind it with increasingly warmer
air.
Dell Global Services13
ASIS: RETURN AIR STREAMS
CACs
3 CACs in center of room are drawing server exhaust from servers that are closer to other CACs
Dell Global Services14
ASIS: AS400 AIRFLOW CONTAMINATION
Exhaust air from the front rows is directed at air inlets of rows behind them. This increases the inlet air temp and puts the back rows at risk of over heating.
Dell Global Services15
ASIS: BEST PRACTICES OBSERVED
• Use of Blanking Panels good Best Practice
• Use of hot-aisle/cold-aisle allows for enhancements
• Plumbing and wire trays are well implemented
• Lack of combustible materials
• Lead-Acid batteries not on raised floor
• Site is clean and well maintained
• Cable cut-outs and floor leakage is minimal
Dell Global Services16
ASIS: Design Weaknesses
• AS400 racks are not optimized (facing same direction)
• SAN room suffers from very low UF pressure
• Room cooling is greater than needed to compensate for weak airflow management design
• CACs are of 2 different designs for high availability, yet redundancy is no longer an option
• CACs are aging and lack VFDs and multi-stage compressors
• Hot server exhaust contaminates server inlets as it flows towards CACs – no airflow controls
• Room capacity is limited by cooling effectiveness
Dell Global Services17
Solution Option #1: Design
Basic Airflow Control: Return Plenum
By connecting the return ducts of CAC units to the area above the false ceiling, and installing return air grills above hot aisles and other heat sources, the area above the false ceiling becomes a hot air return plenum.
Hot air is “pulled” from the room and returned to the CAC unit to be re-cooled.
This is a foundational requirement to installing more advanced airflow control measures and yields a number of benefits.
Dell Global Services18
Solution Option #1: Design
Hot Air Return Plenum
CAC Duct Extensions
Hot server exhaust air is removed from the room through ceiling
grills and directed back to CACs at higher
temperatures
CAC ducts connected to false ceiling turns ceiling void into a hot air return plenum
Dell Global Services19
Solution Option #1
Operational Statistics
Number of Cabinets 365 379
Total Cooling Airflow 199,538 CFM 187,138 CFM
Total Equipment Demand 142,069 CFM 142,069 CFM
Cooling Capacity 1213kW 1213kW
Cooling Capacity in Use 947kW 947kW
Maximum Inlet Temp 93F 87F
Number of Active ACUs 18 18
ACU Power Consumption 107.6kW 102kW
Note: Number of cabinets increased as some new racks were added to establish row symmetry (racks in purple)
Description ASIS SO#1
Dell Global Services20
Solution Option #1: Benefits
Impact of Enhanced Heat Removal
• Hot exhaust air mixing with cool room air is reduced
• Return air is warmer: CAC can reject more heat
• Reduced contamination of nearby server inlets
• Supply air temperature increases from 58F to 64F
• Room can sustain increased IT power loads
• Can cool using reduced supply air
Dell Global Services21
Solution Option #2: Design
ESD Airflow Control Curtains
Solution Option #2 introduces Electro-Static Discharge curtains to provide enhanced isolation of hot and cold air, and to direct hot exhaust air into the hot air return plenum.
These curtains are located between the tops of racks and the false ceiling and help to isolate hot and cold air masses
ESD curtains have been adopted as a low-cost yet highly effective mechanism for simple airflow management.
Dell Global Services22
Solution Option #2: Design
Install ESD Airflow Control Curtains
ESD Curtains
Note: false ceiling not shown
Dell Global Services23
Solution Option #2: Design
Install ESD Airflow Control Curtains
Ceiling Grill
ESD Curtains
Hot Aisle
ESD curtains prevent hot exhaust air from flowing over the tops of racks into cold aisles where it would contaminate server inlets
Dell Global Services24
Solution Option #2 Design
ESD Curtains and Airflow Management
False Ceiling is not shown, but hot air removal and effect on cold aisles is apparent as a result of deployment of ESD curtains
Dell Global Services25
Solution Option #2 Benefits
Ambient temperatures more stable
A thermal slice at a 6ft elevation now shows dramatic improvements as a result of ESD airflow control curtains. The result is further improvements in effectiveness of cold supply air and reduction of hot air contamination
Dell Global Services26
Solution Option #2 Benefits
Cooling Effectiveness Improved
• Improved isolation of hot and cold air
• CAC return air is warmer due to reduced mixing
• Supply Temperature is increased from 60F to 68F
• Rack Delta-T is reduced
• Volume of supply air is reduced due to higher
effectiveness
Dell Global Services27
Number of Cabinets 365 379
Total Cooling Airflow 199,538 CFM 187,138 CFM
Total Equipment Demand 142,069 CFM 140,237 CFM
Cooling Capacity 1213kW 1213kW
Cooling Capacity in Use 947kW 943kW
Maximum Inlet Temp 93F 84F
Number of Active ACUs 18 18
ACU Power Consumption 107.6kW 97.5kW
Description ASIS SO#2
Solution Option #2
Operational Statistics
Dell Global Services28
Solution Option #3: Design
Containment StrategyContainment represents the most advanced airflow
management system due to the creation of a closed airflow loop. This solution builds on Solution Options 1 and 2, and provides the ultimate solution in terms of achieving the stated project objectives.
The concept behind containment is to completely enclose either a hot or cold aisle. Cold aisle containment prevents cold air exiting perforated tiles from going anywhere but through equipment. But the problem then becomes the effect of hot air entering the room.
Hot aisle containment encloses the hot aisle and provides a number of major benefits.
Dell Global Services29
• Creates slight negative pressure in hot aisle which “pulls” air through the air inlets
• Pressure difference eliminates seeping of hot air through the rack into the air inlets
• Entire room becomes cold air plenum: the volume of air supplied at a given perforated tile is no longer critical
• Allows for racks in areas where there is low UF pressure to still get enough air as long as total supply is > total demand.
• Rack Delta-T is very low allowing for increase in supply temps
• Consumes 30% less air than Solution Option #1
• Supply Temperature raised to 74F
Solution Option #3: Benefits
Hot Aisle Containment Benefits
Dell Global Services30
Solution Option #3: Design
Containment Strategies: CAC
Example of a Cold Aisle Containment Strategy. Notice the return air ceiling grills above the hot aisles
PROS:Reduces require air volume by 20%Isolation of hot exhaust airDoes not require ceiling plenum
CONS:Hot air enters roomVery little excess cold air capacity
Dell Global Services31
Solution Option #3: Design
Containment Strategies: HAC
Example of a Hot Aisle Containment Strategy.
PROS:Reduces require air volume by 30%Isolation of hot exhaust airPulls air through servers
CONS:Requires hot air plenumCreates warm work area
Dell Global Services32
Solution Option #3: Design
Combined Containment Model
Example of a Contained and non-contained areas
Dell Global Services33
Solution Option #3: Benefit
Containment Benefits: CAC vs. HACCold aisle Containment:
Reduces bypass air by enclosing cold air supplyEliminates hot air exhaust contaminationDoes not require hot air plenumReduces total supply air by 20% over SO#1
Hot Aisle Containment:Maximizes air bypass reductionHigher degree of airflow control (pull vs. push)Removes all hot air from roomReduces total supply air by 30% over SO#1Increases rack densityExtremely even Delta-T across rack face
Dell Global Services34
Solution Option #3: Benefit
Increased Supply Air Temperature
Raising supply temps reduces costs of cooling, and requires less cold air volume.
Ambient room temp varies only slightly and is a comfortable 75F.
Heat rejection is 96% done by the CAC
Dell Global Services35
Solution Option #3: Benefit
Stable Equipment Supply TemperaturesA combined containment strategy has resulted in a significant increase in supply temperature and reduction of total cold air supply needed. This translates into reduced OPEX while providing a more stable operating environment as well as the ability to increase equipment power loading.
Dell Global Services36
CAC Cluster Creates Low Pressure Zone
CACs
Low Pressure Zone
One factor that is limiting CAC effectiveness is caused by 3 CACs near the middle of the room. These units and their close proximity to each other creates a low pressure zone that pulls exhaust air from
Dell Global Services37
Solution Option #3Operational Statistics
Number of Cabinets 365 379
Total Cooling Airflow 199,538 CFM 189,419 CFM
Total Equipment Demand 142,069 CFM 142,069 CFM
Cooling Capacity 1213kW 1213kW
Cooling Capacity in Use 947kW 947kW
Maximum Inlet Temp 93F 81F
Number of Active ACUs 18 17
ACU Power Consumption 107.6kW 97kW
Description ASIS SO#3
Dell Global Services38
Special Design Review APC High Density ClusterHot Aisle Containment: sub-optimum
APCs leaking warm air
APC’s InRow
coolers are an efficient
solution, but their
effectiveness is lost on
hot aisle containmen
t8 APC InRow Coolers with 12-12kW racksContained area is so hot, the coolers are sending 85F air into the room
Dell Global Services39
Solution Design ReviewAPC High Density ClusterCold Aisle Containment
APC units are optimized when in a cold aisle containment design. Total load is 216kW over the hot
aisle maximum load of 144kW
and all equipment is experiencing inlet temps in the mid-70F
range. 6 APC InRow Coolers with 12-16kW and 2-12kW racks. All rack inlets are below 78F and hot air is not affecting local racks.
Dell Global Services40
Solution Option Comparative Analysis
Number of Cabinets 365 379 379 379
Total Cooling Airflow 199,538 CFM 187,138 CFM 178,138 CFM 165,419 CFM
Total Equip. Demand 142,069 CFM 142,069 CFM 142,069 CFM 142,069 CFM
Cooling Capacity 1213kW 1213kW 1213kW 1213kW
Cooling Capacity in Use 947kW 947kW 947kW 947kW
Maximum Inlet Temp 93F 87F 84F 81F
Number of Active ACUs 18 18 18 17
ACU Power Consump. 107.6kW 102kW 97.5kW 89kW
AVG. Supply Temp. 60 62 63 68
Description ASIS SO#1 SO#2 SO#3
Dell Global Services41
Cost Basis Metrics and Calculations
1kW of IT Electrical Load - Annual cost: $6,000(space, power, cooling)
Annual OPEX expense per 3kW rack $18,000
1F of Set Point increase = 1% energy reduction(CAC compressor efficiency*)
• If VFD’s were deployed, an additional 20% of total CAC energyreduction would be realized as a result of reductions in fan energy.Dependency is that Solution Option #3 would be deployed.
Dell Global Services42
Data Center Design Comparative Models
CURRENT STATE METRICS Number of Racks 365Total Cooling Capacity 1213kWTotal Cooling Load in Use 947kWTotal Connected Power 905kWServer Temperatures 65-95F
CONTAINMENT DESIGN UNDER MAX LOADNumber of Racks 397Total Cooling Capacity 1213kWTotal Cooling Load in Use 1156kWTotal Connected Power 1114kWRoom Temperatures 72-82F
DELTA
+32
+209kW+209kW-9F (max)
Dell Global Services43
FINANCIAL ANALYSIS – Containment Costs
Ceiling Grill ConstructionGrills (1540 sqft @ $1/sqft) = $
1,540 Containment Curtains and Doors* = $ 78,000
Additional Racks (12) =$ 18,000
TOTAL $ 97,540
*includes CAC ducts to ceiling plenum
Recommended:Replace CAC’s with more efficient units $1,800,000
Dell Global Services44
Containment Cost Reduction Estimates
OPEX Reduction
1. Reduce cooling costs by 6% annually @ $360/rack365 racks X $360 = $131k/yr
2. Equipment replacement (annual) $20k/yr3. Upgraded CAC cooling efficiency gains $48k/yr
TOTAL $199k/yr
CAPEX Reduction
1. Increase rack power loads by 30% (avoid new racks) $200k2. Avoid adding new CACs (3 @ $60k) $180k3. Increase cooling system effectiveness (no cooler exp.) $500k
TOTAL $880k OTC
COST AVOIDANCE
4. New Data Center (20 sqft) Estimated at ~$30m
Dell Global Services45
Recommendations
1. Deploy Solution Option #3: combined containment strategy. This will reduce OPEX while extending the capacity of the existing Cooling infrastructure and data center.
2. Virtualize as much of the environment as possible. This will have a multiplier effect in terms of power reduction (each W of IT power removed equates to 2-3 Watts of electrical and cooling overhead reduction
3. Shorten the server refresh cycle. Consider a 3-year plan. This will provide benefits both in terms of service levels from faster servers, and will also have a multiplier effect in terms of power savings as a result of more power efficient processors.
Dell Global Services46
Additional Recommendations (1)
1. Replace all CACs with more efficient models:VFD fans (10% reduction in speed = 50% reduction in power)Multi-phase compressors (2 or 3 stage compressors)
2. Relocate new CACs for better distribution and coverage
VFD’s and Multi-phased compressors will help reduce power Consumption for the cooling infrastructure particularly for dataCenters with containment strategies.
Containment allows for the same degree of cooling using less volumeof air. So, VFD’s, totaling over 80kW today could be cut in half.
Multi-phase compressors allow CACs to provide required coolingrunning in a more power efficient mode (stage 1 vs. stage 2)
Dell Global Services47
CAC SET POINTS and MIN SUPPLY TEMPS CAC # ASIS SO#1 SO#2 SO#3
MST SP MST SP MST SP MST SP
CAC-1 60 68 64 88 66 88 62 88
CAC-2 60 68 66 86 66 90 64 90
CAC-3 60 68 62 86 66 88 64 86
CAC-4 60 68 64 86 66 86 60 86
CAC-5 60 68 64 86 66 88 64 88
CAC-6 60 68 64 86 64 86 64 84
CAC-7 60 68 64 88 64 88 66 90
CAC-8 60 70 66 90 64 90 66 88
CAC-9 60 67 64 78 64 84 60 82
CAC-10 60 68 64 86 66 88 60 90
CAC-11 60 72 62 76 64 84 60 76
CAC-12 60 68 64 86 64 90 60 88
CAC-13 60 66 64 88 66 88 62 84
CAC-17 60 68 62 78 62 80 62 78
CAC-18 60 68 64 80 60 76 OFF OFF
CAC-19 60 68 64 84 62 78 62 80
CAC-21 60 65 66 88 64 88 60 84
CAC-22 60 70 64 88 64 86 64 88
AVG 60.0 68.1 64.0 84.9 64.3 85.9 66.25 85.3
Dell Global Services48
Thank You