energy-proportional computing: a new definition - david wood
TRANSCRIPT
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
1/72
Energy
-
Proportional Computing:
A New DefinitionDavid A. Wood
Professor, Computer Sciences
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 1
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
2/72
Outline
! Background & Motivation
! New ideals
! New metrics
! Management
! Summary
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 2
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
3/72
Datacenter Energy Consumption
! U.S. data center energy [NRDC, Anthesis]
! 2013: 91B KWh ! 34 coal-fired 500-MW power plants, 1yr
! 2020: 140B KWh ! 50 coal-fired 500-MW power plants , 1yr
!
! $13B/yr electricity costs
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 3
Typical datacenterenergy consumption
breakdown[Dimension Data]
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
4/72
PDUsSwitchgearsGeneratorsUPS
Power
ChillersPumpsCRAHsCRACsDX
Cooling
Lighting
Misc
Building Load
Servers
StorageNetworksMonitorsWorkstationsLaptops
IT LoadTotal
FacilityEnergy
IT EquipmentEnergy
PUE: Power Usage Effectiveness
! PUE =!"#$% '$()%)#* +,-./*0! +12)34-,# Energy
!1
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 4
[The Green Grid]
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
5/72
Datacenter Energy Consumption
! U.S. data center energy [NRDC, Anthesis]
! 2013: 91B KWh ! 34 coal-fired 500-MW power plants, 1yr
! 2020: 140B KWh ! 50 coal-fired 500-MW power plants , 1yr
!
! $13B/yr electricity costs
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 5
Typical datacenterenergy consumptionbreakdown[Dimension Data]
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
6/72
PUE: Power Usage Effectiveness
! PUE =!"#$% '$()%)#* +,-./*0! +12)34-,# Energy
!1
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 6
[The Green Grid]
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
7/72
PUE: Power Usage Effectiveness
! PUE > 1 " energy waste due to non-IT infrastructure
! Strong impact in reducing non-IT energy waste
! PUE estimates for current datacenters! Allied Control: 1.02, Facebook: 1.08, Google: 1.08, avg. 1.12,
Green IT Cube: 1.07, Microsoft: 1.125, OVH: 1.09
! Average: 1.7
! Missing: IT equipment energy waste
! This talk focuses on server energy
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 7
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
8/72
Energy Efficiency
! # =5".6
Energy=
7-.8".4$,(-Power
!Performance = Load served
! Examples: BIPS/Watt, ssj_ops/Watt, Instructions/nJ,
! #max $Emin
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 8
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
9/72
EP: Energy Proportionality
! # =5".6
Energy=
7-.8".4$,(-Power
!EP: Energy%Work [eliminates energy waste ]! equiv., Power % Performance
! equiv., constant #
! First proposed by Barroso and Hlzle, 2007
! Does EP imply Emin? Converse? Identical goals?
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 9
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
10/72
Conventional Wisdom
! We see that peak energy efficiency occurs atpeak utilization and drops quickly as utilizationdecreases. Luiz Andr Barroso and Urs Hlzle,IEEEComputer, 2007
! Energy waste at low utilizations/loads! Servers are typically 10 50% loaded
! EP as a primary design goal!
Strong impact! Low idle power, wide dynamic power range
! Later, Dynamic EP [Lo et al., ISCA 2014]! Accounts for unavoidable idle power
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 10
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
11/72
Example: SpecPower on Haswell
! Java-based load-varying workload
! 3 calibration intervals to detect max. load (100%)
! Measurement interval Loads: 100%, 90%, 80%, , 10%, Idle
!Performance: transactions per second
! Power: system power
! Quad-core (eight-thread) Intel Haswell server
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 11
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
12/72
EP on Conventional Server
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 12
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
13/72
EP on Conventional Server
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 13
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
14/72
EP on Conventional Server
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 14
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
15/72
EP on Conventional Server
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 15
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
16/72
Results from 2014
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 16
[Lo, et al. ISCA 2014]
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
17/72
EP on Conventional Server
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 17
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
18/72
EP on Conventional Server
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 18
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
19/72
Efficiency of Conventional Server
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 19
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
20/72
Efficiency of Conventional Server
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 20
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
21/72
(more) Conventional Wisdom
! In an energy-proportional system, explicit power
management is unnecessary, as power consumptionvaries naturally with utilization. Meisner, Gold, Wenisch,PowerNap: eliminating server idle power,ASPLOS 2009
! Ideally systems would exhibit energy-
proportionality, wherein servers consume power in
proportion to their load. Meisner, Sadler, Barroso, Weber,Wenisch,Power management of online data-intensive services, ISCA 2011
! ...; modern servers are only maximally efficient at
100%. Meisner, Wenisch,Does low-power design imply energyefficiency for data centers?, ISLPED 2011
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 21
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
22/72
Current (Reconfigurable) Systems
! Reconfigurable resources
! DVFS
! Number of cores
!
Threads per core! Prefetching
! Cache size
!
! Multiple configurations may serve the same load
! Different power & energy consumptions
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 22
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
23/72
EP on Reconfigurable Systems
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 23
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
24/72
EP on Reconfigurable Systems
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 24
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
25/72
EP on Reconfigurable Systems
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 25
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
26/72
EP on Reconfigurable Systems
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 26
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
27/72
Pareto Frontier
! Pareto efficiency/optimality
! Subset of (performance, power) tuples in state space
! Cannot increase performance without also using more power! Cannot reduce power without also decreasing performance
! States that achieve goals will lie on the frontier
! Minimum E, ED, ED2
, ! Minimum power with/without a performance target
! Maximum performance with/without a power cap
!
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 27
Vilfredo
Pareto18481923[Wikipedia]
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
28/72
EP on Reconfigurable Systems
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 28
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
29/72
EP on Reconfigurable Systems
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 29
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
30/72
Reconfigurable System Efficiency
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 30
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
31/72
Reconfigurable System Efficiency
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 31
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
32/72
Reconfigurable System Efficiency
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 32
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
33/72
Limitations of conventional views
! Systems may achieve better than ideal efficiency
! Better than the best!!
! Neither EP nor Dynamic EP is ideal
! Conventional models based on fixed-resourceassumptions
! Modern systems are highly reconfigurable
! New need ideals and new metrics
! Quantify energy waste w.r.t ideal
! Quantify factors that contribute to waste
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 33
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
34/72
Outline
! Background & Motivation
! New ideals
! New metrics
! Management
! Summary
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 34
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
35/72
Conventional Ideals
! Design Ideal! For system designers
! Help design energy efficient systems
! Examples: EP
! Helped drive down idle power in current systems
! Operational Ideal! Characterizes maximum operating efficiency
!
Help system operators and OS schedulers?! Example: Dynamic EP
! Quantifies divergence from ideal linear behavior
! Need new ideals for reconfigurable systems
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 35
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
36/72
New Ideals
! Design Ideal
! EOP Energy Optimal Proportional
! Determined by #max
! Current systems energy optimal configuration
! Best we can do irrespective of of offered load
! Operational Ideal
! Dynamic EO Dyanamic Energy Optimal
! Determined by the Pareto frontier
! Realizable ideal for the current system
! Best we can do for the offered load
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 36
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
37/72
Conventional vs New Ideals
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 37
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
38/72
New Design Ideal
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 38
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
39/72
New Operational Ideal
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 39
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
40/72
Conventional vs New Ideals
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 40
EP EOP
Dynamic EP Dynamic EO
Design
Operational
Conventional New
Dynamic EP " EP " EOP
Dynamic EO " EOP
non-Sub-Linear" Dynamic EP " Dynamic EO
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
41/72
Optimality vs Proportionality
! Optimality at every load Proportionality
! EOP is always proportional
! Uses Emin at every load
!
Proportionality Optimality! EP may not be optimal
! Uses !Emin energy at every load
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 41
Emin = Energy needed by EOP
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
42/72
Goal Dependence
! Minimize E
! EOP uses power linearlyproportional to load
! Constant # regardless of load
! Minimize ED
! EDOP uses power quadratically proportional to load
! Minimize ED2
! ED2OP uses power cubicallyproportional to load
!
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 42
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
43/72
Outline
! Background & Motivation
! New ideals
! New metrics
! Management
! Summary
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 43
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
44/72
PDUsSwitchgears
GeneratorsUPS
Power
ChillersPumpsCRAHsCRACsDX
Cooling
Lighting
Misc
Building Load
Servers
StorageNetworksMonitorsWorkstationsLaptops
IT LoadTotalFacility
Energy
IT EquipmentEnergy
PUE: Power Usage Effectiveness
! PUE =!"#$% '$()%)#* +,-./*0! +12)34-,# Energy
!1
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 44
[The Green Grid]
?
CPUE
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
45/72
PUE: Power Usage Effectiveness
! PUE =!"#$% '$()%)#* +,-./*0! +12)34-,# Energy
!1
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 45
[The Green Grid]
?CPUE
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
46/72
CPUE: Computational PUE
! Goal: To quantify waste in computational energy! Inspired by PUE
! Informally:
! Depends upon configuration c and load l> 0
! E(c,l) = CPUE(c,l)# Emin
! CPUE(c,l) > 1wastes energy
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 46
CPUE(c,l) =Actual server energy with catl
EOP server energy =
E(c,l)Emin
!1
CPUE =
Actual IT server energy
EOP server energy !
1
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
47/72
Decomposing CPUE
! For configuration c and load l> 0
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 47
CPUE(c,l) =Actual server energy with catl
EOP energy
=E(c,l)Emin
=&:max&:;c,l?@A
BCDECF@G
HIJKCLMK?DIJ
BCDECF@ N
OPM=AJ
HIJKCLMK?DIN
>?@A
OPM=A
! Factor performance to focus designers attention
! I/P: Instruction set architectures, compilers, algorithms
! CPI: Pipelines, caches, predictors, out-of-order execution
! T/C: Technology, circuits, pipelines
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 48
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
49/72
Iron Law of Energy Efficiency
! Iron Law of Energy Efficiency:
E(c,l) = LUE(l)# RUE(c,l)# Emin
! Factor energy efficiency to focus designers attention
! LUE: Load Usage Effectiveness
! Operate server at efficient load level
! RUE: Resource Usage Effectiveness
! Configure server to run efficiently at a given load level
! Emin: Optimal energy for a given computation on this server
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 49
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
50/72
LUE: Load Usage Effectiveness
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 50
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
51/72
LUE: Load Usage Effectiveness
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 51
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
52/72
RUE: Resource Usage Effectiveness
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 52
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
53/72
RUE: Resource Usage Effectiveness
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 53
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
54/72
Energy Waste: Root Cause Analysis
! Non-optimal loads
! LUE(l) > 1" energy waste
! There exists another load that uses less energy
! LUE(l) = 1$ system operating with Q&max! Only at points where Dynamic EO and EOP lines meet
! Non-optimal configurations
! RUE(c,l) > 1" energy waste
! There exists another configuration that uses less energy but canserve the same load
! RUE(c,l) = 1$ c is at Dynamic EO
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 54
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
55/72
Outline
! Background & Motivation
! New ideals
! New metrics
! Management
! Summary
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 55
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
56/72
Load Management
! Goal: Choose lsuch that LUE(l) is reduced
! Select subset of Dynamic EO close to EOP
!Global: Inter-server! Load distribution among servers
! Difficult for stateful, time-sensitive services
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 56
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
57/72
Load Management
! Optimization problem
! Assume n servers, ith servers Dynamic EO is described by set
{(xi,R, yi,R), , (xi, j, yi, j), }
! LetLbe the total load to be served
Minimize Pwr = S S H? TU#P?TUVWXYZ
Such that L ' S S H?TU #[?TUVWXYZ
Ii, j( {0,1} for all i,jS H?TUV ' 1 for all i
! Load for server iis S H? TU#[? TUV
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 57
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
58/72
Configuration Management
! Goal: Choose c such that RUE(c,l) is reduced
! Operate system at/near Dynamic EO
! Local: Intra-server
! Existing Linux governors not adequate
! Limited knobs (DVFS)
! Ondemand governor ) peak performing configuration
! Powersave governor (lowest frequency configuration)
! Limits performance
! Operates in high LUE region
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 58
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
59/72
Reactive Governor
! Knobs: core frequency and cache prefetch enable
! R(t): parametrized by length of interval = t ms
! Track Active Cycles for last interval
! Select highest frequency such that all cycles are expected to be
active
! Exponential Ramp-up
! Inspired by Ethernet backoff
! If selected frequency !current frequency
! Increase selected frequency by step
! Double step
! Prefetch control
! Profile with OFF and ON for (10 + 10) ms within every interval
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 59
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
60/72
Reactive Governor
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 60
T=s T=s+10 T=s+20 T=s+t-20
PF PF
perf1 perf2
f = current frequency
perf1 >perf2?
PF
Y
N #Active Cyclesin last t ms?
Calculate target
freq (tfr)
tfr!f?
YN
Init steptfr+=stepstep*=2
f"tfr
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
61/72
Reactive Governor
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 61
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
62/72
Reactive Governor: RUE
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 62
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
63/72
Reactive Governor
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 63
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
64/72
Reactive Governor: RUE
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 64
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
65/72
SLA-Aware Governors
! Disconnect between user needs and OS governors
! SLAs! Maximize energy efficiency
! Subject to max. response time
! Maximize performance within a power cap! RAPL mechanism only seems to deal with DVFS
! Other knobs?
! Minimize power for a performance target
!
! Intel Skylake Speed Shift! Hardware P states (DVFS)
! Other knobs?
! OS specifies performance hints
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 65
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
66/72
Challenges
! Available knobs?
! Hardware has many knobs
! But few exposed to operating system
! Efficient construction of Pareto frontier (Dynamic EO)
! Transition overhead! Speed Shift: )1 msec
! Caches? Interconnects?
! Measurement interval! RAPL: few (>1) ms
! Wall power: 0.1 1 sec
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 66
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
67/72
RAPL Power # System Power
! Advantage: shorter reaction times
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 67
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
68/72
Summary
! Conventional ideal models no longer adequate
! EP does not imply Emin
!
New ideals! EOP as a primary design goal
! Dynamic EO as a primary operational goal
!
New metrics: CPUE, LUE, RUE! Quantify & attribute server energy waste
! New SLA-aware governors needed
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 68
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
69/72
Acknowledgments
! Funding
! NSF grant CCF-1218323
! NSF grant CNS-1302260
! Financial interest in AMD, Google
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 69
Rathijit SenPh.D. Candidate
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
70/72
Backup
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 70
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
71/72
References
! NRDC, Anthesis: http://www.nrdc.org/energy/files/data-
center-efficiency-assessment-IP.pdf
! Dimension Data:
https://www.dimensiondata.com/Global/Downloadable%20Documents/The%20Relationship%20Between%20Data%20Centre%20Strategy%20and%20Energy%20Efficiency%20Whitepa
per.pdf
! The Green Grid:
http://www.thegreengrid.org/~/media/WhitePapers/WP49-PUE%20A%20Comprehensive%20Examination%20of%20the
%20Metric_v6.pdf
2/4/16 UNIVERSITY OF WISCONSIN-MADISON 71
-
7/25/2019 Energy-Proportional Computing: A New Definition - David Wood
72/72
References
! PUE numbers! Allied Control: http://multimedia.3m.com/mws/media/1127920O/2-
phase-immersion-coolinga-revolution-in-data-center-efficiency.pdfFacebook: https://www.facebook.com/PrinevilleDataCenter/app/399244020173259/Google: http://www.google.com/about/datacenters/efficiency/intern
al/index.html#measuring-efficiencyGreen IT Cube: http://www.heise.de/newsticker/meldung/Green-IT-Cube-Hocheffizientes-Supercomputer-Domizil-eingeweiht-3082605.htmlMicrosoft: http://download.microsoft.com/download/8/2/9/8297F7C7-AE81-4E99-B1DB-D65A01F7A8EF/Microsoft_Cloud_Infrastructure_Datacenter_and_
Network_Fact_Sheet.pdfOVH: https://www.ovh.com/ca/en/about-us/green-it.xmlAverage: http://www.datacenterknowledge.com/archives/2014/06/02/survey-industry-average-data-center-pue-stays-nearly-flat-four-years/