green datacenter initiatives at sdsc
DESCRIPTION
Green Datacenter Initiatives at SDSC. Matt Campbell SDSC Data Center Services Manager [email protected]. SDSC Data Center Overview. 19,000 sq ft 13 MW of on-site power 100+ projects operating IT equipment in data center. Key Points. How do we illustrate energy loss and quantify savings? - PowerPoint PPT PresentationTRANSCRIPT
SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO
Green Datacenter Initiatives at SDSC
Matt CampbellSDSC Data Center Services Manager
SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO
SDSC Data Center Overview• 19,000 sq ft• 13 MW of on-site power• 100+ projects operating IT equipment in data center
G105G111
G110G109
G108G107
G106G119
G125G124
G123G122
G121G112
G118G117
G116G115
G114G113
J105
J111
J110
J109
J108
J107
J106
G126G132
G131G130
G129G128
G127
J112
J113
G133G134
G135
O105O111
O110O109
O108O107
O106O121
O119O125
O124O123
O122O112
O118O117
O116O115
O114O113
O126O132
O131O130
O129O128
O127O133
O134O135
R105R111
R110R109
R108R107
R106R121
R119R125
R124R123
R122R112
R118R117
R116R115
R114R113
R126R132
R131R130
R129R128
R127R133
R134R135
J128
J134
J133
J132
J131
J130
J129
J135
J117
J123
J122
J121
J119
J118
J124
J125
Air re
turn
Air re
turn
Air re
turn
Air re
turn
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
G120
O120
R120J120
XWVUTSRQPONMLKJIHGFEDCBA
101102
103104
105106
107108
109110
111112
113114
115116
117118
119120
121122
123124
125126
127128
129130
131132
133134
135136
137138
139140
141142
143144
145146
147
PD
U-U
1
PDU-N3
PD
U-N
2P
DU
-N1
PD
U-U
3P
DU
-U2
VentedTile
VentedTile
VentedTile Vented
TileVented
TileVented
TileVented
TileVented
TileVented
TileVented
TileVented
TileVented
TileVented
TileVented
Tile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
Iso-base area
184 sq. ft.
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
Iso-base area
96 sq. ft.
VentedTile
Iso-base area
129 sq. ft.
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
Iso-base area
30 sq. ft.
Iso-base area
96 sq. ft.
Iso-base area
96 sq. ft.
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
Iso-base area
96 sq. ft.
Power Pillar
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
U2F 40/42&
U1C 22/24
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
65 66 67 68 69 70
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y
Z
BB
CC
DD
EE
FF
AA
GG
HH
II
JJ
KK
LL
MM
NN
OO
PP
RR
B
A
C
D
E
F
H
I
J
G
K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y
Z
AA
BB
CC
DD
EE
FF
GG
HH
II
JJ
KK
LL
MM
NN
OO
PP
RR
716968676665646362616059585756555453525150494847464544434241403938373635343332313029282726252423222120191817161514131211109876541 2 3
U2F 10/12 & 14/16
U1C 14/16&
U2H 10/12
Pod A 37/39&
U4B 9/11(with in-rack UPS)
Pod A 14/16&
U2K 26/28U2F 17
U2E 35/37&
U4B
U1C 2/4&
U2H 2/4
U4B 32/34/36 & 38/40/42
U4B 26/28/30 & 37/39/41
U4B
Pod A 64/66, 68/70 & Pod B 64/
66, 68/70
U1C 5
U2I U1C
U2F U2E
PDBMillenniumU4B 1/3/5
PDB
From Millennium
PDB
From Millennium
PDBMillenniumU4A 1/3/5
PDB
From Millennium
U2K 1/3 (9311)U2K 14/16 (3590)
U2K 2/4 & U2F 35/37 (drive racks)
U2K 9/11 (9311)U2K 18/20 (3590)
U2I 35/37 (drive rack)
U2K 5/7 (9311)U2K 6/8 (3590)
U2I 6/8,10/12 (drive racks)
U2K 13/15 (9311)U2K 10/12 (3590)
U2K 17/19 (9311)U2K 22/24 (3590)
U3E-A 6/8 & 10/12U2K 29/31 & U2K 36
U2K 38
Power PillarU4A
Liebert air handler Liebert air handler
Liebert air handler Liebert air handler
Liebert air handlerLiebert air handler
Power PillarPower Pillar
Power PillarDistribution
panel
SATA 3 & 4
SAN 41HPSS
SAN 42HPSS
SAN 43HPSS
SAN 44HPSS
SAN 45HPSS
SAN 46HPSS
SAN 47HPSS
SAN 48HPSS
Distribution panel
SATA 1 & 2
SAN 56HPSS
SAN 55HPSS
SAN 54HPSS
SAN 53HPSS
SAN 52HPSS
SAN 51HPSS
SAN 50HPSS
SAN 49HPSS
U3A 1/3 & 5/7
U2K U2J
U2B
U2A
U1A
U2G (empty)
U3C
U3C-A
U3B
U3B-A
U3E U3D
U3E-A U3D-A
U1C 38/40&
U2J 26/28
U3E-A 2/4, 6/8, 10/12U4A10/12, 34/36, 37/39
U1C 7/9, 35/37U2J 13/15, 17/19, 21/23, 25/
27
U2I 18
U2I 26/28&
U4B 29/31
Pod B&
U1C 10/12
U2F 36/38(both PDU’s feed
from in-rack)
U2I 9/11&
U4B 21/23
U1C 31 & 33& U2E 17& U2I 21
Empty
U2I 39/41
Pod B 5/7&
U2I 38/40/42
Empty
72 73 74 75 76 77 78 79 8070 81
71 72 73 74 75 76 77 78 79 80 81
Power Pillar
U2K 21/23
GPFS-WANserver
GPFS-WANserver
Pod BPod A 26/28, 55/57
& Pod B 63/65
Pod A 6/8&
Pod B 55/57
On-Demand cluster
On-Demand cluster
On-Demand cluster
On-Demand cluster
GPFS-WAN
SAM-QFS
SAM-QFS
GPFS-WAN
GPFS-WAN
Pod A 30/32&
U2I 1/3
Physicsstorage
VentedTile
U2F 29/31&
U4B 17/19
U2F 5/7&
U1C 6/8
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
New APC
Pod A 21/23 & Pod B 21/23
240v
Pod A 22/24 & Pod B 22/24
240v
Pod A 25/27 & Pod B 25/27
240v 240v
IT-L7
240v
SDSCMars
240v
Nesis and GEON
240v
Pod A 10/12&
Pod B 47/49
240v
Pod A 1/3&
Pod B 29/31
240v
IT-J14HPSS tape movers
240v
6140 disk
240v
Engenio rack
240v
U1B 17/19&
U2H 25/27
240v
XFMRCDSA
XFMRCDSD
XFMRCDSE
XFMRCDSC
XFMRCDSB
SBCDS2
CDSA
CDSB CDSC CDSD CDSE
240v
Pod A 18/20 & Pod B 18/20
240v
Grey 2 Grey 1
240v
U2E 14/16
Power pod A Power pod B
VentedTile
VentedTile
240v
240v
240v
240v
240v
240v
240v 240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
240v
CMS disk
CMS blades
CMS networking
CMS disk Empty
Empty
EmptyEmptyEmpty
VentedTile
VentedTile
VentedTile
240v
240v
240v
240v
240v
240v
240v
240v
240v
U2E 32/34&
U2I 5/7
U2E 31/33&
U2E 7/9&
U2E 36/38&
U2I 14/16
U2F 6/8&
U1C 26/28/30
U2F 32/34&
U2I 13/15
240v
240v
240v
240v
240v
240v
U2F 9/11 & 13/15&
U2I 27/29 & 31/33
240v
New APC
U2F 26
GPFS-WAN
240v
Ne
twor
king
C
ore
U3A
U1C 32
U2E 24
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
240v
240v
240v
Liebert air handler
240v
Iso-base area
261 sq. ft.
Iso-base area
261 sq. ft.
UC colo
UC colo
UC colo
240v
240v
240v
240v
240v
240v
240v
240v
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
Iso-base area
316 sq. ft.
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
Iso-base area
316 sq. ft.
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
Iso-base area
349 sq. ft.
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
Liebert air handler
Iso-base area
360 sq. ft.
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
Iso-base area
360 sq. ft.
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
Iso-base area
360 sq. ft.
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC coloUC colo UC colo UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
Liebert air handlerLiebert air handler Liebert air handler Liebert air handlerLiebert air handler Liebert air handlerLiebert air handlerLiebert air handler
Gray system
Black system distribution cabinet
Blue jump-out
Power PillarU2H U1B
U2D U2C
New APC
New APC
New APC
U2B
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
VentedTile
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
UC colo
Power
Power
SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO
Key Points• How do we illustrate energy loss and quantify savings?
• Establishing a common unit of measurement for data centers• What areas can be improved?
• Mechanical• Electrical• IT equipment
SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO
Power Usage Effectiveness PUE =
• Industry accepted unit of measurement• 2+ is still the data center industry ‘norm’• SDSC’s datacenter came was audited last year with a resulting
PUE of 1.35 – 1.42.• Systems located at SDSC would save 28-65% on energy alone!
Total Facility PowerIT Equipment Power
SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO
Energy Savings Multiply
Courtesy Jack Pouchet, Emerson
SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO
Working on Many Fronts
Air handling retrofit with variable frequencydrive fans and modulating chilled water
valves to match cooling to real-time loads.
High-efficiency UPS Retrofit
Cold / Hot Aisle Containment
“Free Cooling”Opportunities
Efficient SystemLifecycles
Distributing Higher-Voltage(277/480V) Power to the
floor to reduce transformerand distribution losses.
Experimenting with DC-OnlyPower Distribution
SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO
Data Center Improvements to Lower PUE
• Mechanical Systems• Aisle containment• CRAH VFD retrofits• Blanking panels, floor brushes
• Electrical Systems• High efficiency UPS
• System Lifecycles• Virtualization• Consolidation of ‘ghost’ servers
SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO
Cold/Hot Aisle Containment
Cold Aisle Containment• Knuerr Coolflex
• First deployment in the United States• Requires a standardized datacenter• Enclosed cold aisles (70-78F)• Servers exhaust to room as a whole,
which runs hot (90-100F)
Hot Aisle Containment• Essentially a “heat chimney” on top of hot
aisles with doors at each end.• Flexible, can be phased incrementally• Cold room as a whole (70-78F)• Exhaust to enclosed hot aisle (100+F),
ducting directly to cooling systems
• Expect 50-80% better cooling airflow efficiency than traditional hot/cold aisle• Separates cold/hot air, eliminating the need for over-cooling, over-blowing• Allows for high temperature differentials, maximizing efficiency of cooling equipment
SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO
Filling In The Gaps
• Brushes to cover tile cutouts• Allows cables to pass through
while blocking air
•Cost effective blanking panels•Minimizes leakage
SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO
CRAH VFD Retrofits• Added variable
frequency drives to computer room air handlers• Allows CRAH units to
modulate based on control signal
SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO
Electrical Systems• High efficiency UPS units that can save ~500,000
kWh/year • Higher voltage to IT equipment
• Providing 277/480V power on the floor to more efficiently power HPC systems without transformer and distribution losses
• Exploration of DC power for IT equipment
SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO
Hardware Lifecycle Efficiency• Hardware retirement
• Equipment refresh more often leverages technology improvements
• Virtualization• Maximize equipment utilization
• Consolidation of ‘ghost’ servers• Underutilized IT equipment• ‘Forgotten’ equipment
SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO
QUESTIONS?