t5-deepdive-part_1 (1)
DESCRIPTION
ldom config relatedTRANSCRIPT
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal1
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal2
Drag picture to placeholder or click icon to add
SPARC T5 Servers – Deep Dive NDA – Part 1Insert Presenter’s Name HereInsert Presenter’s Title Here
Drag picture to placeholder or click icon to add
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal3
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.
The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal4
•Scales to 8 sockets using directory
•Minimizes latency •Avoids congestion•Maximize bandwidth
•Double cores and cache
•Balance single thread and throughput
•Dynamically thread
•Oracle workloads •Engineered Systems•Extends
ü on-chip crypto acceleration
ü RAS
•Maximizes peak performance
•Manages thermal and current loads
•Scales elastically
Optimize
SystemsMultiply
Performance
SPARC T5
Scale Efficiently
Advance Power
Management
Design Objectives Achieved
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal5
Next Generation SPARC T5 ServersFaster. Optimized. Secure.
SPARC T5-1B
SPARC T5-8
SPARC T5-4
SPARC T5-2
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal6
SPARC T5-1B SPARC T5-2 SPARC T5-4 SPARC T5-8
Processor SPARC T5 3.6GHz SPARC T5 3.6GHz
SPARC T5 3.6GHz
SPARC T5 3.6GHz
Max Processor Chips 1 2 4 8
Max Cores/Threads 16, 128 32, 256 64, 512 128, 1024
DIMM Slots 16 32 64 128
Max Memory 128GB or 256GB 256GB or 512GB 1TB or 2TB 2TB or 4TB
Drive Bays 2 6 8 8
I/O Slots 2 x PCIe 2.0 EM, 2 NEM,1 FEM slots
8 LP x8 PCIe 3.0, 4 x 10GbE ports
16 LP x8 PCIe 3.0,4 x 10GbE ports
16 LP x8 PCIe 3.0,4 x 10GbE ports
Form Factor/RU Blade Rack 3RU Rack 5RU Rack 8RU
Max Power Consumption 689W 1927W 2410W 4850W
SPARC T5 ServersProduct Line Overview
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal7
New l SPARC T5-1B Blade Server
Compute1x SPARC T5 16-core CPU
16x DDR3 DIMMs
128GB (8GB DIMMs) or 256GB (16GB DIMMs) memory
I/O and Storage2x hot-plug PCIe 2.0 x8 Express Modules
2x 2.5” SAS HDD or SSD drives
Availability and ManagementBuilt-in RAID 0, 1, 1E
Hot plug disks
Integrates with Sun Blade 6000 network architecture
Oracle ILOM service processor
Next Generation 1-socket SPARC Blade Server
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal8
T5-1B Board
T5 CPU
16 DDR3DIMMs
ServiceProcessor
REM
BoBs
PCIeSwitch
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal9
New l SPARC T5-2 Enterprise Server
Compute2x SPARC T5 16-core CPU32x DDR3 DIMMs
256GB (8GB DIMMs) or 512GB (16GB DIMMs) of memory
I/O and Storage8x PCIe 3.0 x8 slots4x 10G-baseT ports6x 2.5” SAS HDD or SSD drives
Availability and ManagementBuilt-in RAID 0, 1, 1EHot-plug disksHot-swap and redundant fans and power suppliesOracle ILOM service processor
Next Generation 2-socket SPARC Server
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal10
T5-2 ChassisT5 CPU
ServiceProcessor
MemoryRisers
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal11
T5-2 Front
DISK 1
DISK 0
DISK 3DISK 2
DISK 5DISK 4
RFID/SerialNumber
2x USB 3.0Ports
HD-15 VGA Port
DVD
LocatorLED/Button
Fault LEDPowerButton
StatusLED
Over Temp Indicator
Status LEDs
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal12
T5-2 Rear
2x USB 3.0
HD-15 VGA Port
4x 10GbE Ports
SP Serial SP Network 10/100
PC
Ie 1
PC
Ie 2
PC
Ie 3
PC
Ie 4
PC
Ie 5
PC
Ie 6
PC
Ie 7
PC
Ie 8
System RearIndicators
AC0AC1
PS
U 0
PS
U 1
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal13
New l SPARC T5-4 Enterprise Server
Compute4x SPARC T5 16-core CPU
64x DDR3 DIMMs
1TB (16GB DIMMs) or 2TB (32GB DIMMs) memory
I/O and Storage16x PCIe 3.0 x8 slots, w/ carriers
4x 10G-baseT ports
8x 2.5” SAS HDD or SSD drives
Availability and ManagementBuilt-in RAID 0, 1, 1E
Hot-plug disks, PCI cards
Hot-swap and redundant fans and power supplies
Oracle ILOM service processor
Next Generation 4-socket SPARC Server
Front
Rear
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal14
SPARC T4-4 Comparison SPARC to T5-4
Feature SPARC T5-4 SPARC T4-4Form Factor 5RU, 31.5” deep 5RU, 28” deep
CPU 4x SPARC T53.6 GHz (512 threads)
4x SPARC T43.0 GHz (256 threads)
MemoryDDR3, 2TB MAX
64x SlotsDDR3, 2TB MAX
64x slots
Network 4x 10GbE 4 x 1GbE + 8x 10GbE (XAUI) Requires 2 Separate QSFP Connectors
Internal Storage Up to 8 x 2.5” SAS 3.0 or SSD, hot-plugUp to 8 x 2.5” SAS 2.0, can use up to
4x SATA SSDs, hot-plug
Removable Media 1x DVD-RW (via rKVMS; not local) 1x DVD-RW (via rKVMS; not local)
Serial 1x RS-232, 4x USB, 1x VGA 1x RS-232, 4x USB, 1x VGA
PCI Express slots16x x8 slots (Hot-Plug Low Profile slots
with carrier card, ver 3.0)16x x8 slots (Hot-Plug Express Module slots,
ver 2.0)
Power Supply 2 x 3000 Watt AC, N+NRedundant/Hot-Swap
4 x 2060 Watt AC, N+NRedundant/Hot-Swap
Fans 5 x Redundant Hot-Swap 5 x Redundant Hot-Swap
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal15
SPARC T5-4 Front Panel
DISK 0DISK 1
DISK 2DISK 3
DISK 4DISK 5 DISK 7
DISK 6
PSU 0 PSU 1
LocatorLED/Button
Over Temp Indicator
Processor Module
Fault LED
PowerButton
Status LED
Processor Module Status LEDs
Main Module(Entire Board)
Dual USB 2.0PortsSP Serial
Port
RFID/SerialNumber
PSU StatusLEDs
Main Module &SP Status LEDs
HD-15 VGA Port
Rear Fan/EM Indicator
PM 0PM 1 Main
Module
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal16
AC3 AC0
SPARC T5-4 Rear Panel
Fan Module 0 Fan Module 2 Fan Module 3 Fan Module 4
PC
Ie 1
PC
Ie 2
PC
Ie 3
PC
Ie 4
PC
Ie 5
PC
Ie 6
PC
Ie 7
PC
Ie 8
PC
Ie 9
PC
Ie 10
PC
Ie 11
PC
Ie 12
PC
Ie 13
PC
Ie 14
PC
Ie 15
PC
Ie 16
Rear I/OModule (RIO)
AC3 OKLED
System RearIndicators
PCIe CarrierHot-Plug Button, LEDs
Fan Module 1
HD-15 VGA Port
AC0 OK LED
SP Serial SP Network 10/100 2x USB 3.04x 10GbE Ports
C19 PlugC19 Plug
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal17
New l SPARC T5-8 Enterprise Server
Compute8x SPARC T5 16-core CPU
128x DDR3 DIMMs
2TB (16GB DIMMs) or 4TB (32GB DIMMs) memory
I/O and Storage16x PCIe 3.0 x8 slots, w/ carriers
4x 10G-baseT ports
8x 2.5” SAS HDD or SSD drives
Availability and ManagementBuilt-in RAID 0, 1, 1E
Hot-plug disks, PCI cards
Hot-swap and redundant fans and power supplies
Oracle ILOM service processor
Next Generation 8-socket SPARC Server
Front
Rear
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal18
Comparing T3, T4, and T5 SPARC T3 SPARC T4 SPARC T5
Processor Clock1.65GHz 2.85GHz, 3.0GHz 3.6GHz
# of cores 16 8 16
Core Architecture S2 S3# of memory controllers 2 4
DIMMs/BoB
2 BoBs/memory controller4 DIMMs/BoB
4 BoBs/CPU socket
2 BoBs/memory controller2 DIMMs/BoB
8 BoBs/CPU socketPCIe Gen support 2.0 3.0
PCIe Card Form Factor
Blade: Express Modules1 and 2 socket rack server: LP
4 socket server: Express Modules
Blade: Express Modules1-2 socket rack server: LP
4-8 socket rack server: LP on carrier card
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal19
T5-4 and T5-8 Processor Module
T5 Processor 1
T5 Processor 0
16 DIMMs per T5
16 DIMMs per T5
2 BoBs per Memory Controller in the T5
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal20
C0 C1 C2 C3 C4 C5 C6 C7
SSI, Present, 12C(FPGA connectivity)
FPGA-T5-48
C0 C1 C2 C3 C4 C5 C6 C7
SSI, Present, 12C(FPGA connectivity)
FPGA-T5-48
C0 C1 C2 C3 C4 C5 C6 C7
SSI, Present, 12C(FPGA connectivity)
FPGA-T5-48
C0 C1 C2 C3 C4 C5 C6 C7
SSI, Present, 12C(FPGA connectivity)
FPGA-T5-48
6
6
6
6
6
6
66
5 5
5 5
5
5
5 5
4
4
4
4
4
4
4
4
3 3
3 3
3
3
3
3
2
2
2
2
2
2
22
1
1 1
1 1
1
1
1
00
0
0 0
0
0 0
PM3
PM2
PM1
PM0
C6 C7
C4 C5
C2 C1
C0 C1
6
6
6
6
66
5 5
5
5
5 5
4
4
4
4
4
4
3 3
3
3
3
3
2
2
2
2
22
1
1 1
1
1
1
00
0 0
0 0
PM3
PM1
PM0
C6 C7
C2 C1
C0 C1
6
6
66
5 5
5 5
4
4
4
4
3 3
33
2
2
22
1
1 1
1
00
0 0
PM3
PM0
C6 C7
C0 C16
6
5 5
4
4 33
22
110 0
PM0
C0 C1
PFM not connected (required for airflow/EMI only
CL routing CL routing CL routing CL routing1 CL between the 8 nodes 1 CL between the 6 nodes 2 CL between the 4 nodes 2 CL between the 2 nodes
T5-8 (8P option) T5-8 (6P option) T5-8 (4P option) T5-8 (2P option)
C00 1
C10 1
PMOC20 1
C30 1
PM1C40 1
C50 1
PM2C60 1
C70 1
PM3
Part #Sw 0
Partition #Switch 1
Partition #Switch 2
Partition #Switch 2
Part #Sw 4
MP8?
PCIe switches on MB in commonT5-4/8 Main Module
PCIe upstream routing (PSR)
0 0 0 01 1 2 22 1 1 03 3 3 1
C00 1
C10 1
PMOC20 1
C30 1
PM1 PFM2C60 1
C70 1
PM3
Part #Sw 0
Partition #Switch 1
Partition #Switch 2
Partition #Switch 2
Part #Sw 4
MP8?
PCIe switches on MB in commonT5-4/8 Main Module
PCIe upstream routing (PSR)
0 0 0 01 1 2 23 3 3 1
C00 1
C10 1
PMO PFM1 PFM2C60 1
C70 1
PM3
Part #Sw 0
Partition #Switch 1
Partition #Switch 2
Partition #Switch 2
Part #Sw 4
MP8?
PCIe switches on MB in commonT5-4/8 Main Module
PCIe upstream routing (PSR)
0 0 0 03 3 3 1
C00 1
C10 1
PMO PFM1 PFM2 PFM3
Part #Sw 0
Partition #Switch 1
Partition #Switch 2
Partition #Switch 2
Part #Sw 4
MP8?
PCIe switches on MB in commonT5-4/8 Main Module
PCIe upstream routing (PSR)
0 0 0 0
PFM1
PFM (not required)Or Empty PM slot
PFM (not required)Or Empty PM slot
Note: the T5-8 2P option isnot a shipping configuration
PFM1
PFM2PFM2
T5-8 Processor Connectivity
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal21
C0 C1 C2 C3 C4 C5 C6 C7
SSI, Present, 12C(FPGA connectivity)
FPGA-T5-48
C0 C1 C2 C3 C4 C5 C6 C7
SSI, Present, 12C(FPGA connectivity)
FPGA-T5-48
6
6
66
5 5
5 5
4
4
4
4
3 3
33
2
2
22
1
1 1
1
00
0 0
PM1
PM0
C6 C7
C0 C16
6
5 5
4
4 33
22
110 0
PM0
C0 C1
CL routing CL routing2 CL between the 4 nodes 2 CL between the 2 nodes
T5-8 (4P option) T5-8 (2P option)
C00 1
C10 1
PMOC60 1
C70 1
PM3
Part #Sw 0
Partition #Switch 1
Partition #Switch 2
Partition #Switch 2
Part #Sw 4
MP4
PCIe switches on MB in commonT5-4/8 Main Module
PCIe upstream routing (PSR)
0 0 0 03 3 3 1
C00 1
C10 1
PMO PFM1
Part #Sw 0
Partition #Switch 1
Partition #Switch 2
Partition #Switch 2
Part #Sw 4
MP4
PCIe switches on MB in commonT5-4/8 Main Module
PCIe upstream routing (PSR)
0 0 0 0
PFM1
T5-4 Processor Connectivity
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal22
x2
x8
x8LSI SAS
x1
x1
Disk1
Disk2
x4
NEM0 NEM1 PCI-EM1
x8x8
FEM0 FEM1 Nalia Niantic
1x4
? here
USB 3.0Host Ctrl USB 1.0
Hub Ctrl
Sideband Mgmt
USB 2.0 EnetStorage Emulex
Pilot 3
VGA(HD15)
Front UCP (Dongle Cable)
Serial(RJ45)
Ethernet Mgmt(to CMM)
NEM0(4:7)
BoB BoBBoB BoBBoB BoB BoB BoB
DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM
T5CPU0
LPC 12C
FPGA
Host &Data Flash
TPM
Dual GigE10/100/1000
x8
NEM0(0:3)
NEM1(0:3)
NEM1(4:7)
1x4 1x4 1x4
PCIeSwitch 0
PCIeSwitch 1
CPUDC/DCs
CPUDebug
PortPCIe
x8PCIe Gen3PCIe Gen2
DBG
SP Module
USB 3.0
USB 2.0
USB
MIDPLANE
T5-1B Block Diagram
USB 1.1 Keyboard/Mouse
PCI-EM0
USB
0 1
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal23
T5-0
CPUDC/DCs
CPUDebugPort
PCIe
USB 2.0 Storage
get rid of all inside small boxes
EnetMgmt
10/100
SerialMgmt
BoB BoB
DIMM DIMM
T5-1
CPUDC/DCs
CPUDebug
Port
PCIe
BoB BoB
DIMM DIMM
BoB BoB
DIMM DIMM
BoB BoB
DIMM DIMM
BoB BoB
DIMM DIMM
BoB BoB
DIMM DIMM
Host &Data Flash
TPM
DBG
Service Processor
SP ModuleFRUID
DRAM
SPIFlash
NAND
USB 3.0 Host
VGADB15
Sideband Mgmt
USB0 USB1
VGA
USB 2.0 Hub
USB
PCIeSwitch 0
Slo
t 1
(8)
Slo
t 2
(8
)
Slo
t 3
(8)
Slo
t 4
(8)
Slo
t 5
(8)
Slo
t 6
(8)
Slo
t 7
(8
)
Slo
t 8
(8)
x8 x8
x8x8
x8x8
x8 x8
x1x1
HDD0
HDD0
HDD0
HDD0
HDD0
HDD0
SATA DVD
USB 3.0Hub
USB
VGA
SAS/SATAIO Controller
SAS/SATAIO Controller
Quad 10Gig Enet
BoB BoB
DIMM DIMM
BoB BoB
DIMM DIMM
x4
x4
FPGA
USB2 USB3
REAR IOFAN BOARD
x4
T5-2 Block Diagram
Internal USB
REAR IO Board
USB 1.1 Keyboard Mouse
VGA
0 1 0 1
PCIeSwitch 1
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal24
SAS 2308SAS1
SW6
DC-DCCONVERTERS
CLOCK SYNTH.&
BUFFERS
SW1 SW2 SW3SW0
Debug C
onn
4-DISK BACKPLANEDISKS 4-7SBP1 SBP04-DISK BACKPLANE
DISKS 0-3
VIDEOMUX
SerialMUX
SERVICE PROCESSOR SP
FPGA, TOD, TPM
10GB NIC 0 10GB NIC 1RIO Rear IO Module[VGA/DB15][Serial Mgmt][Enet Mgmt 10/100]
Quad 10Gig Enet (10GBase-T Copper)
1x USB (Gen2)
TO RIO TO RIO
Enet MgmtNC-SI
2x USB (Gen3)[VGA/DB15][Serial Mgmt]
HDD[4:7] HDD[0:3]
MP4, MP8T5-4 or T5-8
Midplane
CPUPCIE PORTS
CPUPCIE PORTS
CPUPCIE PORTS
CPUPCIE PORTS
1 2 3 4 5 6 7 8 9 10 11 12 13 14
CPUPCIE PORTS
15 16
SW4
EBPCI Express Backplane
MONITOR &CONTROL
MMMain Module
MBMotherboard
T5-4 & T5-8 I/O Block Diagram
RIO
Net 0
REAR USBMOST CTRLLR.
2x USB (Gen3)
SAS 2308SAS0
FRONT USBHOST CTRLLR.
FIO + VGAFRONT IOVGA, SERIAL MGMT, USB
x2 USB Ports
Debug C
onn
SW5
Net 1
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal25
CLR0
CLR0
Memory – Processor Module Block Diagram (T5-8 and T5-4)
L0FSR0
L1FSR1
L0FSR2
L1FSR3
L0FSR4
L1FSR5
L0FSR6
L1FSR7
T5 (CM1)MCU0 MCU1 MCU2 MCU3
IOS0
IOS1
CLR1
CLR2
CLR3
CLR4
CLR5
BoB0
C C0 1
BoB2
C C1 0
BoB4
C C0 1
BoB6
C C0 1
BoB1
C C0 1
BoB3
C C1 0
BoB5
C C1 0
BoB7
C C1 0
CLR6
CMO / MCU0 / L0 = FSR0 = BOB0CMO / MCU0 / L1 = FSR1 = BOB1CMO / MCU1 / L0 = FSR2 = BOB2CMO / MCU1 / L1 = FSR3 = BOB3CMO / MCU2 / L0 = FSR4 = BOB4CMO / MCU2 / L1 = FSR5 = BOB5
L0FSR0
L1FSR1
L0FSR2
L1FSR3
L0FSR4
L1FSR5
L0FSR6
L1FSR7
T5 (CM0)MCU0 MCU1 MCU2 MCU3
IOS0
IOS1
CLR1
CLR2
CLR3
CLR4
CLR5
BoB2
C C0 1
BoB1
C C1 0
BoB4
C C0 1
BoB5
C C0 1
BoB3
C C0 1
BoB0
C C1 0
BoB7
C C1 0
BoB6
C C1 0
CLR6
CMO / MCU0 / L0 = FSR0 = BOB0CMO / MCU0 / L1 = FSR1 = BOB1CMO / MCU1 / L0 = FSR2 = BOB2CMO / MCU1 / L1 = FSR3 = BOB3CMO / MCU2 / L0 = FSR4 = BOB4CMO / MCU2 / L1 = FSR5 = BOB5
Xcede HD 6-Row Midplane Connector
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal26
T5 PCIe Subsystem
Dual x8 PCI Express Gen 3 ports provide 32 GB/s peak b/w Supports Atomic Fetch-and-Add, Unconditional-Swap and Compare-and-Swap operationsAccelerates virtualized I/O with Oracle Solaris VMs
128k virtual function address spaces ensure direct SR-IOV access for all logical domains 64-bit DVMA space reduces IO mapping overhead, improving network performanceGuarantees fault and performance isolation among guest OS instances
Supports PCI Express Power Management
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal27
T5 PCIe Progression
T4 T5
PCI Express revision Gen 2 (dual x8 ports) Gen 3 (dual x8 ports)
Throughput full duplex 16 GBs 32 GBs
Data Management Unit Single shared unit for both x8 PCIe ports
Two independent units one for each x8 PCIe port
Physical Address Support 44 bit 48 bit
Transaction Id Identification on MSI and MSI-X No Yes
PCIe Atomic Transactions No Yes
TLP Processing Hints No Yes, directs data to L3 cache
PCIe 2.0 compliance (ECN “Internal Error Reporting”) Signaled via MSI interrupt Signaled via PCIe message
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal28
T5 and M5 PCIe Carrier Card
Supports standard low-profile PCIe cards
x16 Connector(x8 electrical)
PCIe Retimer
Air Flow
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal29
About the F40 Flash Card (Aura2)
Supported on T5-2, T5-4, T5-8, and M5-32Available on T5-2 and M5-32 at RRAvailable on T5-4 and T5-8 at RR + 1Q
Special carrier card for T5-4 and T5-8 that will span 2 PCIe slots. Required due to thermal characteristics.
Hot-plug support on T5-4, T5-8, and M5-32 at RR + 1QCan not be placed in Slot 8 of IOU on M5-32 due to thermal issues.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal30
CPU0 CPU1
0 1 0 1
PM0
x8 up.Part. 0
x8 up.Part. 0
x8 up.Part. 0
x8 up.Part. 0
x8 up.Part. 0
NET1 x8
SAS0 x8
Front USB x1
Debug Slot x1
SP VGA x1
Rear USB x1
SAS1 x8
NET0 x8
Debug Slot x1
1x8c0
2x8c0
4x8c0
3x8c0
9x8c1
10x8c1
11x8c1
12x8c1
5x8c0
6x8c0
7x8c1
8x8c1
13x8c1
14x8c1
15x8xx
16x8xx
MP
MP
EBPCI-ExpressLow Profile
Hot Plug Slots
Dotted linedevicesresideon RIO
T5-4/8 Native 2-Socket Configuration with One Root Domain• Single non-redundant Domain• Second level Switch 6 is partitioned
differently from other configs
• Block fill color identifies Root Domain ownership• Block outline color identifies association to PM• Switch 2 Slots crossed to maintain consistent Slot population order Slots drawn in order from left to right as in the actual chassis
x8 up.Part. 0
x8 up.Part. 1
x8 up.Part. 0
x8 up.Part. 1
{
Switch 0 Switch 1 Switch 2 Switch 3 Switch 4
Sw
itch 5S
witch 6
Native Config(T5-4 only)
7Config
ID
PM1
PFM
x8 up.Part. 1
T5-4: 1 PM PCIe Connectivity
Slot #8 lanesCPU #
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal31
CPU0 CPU1
0 1 0 1
PM0
CPU2 CPU3
0 1 0 1
PM1
x8 up.Part. 0
x8 up.Part. 0
x8 up.Part. 3
x8 up.Part. 0
x8 up.Part. 3
x8 up.Part. 0
x8 up.Part. 3
x8 up.Part. 1
NET1 x8
SAS0 x8
Front USB x1
Debug Slot x1
SP VGA x1
Rear USB x1
SAS1 x8
NET0 x8
Debug Slot x1
1x8c0
2x8c0
4x8c0
3x8c0
9x8c1
10x8c1
11x8c1
12x8c1
5x8c2
6x8c2
7x8c2
8x8c2
13x8c3
14x8c3
15x8c3
16x8c3
MP
MP
EBPCI-ExpressLow Profile
Hot Plug Slots
x8 up.Part. 0
x8 up.Part. 1
x8 up.Part. 0
x8 up.Part. 1
{
Switch 0 Switch 1 Switch 2 Switch 3 Switch 4
Sw
itch 5S
witch 6
Native Config6Config
ID
T5-4: 2 PM PCIe Connectivity
T5-4/8 Native 4-Socket Configuration with One Root Domain• Single non-redundant Domain• Second level Switch 6 is partitioned
differently from other configs
• Block fill color identifies Root Domain ownership• Block outline color identifies association to PM• Switch 2 Slots crossed to maintain consistent Slot population order Slots drawn in order from left to right as in the actual chassis
Dotted linedevicesresideon RIO
Slot #8 lanesCPU #
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal32
CPU0 CPU1
0 1 0 1
PM0 PM1 PM2
CPU6 CPU7
0 1 0 1
PM3
x8 up.Part. 0
x8 up.Part. 0
x8 up.Part. 3
x8 up.Part. 0
x8 up.Part. 3
x8 up.Part. 0
x8 up.Part. 3
x8 up.Part. 1
NET1 x8
SAS0 x8
Front USB x1
Debug Slot x1
SP VGA x1
Rear USB x1
SAS1 x8
NET0 x8
Debug Slot x1
1x8c0
2x8c0
4x8c0
3x8c0
MP
MP
EBPCI-ExpressLow Profile
Hot Plug Slots
x8 up.Part. 0
x8 up.Part. 1
x8 up.Part. 0
x8 up.Part. 1
{
Switch 0 Switch 1 Switch 2 Switch 3 Switch 4
Sw
itch 5S
witch 6
Native Config6Config
ID
PFMPFM
T5-8: 2 PM PCIe Connectivity
T5-8 Native 4-Socket Configuration with Two Root Domains• Two slot per Root Complex• Two Redundant Path Root Domains
• Block fill color identifies Root Domain ownership• Block outline color identifies association to PM• Switch 2 Slots crossed to maintain consistent Slot population order Slots drawn in order from left to right as in the actual chassis
9x8c1
10x8c1
11x8c1
12x8c1
5x8c6
6x8c6
7x8c6
8x8c6
13x8c7
14x8c7
15x8c7
16x8c7
Dotted linedevicesresideon RIO
Slot #8 lanesCPU #
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal33
CPU0 CPU1
0 1 0 1
PM0
CPU2 CPU3
0 1 0 1
PM1 PM2
CPU6 CPU7
0 1 0 1
PM3
x8 up.Part. 0
x8 up.Part. 1
x8 up.Part. 0
x8 up.Part. 1
x8 up.Part. 3
x8 up.Part. 0
x8 up.Part. 2
x8 up.Part. 3
x8 up.Part. 0
x8 up.Part. 2
x8 up.Part. 3
x8 up.Part. 1
NET1 x8
SAS0 x8
Front USB x1
Debug Slot x1
SP VGA x1
Rear USB x1
SAS1 x8
NET0 x8
Debug Slot x1
1x8c0
2x8c2
4x8c2
3x8c0
9x8c1
10x8c1
11x8c1
12x8c1
5x8c6
6x8c6
7x8c3
8x8c6
13x8c3
14x8c7
15x8c7
16x8c7
MP
MP
EBPCI-ExpressLow Profile
Hot Plug Slots
x8 up.Part. 0
x8 up.Part. 1
x8 up.Part. 0
x8 up.Part. 1
{
Switch 0 Switch 1 Switch 2 Switch 3 Switch 4
Sw
itch 5S
witch 6
Native Config2Config
ID
PFM
T5-8: 3 PM PCIe Connectivity
T5-8 Native 6-Socket Configuration with Two Root Domains• Two slot per Root Complex• Two Redundant Path Root Domains
• Block fill color identifies Root Domain ownership• Block outline color identifies association to PM• Switch 2 Slots crossed to maintain consistent Slot population order Slots drawn in order from left to right as in the actual chassis
Dotted linedevicesresideon RIO
Slot #8 lanesCPU #
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal34
CPU0 CPU1
0 1 0 1
PM0
CPU2 CPU3
0 1 0 1
PM1
CPU4 CPU5
0 1 0 1
PM2
CPU6 CPU7
0 1 0 1
PM3
x8 up.Part. 0
x8 up.Part. 1
x8 up.Part. 0
x8 up.Part. 1
x8 up.Part. 2
x8 up.Part. 3
x8 up.Part. 0
x8 up.Part. 1
x8 up.Part. 2
x8 up.Part. 3
x8 up.Part. 0
x8 up.Part. 1
x8 up.Part. 2
x8 up.Part. 3
x8 up.Part. 0
x8 up.Part. 1
NET1 x8
SAS0 x8
Front USB x1
Debug Slot x1
SP VGA x1
Rear USB x1
SAS1 x8
NET0 x8
Debug Slot x1
1x8c0
2x8c2
4x8c2
3x8c0
9x8c1
10x8c3
11x8c1
12x8c3
5x8c4
6x8c6
7x8c4
8x8c6
13x8c5
14x8c7
15x8c5
16x8c7
MP
MP
EBPCI-ExpressLow Profile
Hot Plug Slots
x8 up.Part. 0
x8 up.Part. 1
x8 up.Part. 0
x8 up.Part. 1
{
Switch 0 Switch 1 Switch 2 Switch 3 Switch 4
Sw
itch 5S
witch 6
Native Config 00Config
ID
T5-8: 4 PM PCIe Connectivity
T5-8 Native 8-Socket Configuration with Two Root Domains• Two slot per Root Complex• Two Redundant Path Root Domains
• Block fill color identifies Root Domain ownership• Block outline color identifies association to PM• Switch 2 Slots crossed to maintain consistent Slot population order Slots drawn in order from left to right as in the actual chassis
Dotted linedevicesresideon RIO
Slot #8 lanesCPU #
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal35
PCIe Data Paths: Full System
Two root complexes per T5 processorEach PCIe port on a T5 processor controls a single PCIe slot
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal36
Root Complex
Each processor’s root complexes must be assigned to the same domainExample:
/pci@300/pci@1/pci@0/pci@6 & /pci@340/pci@1/pci@0/pci@6 from CPU0 are assigned to the same logical domain
PM CPU Switch I/O Slot Root Complex Path
0 0 0 1 /pci@300/pci@1/pci@0/pci@6
0 0 1 3 /pci@340/pci@1/pci@0/pci@6
0 1 2 9 /pci@380/pci@1/pci@0/pci@a
0 1 3 11 /pci@3c0/pci@1/pci@0/pci@e
1 2 0 2 /pci@400/pci@1/pci@0/pci@c
1 2 1 4 /pci@440/pci@1/pci@0/pci@c
1 3 2 10 /pci@480/pci@1/pci@0/pci@4
1 3 3 12 /pci@4c0/pci@1/pci@0/pci@8
2 4 1 5 /pci@500/pci@1/pci@0/pci@e
2 4 2 7 /pci@540/pci@1/pci@0/pci@e
2 5 3 13 /pci@580/pci@1/pci@0/pci@a
2 5 4 15 /pci@5c0/pci@1/pci@0/pci@8
3 6 1 6 /pci@600/pci@1/pci@0/pci@8
3 6 2 8 /pci@640/pci@1/pci@0/pci@8
3 7 3 14 /pci@680/pci@1/pci@0/pci@4
3 7 4 16 /pci@6c0/pci@1/pci@0/pci@4
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal37
Zakim (ZK) – M5/T5 Systems Memory Interface
Also referred to as a BoB (Buffer-on-Board)Features and Technology:
4 ZKs per M5 memory controller2 ZKs per T5 memory controllerSupports DDR3 DIMMsMemory Link to DDR Interface2 Memory Link Ports, 2 DDR ports8 Write FIFOsPass through commands to DIMMsCore frequency: 1066 MHz
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal38
T5-4 and T5-8 Memory Config Rules
• All DIMMs must have the same Oracle Part Number on a PM (Processor Module).
• All PM's must be fully populated (32 DIMMs)• No partial memory configs available at RR
• All PM's in the system must be configured the same (ATO specific rule)
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal39
T5-4 and T5-8 Airflow/Cooling
T5-8(top 3U)
T5-4(5U)
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal40
SPARC T5 Disk Drives
• Disk drives are 2.5” form factor• Chassis supports SAS HDD and SATA SSDs• SAS-2 HDDs
• 300GB @ 10K RPM• 600GB @ 10K RPM• 900GB @ 10K RPM
• SATA SSDs• 100GB• 300GB
Ready to Remove
FaultStatus
Disk LED's
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal41
SPARC T5 Disk Controller
T5 has Dual LSI SAS2008 8port SAS3/SATA2 controllersSupport for RAID 0 (striping) and RAID 1 (mirroring) using 'raidctl'
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal42
The SPARC T5 Processor
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal43
SPARC T5 Processor
• Features• 16 S3 cores, 16-128 Strands
@ 3.6Ghz• Single or multi-threaded
operation per core• System scalability to 8
sockets• SPARC Core S3
• 1-8 Strand Dynamically Threaded Pipeline
• ISA-based Crypto-acceleration
• 8MB Shared L3$
• Integrated I/O• Double I/O bandwidth over T4• 2 x8 Lane PCIe 3.0 @ 8GT/s
• System Scalability• 7 Coherence Ports for scalability
to 8S• Power Management
• Dynamic Voltage Frequency Scaling
• Downclock, Overclock
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal44
SPARC T5 CPU Block Diagram
2 x
8 P
CIe
3.0
@ 8
GB
ps16
GB
ps e
ach
dir
ectio
n
8 threads per Core
IOSubsystem
BoB BoB BoBBoB BoB BoB BoB
DDR3 – 1066 MHz
Memory Control
Co
he
ren
cy 4
x4
Sw
itch
Memory Control
Memory Control
Memory Control
8 x 9 Crossbar (~1TBps bandwidth)
C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15
L3$ B0L3$ B01MB,16-way
L3$ B01MB,16-way
L3$ B21MB,16-way
L3$ B11MB,16-way
L3$ B31MB,16-way
L3$ B0L3$ B01MB,16-way
L3$ B41MB,16-way
L3$ B61MB,16-way
L3$ B51MB,16-way
L3$ B71MB,16-way
128 KB L2$
16 KB L1I$
16 KB L1D$
FGU
Crypto
SPARC S3Core
BoB
DDR3 – 1066 MHz DDR3 – 1066 MHz DDR3 – 1066 MHz
Coherence Unit Coherence Unit Coherence Unit Coherence Unit Link 0
Link 1
Link 2
Link 3
Link 4
Link 5
Link 6
Coherency Links 12.8 Gbps per lane - 12 lanes per link
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal45
T5 Processor Overview§ 16 S3 cores @ 3.6GHz
§ 8MB shared L3 Cache
§ 8 DDR3 BL8 Schedulers providing 80 GB/s BW
§ 8-way 1-hop glueless scalability
§ Integrated 2x8 PCIe Gen 3
§ Advanced Power Management with DVFSSPAR
C Core
SPARC
Core
SPARC
Core
SPARC
Core
SPARC
Core
SPARC
Core
SPARC
Core
SPARC
Core
SPARC
Core
SPARC
Core
SPARC
Core
SPARC
Core
SPARC
Core
SPARC
Core
SPARC
Core
SPARC
Core
Cross Bar
MCU
Coherence
L3 L3 L3 L3
L3 L3 L3 L3
SerDes
SerDesSerDes MI/O
SerD
es
SerD
es
MCU
MCU
MCU
PCIe Gen3
Pwr
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal46
SPARC T5: Processor Key Features
SPARC T5 based on SPARC S3 core: Same core as used in T416 S3 cores, dual PCI Express 3.0 root complexes
Up to eight T5 processors per system, 16 cores x 8 threads → 128 (on T5-1B) or up to 1024 threads (on T5-8)
Clock frequency is 3.6 GHz
Each SPARC S3 core contains:
● 2 Integer pipelines
● 1 FGU pipeline (consisting of 3 physical sub-pipelines):
– FPX pipeline– FGX pipeline– FPD pipeline
● 1 Load-Store (Memory) pipeline
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal47
SPARC T5: Processor Key Features
Each Core has two Level 1 cache memories, one for data and one for instructions, each 16KB in sizeEach Core has a Level 2 unified cache, 128KB in size
• Caches are all inclusive: L3 inclusive of L2; L2 inclusive of L1 (in this context, ‘inclusive’ refers to the fact a cached entry is always present in the next higher level of cache)
• Each core on SPARC T5 is capable of OoO execution, dual-issue of instructions but in order commit.
• Each core on SPARC T5 also includes cryptographic acceleration hardware, accessible via user-level instructions.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal48
SPARC T5: Processor Key Features
Memory – systems based upon SPARC T5 use DDR3 Quad Rank DIMMsDIMMs used are Registered DIMMS (RDIMMs)
16 DIMM slots per Processor→ memory capacity
depends on size of DIMM chosen for system
8 GB, 16 GB DIMM
ECC DIMMs required to be used at 1066 MT/s
Newly designed BoB between MCU and DIMMs
Four MCUs per processor; each maintains a memory link speed of 12.8 Gbps
Protocol between each MCU and its two BoBs is proprietary in nature
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal49
S3 Core Recap
28nm port from 40nm T4Out-of-order, dual-issueHigh frequency achieved with 3.6GHz 16 stage integer pipelineDynamically threaded, one to eight strandsAccelerates 16 encryption algorithms and random number generation
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal50
SPARC T4/T5/M5 Leads in On-Chip Encryption Acceleration
Processor / Mechanisms SPARC T4, T5, and M5 IBM Power7 IBM Power7+Intel Westmere/
Sandybridge
Operational ModelUserland unprivileged access
to on-core cryptographic functions
none3 accelerators shared
across 8 cores Userland
Asymmetric /Public Key Encryption RSA, DH, DSA, ECC none RSA, ECC RSA, ECC
Symmetric Key / Bulk Encryption AES, DES, 3DES, Camellia, Kasumi none AES AES
Message Digest / Hash Functions
CRC32c, MD5, Sha-1, SHA-224, SHA-256, SHA-
384, SHA-512none
MD5, SHA-1, SHA-256, SHA-512 none
Random Number Generation Supported none Supported none
API Support PKCS#11, Ucrypto APIs, JCE none PKCS#11 Intel IPP libraries
Virtualization SupportSolaris Zones
Oracle VM for SPARCnone ?? Intel VT
Crypto acceleration now available from 1 processors systems up to 32-way systems!
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal51
SPARC Core Roadmap
SPARC64 VII/VII+ Core
M-Series
T3 Servers
M5 Servers
T4 Servers
M6 ServersT5 Servers
S2 Core
S3 Core
S5 Core
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal52
S3 Core Overview
8-way threaded, dual-issue, OoO execution, in order commitDynamically threaded with hardware-optimized resource sharingSupport for Critical ThreadsDeep pipeline for high frequency operation (3 GHz in 40 nm)Balanced single-thread and multi-thread performance
- 5X better single-thread than SPARC T3 with equivalent multi-thread performance
Enhanced instruction set to accelerate Oracle SW stack- PAUSE, fused compare-branch
Integrated user-level cryptographic acceleration- DES/3DES, AES, Kasumi, Camellia, MD5, SHA-1, SHA-
224/256/384/512,RSA, DSA, CRC32cFoundation core for future technology / product nodes
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal53
Oracle SPARC S3 CoreEnables T4, T5, M5
• Design Goals- Develop a common replacement core for T-series processors
- Significantly improve the single strand performance of the T3 processor
- Significantly improve the throughput performance of the M3 processor
- Improve the RAS and power management capabilities- Maintain backward ISA, Solaris and OVM for SPARC compatibility
Core S1: M2, M3 Core S2: T2, T2+, T3 Core S3: T4, T5, M5
Frequency2.4 – 3.0 GHz 1.4 – 1.65 GHz T4: 2.85 – 3.0 GHz
T5, M5: 3.6 GHz
L1 Instruction Cache 64KB 16KB 16KB
L1 Data Cache 64KB 8KB 16KB
L2 Cache - - 128KB
# of Pipelines 1 2 1
# of Threads per Pipeline 2 4 1-8 Dynamic
Instructions per Thread 4 per cycle 1 per cycle 2 per cycle
Out of Order Issue Yes No Yes (36 instr window)
Cryptography Acceleration None SPU ISA Based
OVM for SPARC Compatible No Yes Yes
SPARC V9 ISA Compatible Yes Yes Yes
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal54
S3 Core: Dynamic Threading
While software can activate up to 8 strands on each core at a time, hardware dynamically and seamlessly allocates core resources such as instruction, data, and L2 caches and TLBs, and out-of-order execution resources such as the 128-entry re-order buffer in the core among the active strands. Software activates strands by sending an interrupt to a HALTed strand. Software deactivates strands by executing a HALT instruction on each strand it wants to deactivate. No strand has special hardware characteristics; all strands have identical hardware capabilities.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal55
S3 Core: Dynamic Threading
Since the core dynamically allocates resources among the active strands, there is no explicit "single-thread mode" or "multi-thread mode" for software to activate or deactivate. If software effectively halts all strands except one on a core via Critical Thread Optimization, the core devotes all of its resources to the sole running strand. Thus, that strand will run as quickly as possible
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal56
S3 Core: Critical Thread Optimization
The S3 core, starting with Oracle Solaris 10 8/11, can optimize performance by assigning one software thread exclusive access to all of a core's hardware resources. That software thread is considered to be a "critical thread." Solaris automatically detects opportunities to perform this assignment: when one software thread has high CPU utilization and there are more cores than runnable threads. We recommend that users allow Solaris to automatically perform Critical Thread assignment. A privileged user can tell Solaris that a particular software thread should be a critical thread, via the nice(1) command. Solaris will then assign that thread to a core, even if there are more runnable threads than cores.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal57
S3 Core: Critical Thread Optimization
Solaris Critical Threads optimization for S3 core, tries to provide exclusive access of certain hardware resources to certain application threadsSince the core dynamically allocates resources among the active strands, there is no explicit "single-thread mode" or "multi-thread mode" for software to activate or deactivateThere is no new API for declaring threads as ‘critical’; that would require significant changes to source codeRather, to invoke Critical Thread Optimization, use the following CLI or system calls to flag a thread as critical by raising its priority to 60:
priocntl(1)priocntl(2)priocntlset(2)
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal58
S3 Core: Critical Thread Optimization
Starting with Oracle Solaris 10 8/11, a thread is declared to be critical if raised to priority 60; the thread can be in any scheduling classIn Oracle Solaris 11, to be considered critical by the scheduler, a thread must be:
in the FX (Fixed Priority) or RT (Real-Time) scheduling classes
be raised to priority 60 by one of the previously mentioned mechanisms
In either of the above instances, this one thread will run as quickly as possible as it has exclusive access to all core resources
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal59
Critical Threads for key applications
Applicability Opportunity Current Status
Database Logwriter, LMS Up to 30% improvement in efficiency
LMS is already CT ready. LGWR planned for 12c
JAVA (JVM)Compiler threads, GC and priority mapping
support
Up to 2x improvement for app startup, Smooth GC
Support for JVM and JAVA apps to be CT aware is integrated in JDK7U4
Coherence Packet writer, service thread
Up to 20% improvement in throughput
Integrated in Coherence version 3.7.1 Patch 1
Solaris S11U1 / S10U11Improve CT perf to be within
10% of best case (hand optimized)
Optimizations for decayed PG util and stealing being
integrated in S11U1
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal60
Internode Coherency Overview
Glueless 1-hop scaling to eight socketsGlueless means no hub required to connect all 8 CPUs
A precise directory tracks all L3s in the system
striped across all processors stored in on-chip SRAMsflexible for different socket counts
Higher BW efficiency than snoop-based protocols enables better scaling
50% more effective bandwidth than comparable snoopy implementation
T -
T5 T5
T5 T5
T5 T5
T5T5
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal61
Internode Performance Optimizations
Speculative memory reads prior to cache line serialization in the directoryCache-to-cache line transfers between nodesDynamic congestion avoidance routes inter-node data around congested links
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal62
T5 System Interconnects
1-WaySingle Socket
2-WayDual Socket
4-Way
6-Way
8-Way
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal63
8 Socket Local Coherency & Data Interconnect
DIMMS
M5/T5
M5/T5
M5/T5
M5/T5
M5/T5
M5/T5
M5/T5 M5/T5 DIMMSDIMMS
DIMMS
DIMMS
DIMMS
DIMMS
DIMMS
POINT-TO-POINTLOCAL
INTERCONNECT
All-to-All InterconnectWhere the Node-to-Node Fabric is 12 diff pairs per link in each direction.
T5 interconnect bandwidth= 157.5 GB/sec T5-8 interconnect bandwidth= 1260 GB/sec
DDR3-1066 Memory BandwidthT5 is 133 GB/secT5-8 is 1064 GB/s
PCIe Gen3 Bandwidth8 diff pairs per portsAt 8Gb/sec ~8GB/sec/direction
~8Gb/sec/lane X 8 lanes = 64 Gb/sX 2 directions X 2 Ports/chip= 256Gb/s/chip= 32 GB/s per chip
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal64
Latency for T5
T5-2 T5-4 T5-8
Local Memory 136ns
Remote Memory 209ns
Cache to Cache 127ns 146ns 155ns
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal65
Scalability of T4 vs T5
T4 – Snoopy BasedCoherence Protocol
T5 – Directory BasedCoherence Protocol
By numbers8 node snoops will consume 25% Link B/W & increases linearly w/more nodes
8 node directory based will consume 5% of Link B/W
Feature
Address serialization is done at Home Node. Home Node broadcasts snoop request to all nodes. All nodes except the requesting node require to participate the snoop operation and provide snoop response back to requesting node.
Address serialization is done at Directory Node. Directory Node keeps track of which node hold each cache line. Eliminating the need for broadcasting, and relieve the L3$ from unnecessary foreign snoop operation.
Link BandwidthMessage broadcast and response consume a lot of link bandwidth.
Directory filter the snoops sent to the share nodes. Allow link bandwidth to be used more efficiently.
L3$ PerformanceL3$ need to participate every snoop request from any other node. The L3$ performance can be dropped due to lots of foreign snoop requests.
Only the L3$ from the selective node require to participate the foreign snoop operation. L3$ has less distraction from foreign snoop request.
Scalability Limited to small scale of system.Easy to scale to large number of processor environment.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal66
Memory Controller of T4 vs T5
T4 MCU T5 MCU
Memory Link Speed 6.4 Gb/s 12.8 Gb/s
Memory Link ProtocolLegacy Intel FBDIMM2 Protocol Advanced In-house Link Protocol
Memory Link Low Power Feature
Not Supported L0s, L1
Memory Buffer Intel Milbrook2 MB Advanced In-house MB
DDR3 Protocol Burst length of 4 Burst length of 8
DDR3 Speed 800/1066 1066
DDR3 Device 1Gb/2Gb 2Gb/4Gb
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal67
RAS
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal68
Definition of Terms
• Hot-plug:• refers to the fact that a component can be plugged and unplugged without
powering down the platform. It applies to both hot swap and hot service.• Hot service:
• refers to the ability to perform hot-plug operations, with the additional necessity of some operator actions (invocation of a CLI or actuating a hot service button on the component to be removed).
• The system will notify the user when it is safe to remove the component. • Typical examples would be PCIe Express modules.
• Hot swap: • refers to an operation where a component is unplugged and plugged in with
no interaction with the ILOM or domain required. • Typical examples here are a single RAID disk or a power supplies.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal69
SPARC T5 System RAS Overview
• Designed to minimize part count and operating temperature to enhance reliability
• End-to-end data protection detecting and correcting errors throughout server – ECC everywhere
• Processor and Memory protection• CPU core and thread off-lining• Memory with ECC, x4/x8 DRAM Extended ECC, page retirement, and lane failover
• Major components redundant & hot-pluggable• Fan, Power Supply, and internal disks• RAID capability for internal disks
• Fault Management Architecture (FMA) support on ILOM
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal70
End-to-End RAS
Built RAS from the inside outStart with the processor, then memory, system and IO, virtualization layer, and the OSAdd Oracle Solaris Cluster software for additional service availabilityFault Management Architecture (FMA) binds all the layers together
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal71
T5/M5 Systems RASS11 FMA• Diagnosis engine on SP• Auto reconfigure on failure• Soft Error Rate Discrimination (SERD)• Bad page retirement• OS and SP watchdogs• FMA Component hot-upgradeable
T5/M5 Processor• L1$ Tag, Status $ Data
• Parity protection• Retry on error• L2$/L3$ Data• SEC/DED protection• Cache-line Sparing• L2$/L3$ Tags• SEC/DED protection• Inline Correction• Cache-line Sparing• L2$/L3$ Status & Directory• SEC/DED protection• Inline Correction• Architectural RegistersL2 Cache• SEC/DED protection• Precise Trap andHypervisor Correction and Retry
System• Redundant SPs with automatic failover• Redundant clock boards• Diagnosis to the FRU level on first fault
Power and Cooling• Advanced Power Management• Redundant hot-swap fans• Redundant hot-swap AC/DC• Dual grid power
System I/O• PCI-Express end-to-end CRC• PCI Express link retry• Hot-plug low profile PCI Express cards• Redundant, hot-plug boot disks• Alternate connections between M5 and IOcontrollersMemory• SDRAM Soft Errors
• ECC Protection and Correction• Extended ECC Protection• 4-bit Correction• Pin Steering
• Channel Interconnect• CRC protection/Message Retry• Lane Sparing
Hypervisor• Enables software partitioning (LDoms)virtualization and failure containment• Processor support for error clearing, correction and collection
Central Directory and Switch• SEC/DED protection with in line correction• Physical domain isolation • CRC protected System Interconnect with message retry and lane sparing• Deconfigurable directory chips, no loss of functionality, minimized bandwidth loss• Redundant Scalability Switch BoardsUnique to M5
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal72
RAS Features T4-1/T4-2 T4-4 M4000/M5000 T5-2 T5-4/T5-8 M8000/M9000 M5-32
Dynamic Reconfiguration X X X X X X X
Processor L2$ Degradation X X X X X X X
Processor L1$ Degradation X X
L3$ Line Sparing X X X
Hot-plug PCIe X X X X X
Memory Mirroring X X
Memory Lane Sparing X X X
CPU interconnect Lane Sparing X X X
Hard Domains X X X
Logical Domains X X X X X
Redundant System Interconnect X X X X
Hot-plug CPU and Memory X
Redundant/Hot-plug SP X X
Redundant Clock X X
•Hot-swap power supplies, fans•Hot-plug disks•ECC Memory
•Extended-ECC Memory•Instruction Retry•Partial ASIC Recovery
Common Features
SPARC RAS Features
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal73
Fault Management
• Knowledge Articles in MOS• ILOM fdd Diagnosis• Faults and Alerts• No ALOM Compatibility• ILOM FMA Captive Shell• Sideband Service Processor Network Connection• New ILOM Fault Notification (SNMP Trap)• ASR Support• FMA on M5 ILOM also applies to T5 ILOM, except for M5 specific features
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal74
FMA – restricted shell in SPSH
To get FMA details in SPSHstart -script /SP/faultmgmt/shellReturns a faultmgmtsp> prompt
Available build-in commands:echo - Display information to user.
● Typical use: echo $?
helpExit – exits restricted shell
External commands:fmadm - Administers the fault management servicefmdump - Displays contents of the fault and ereport/error logsfmstat - Displays statistics on fault management operationsetcd - ereport injector
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal75
Fault Management on T5 systems
T5 CPU and Memory faults are now diagnosed by ILOM FMA's Fault Proxy is used to keep ILOM's fault manager in sync with Solaris' fault
manager. Both will display the sum of all faults in the system. Faults can be repaired from either side. Fault Proxy communicates via the Ethernet Over USB connection. IO faults are still diagnosed by Solaris.
Disabled Database (DDB) owned by ILOM For faults which diagnose resources as unusable, ILOM will add those resources to
the DDB. Resources excluded on next host reset. When faults are repaired, ILOM automatically updates the DDB. Bringing
components back online requires a host reset. Extended SP-POST (Power on Self Test)
Runs at SP boot. Tests devices on the SP FRU and its Ethernet port. Status stored and converted to ereports after ILOM boots.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal76
Fault proxy
IO ereports are “forwarded” from the SP to the control domain, and then on to any relevant IO domainFaults are “proxied” between the SP, the control domain and any IO domains to provide a single view of faults in the system.
Non-servicable faults such as memory faults are not proxied.
The SP and the control domain can view and manage all faults in the system.An IO domain can only view and manage faults local to the domain.
Control Domain IO Domain
LDC LDC
TCP/IP
SP
hostd
FETDip-transprt
ETM ETM
ETM
ETM
ETMip-transport
ereports ereports
faultsfaultsLDC
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal77
Degraded HW Configurations – T5-8
If a CPU has no configurable memory, i.e., all memory links are unconfigurable, then the CPU itself is unconfigurable
A T5-8 will not operate with only 5 or 7 CPUs configuredOne more CPU(s) must be chosen to be deconfiguredFor an 8-way, if we fault a CPU, we will offline the other CPU on the same PMFor a 6-way, if we fault a CPU, we will also offline the other CPU on the same PM
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal78
ASR Support
• SPARC T5 servers will be supported by ASR (Automatic Service Request) at release
• Continues use of sunHwTrapFaultDiagnosed SNMP notification
• Telemetry for ILOM fdd diagnosis
• Supports platform and FRU identity
• Supports multi-suspect list
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal79
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal80