project olympus 3u pcie expansion server deep dive… · project olympus rack fully compatible....
TRANSCRIPT
Project Olympus 3U PCIe Expansion Server Deep Dive
Raymond Miles, VP Architecture, ZT SystemsMark D. Chubb, Director Architecture ZT Systems
Executive Track
ZT Systems OverviewProject Olympus Building BlocksZT 3U PCIe Expansion Server ProductsChassisFlexible Riser Card OptionsPower Distribution BoardSlide Rails
Agenda
ZT Systems Overview
A solution design and manufacturing partner with a record of success serving the world’s largest data center customers
Leading provider of optimized servers and storage servers for hyperscale data centers
Founded in 1994 HQ in Secaucus, NJ; global sites & capabilities Private company
Built for Hyperscale Maximum design flexibility Continuity of Supply Industry leading achievement to aggressive delivery SLAs Full range of flexible service and support options Broad range of component options
ZT OCP Engagement• History
• Engaged since 2012• Gold Member since 2014• Engineering projects• Solution Provider• Sponsored 8 summits
• Products• 2U Open Storage (2014)• 1U XPO200 (2017)• XPO200-3UN PCIe Expansion System (2019)
Project Olympus HW Building Blocks• Motherboards: Flexible expansion
• Power Supply: 1KW three-phase
• Universal PMDU: AC power cord adaption
• Rack: 19” EIA-310 standard
• Rack Manager: Restful API or Redfish via Ethernet
Project Olympus is an
OCP Accepted™
Specification
Intel® SKU
3U PCIe Expansion Server3 x Load Balanced
1kW PSUs
Power Distribution
Board(PDB)
6 x FHFL, Double-wide 300W x16 PCIe Cards
+1 x FHHL Single-wide 75W x16 PCIe Card
OR
Project Olympus Rack Fully Compatible
2.5”/3.5” SSD or HDD
6 x 60mm Dual Rotor Fans
(room for 8 Fans)
Supports Slide Rail Kit for smoother travel
PSUs Blind mate into Rack PMDU
12 x FHFL, Single-wide 75W x16 PCIe Cards
+1 x FHHL Single-wide 75W x16 PCIe Card
Supports Project Olympus motherboards
SERVER
AMD® SKU
5 x FHFL, Double-wide 300W x16 PCIe Cards
+1 x FHHL Single-wide 75W x16 PCIe Card
ZT Solution: 3U PCIe Expansion Server(AMD® EPYC™ Based; ZT-XPO200-3UA)
Feature Qty DescriptionChassis 1 3U, 19” EIA310-D Compliant supporting Project Olympus PMDU connections
Motherboard 1 Project Olympus 2-Slot AMD® EPYC™ Motherboard (OCP leveraged) from ZT SystemsProcessor 2 AMD® EPYC™ 7551 (Naples), 180W, 32C, 2GHzMemory 16
(32max)32GB DDR4, DR (1DPC), 2667 R-DIMM; Total System Memory: 512GB
PCIe Riser 3 & 5 2 2-Slot Active PCIe x16 Riser Card with 48-lane PCIe SwitchPCIe Riser 4 1 2-Slot Passive PCIe x32 Riser CardGPU Card 4 (5 max) AMD® MI25 FHFL, Double-Wide, 300W PCIe x16 CardNetwork Card 1 10G Single Port SFP+ PCIe2.0 x8 5GT/sHDD/SSD 5 (8 max) M.2 960GB NVMe SSD, PCIe x4 110mmM.2 Riser 1 (2 max) 2-Slot M.2 Riser (in PCIe Slot 1)Security 1 TPM2.0 SPI ModuleSystem Fans 6 60mmx56mm Dual Rotor FanPower Supply 3 Project Olympus 1020W 3-Phase, non-LES PSUPower Distribution 1 PDB and Cable Harnesses to support 12V Power to MB, Risers, GPUs, and System Fans
Items in Orange are being provided to OCP by ZT Systems
ZT Solution: 3U PCIe Expansion Server(Intel® Xeon®-SP Based; ZT-XPO200-3UN)
Feature Qty DescriptionChassis 1 3U, 19” EIA310-D Compliant supporting Project Olympus PMDU connections
Motherboard 1 Project Olympus 2-Slot Intel® Xeon®-SP MotherboardProcessor 2 Intel® Xeon® Platinum 8168 (Skylake), 205W, 24C, 2.7GHzMemory 12
(24max)32GB DDR4, DR (1DPC), 2667 R-DIMM; Total System Memory: 384GB
PCIe Riser 3 & 5 2 4-Slot Active PCIe x16 Riser Card with 96-lane PCIe SwitchPCIe Riser 4 1 5-Slot Active PCIe x16 Riser Card with 96-lane PCIe Switch & Sliver to Edge Conn x8 CablesGPU Card 12 Nvidia® P4 Low Profile, Single-Wide, 75W PCIe x16 CardNetwork Card 1 10G Single Port SFP+ PCIe2.0 x8 5GT/sHDD/SSD 1 (4 max) M.2 960GB NVMe SSD, PCIe x4 110mmSecurity 1 TPM2.0 SPI ModuleSystem Fans 6 60mmx56mm Dual Rotor FanPower Supply 3 Project Olympus 1020W 3-Phase, non-LES PSUPower Distribution 1 PDB and Cable Harnesses to support 12V Power to MB, Risers, GPUs, and System Fans
Items in Orange are being provided to OCP by ZT Systems
3U PCIe Expansion System Options
ZT Systems: ZT-XPO200-3UA ZT Systems: ZT-XPO200-3UN
®
ZT Systems in process of submitting these 3U Server Design Specifications to OCP Community!
System Block Diagram
PCIe x16 SLOT #3
PCIe x16 SLOT #5
PCIe x32 SLOT #4
CPU1 CPU0
DIMMDIMM
DIMMDIMM
DIMMDIMM
DIMMDIMM
DIMMDIMM
DIMMDIMM
DIMMDIMM
DIMMDIMM
CHA
CHB
CHC
CHD
CHE
CHF
CHG
CHH
PEX8747
M.2 Module #3
M.2 Module #4
Oculink x8
BMC
BMCFlash #1 32 MB
BMCFlash #2 32 MB
DDR4512 MB
Debug Connector
VGADE-15
RJ45 BCM546121GBe PHY
NCSIConnector
DIMMDIMM
DIMMDIMM
DIMMDIMM
DIMMDIMM
DIMMDIMM
DIMMDIMM
DIMMDIMM
DIMMDIMM
CHA
CHB
CHC
CHD
CHE
CHF
CHG
CHH
GMI
GMI
GMI
GMI
BIOSFlash
32 MB
BIOSFlash
32 MB
TPM2.0
Internal USB 2.0
InternalUSB 2.0 Header
GL850GUSB2.0
HUB
SATA0 x1
SATA1 x1
SATA2 x1
SATA3 x1
BMC5781010G Controller
Front PanelSFP+
Front PanelSFP+
Front PanelUSB 3.0
Front PanelUSB 3.0
PCIE x8 SLOT #1
PCIE x8 SLOT #2
M.2 Module #1
M.2 Module #2
Mini SASHD x4 #1
Mini SASHD x4 #2
PCIe x16 SLOT #2
PCIe x16 SLOT #1
PCIe x16 SLOT #2
PCIe x16 SLOT #1
33MHz
480 Mb/s
480 Mb/s
480 Mb/s
480 M
b/s
USB2A
USB2B
LPC
SMBUS
PCIe
NCSI
SPI
DDR4
UART
RGMII
VGA
P2[15:0]
P0[15:0]
P1[15:0] P3[3:0]
P3[7:4]
P3[15:8]
USB1(3)
GMI1
GMI0
GMI3
GMI2
GMI3
GMI2
GMI1
GMI0
SPI
USB1(3)
LPCSMBUSP1[12]
P1[11:8]
P1[7:4]
P3[15:12]P3[11:8]P2[15:0]P0[15:8]
P0[7:0]
P1[0]P1[1]P1[2]P1[3]
P3[7:0]USB0(0)USB0(1)
10 Gb/s
Gen3 x8
10 Gb/s
5 Gb/s
5 Gb/s
6 Gb/s
6 Gb/s
6 Gb/s
6 Gb/s
Gen2 x1
Gen3 x8
Gen3 x8
Gen3 x16
Gen3 x4
Gen3 x4
Gen3 x4 or 4 SATA 6 Gb/s
Gen3 x4
Gen3 x16
Gen3 x16
Gen3 x16
Port 0
Port 8 Port 16
Gen3 x16
Gen3 x16
PEX8747
PCIe x16 SLOT #2
PCIe x16 SLOT #1
Port 0
Port 8 Port 16
Gen3 x16
Gen3 x16
2-Slot Active PCIe x16 Riser Card Layout 2 x PCIe x16 slots supporting (Slots #1-2, Bottom to Top):
• 2 x FHFL Double-Wide PCIe cards
• Requires PWRBRK_N (pin B30) signal for High Power PCIe Card throttling
Broadcom® 48-Lane PCIe Gen3 Switch (PEX8747)
• Switch Bifurcated into 2 x16 Ports (1 x16 Port to each x16 Slot)
• Switch is located on Backside as required by Chassis Thermals (Heatsink is required)
• Use of all 48-lanes and requires 15.5W
Differential Clock Buffer on board for additional Clock Fanout
I2C MUX used to avoid Address contention (see I2C block diagram)
8 Layer Stackup, Mid-Loss Material (0.062”)
Enclosed in a Riser Cage for easy Serviceability and Retention
12.68”
4.66”
PCIe x16 slot 2
PCIe x16 slot 1
Topside
2-Slot Passive PCIe x32 Riser Card Layout For Project Olympus AMD® EPYC™ MB Only (Middle Riser)
Uses 2 x HSEC8 connectors in-line for x32 Lanes
1 x PCIe x16 slot supporting (Slot #1, Bottom):• 1 x FHHL Single-Wide PCIe card
• Designated for the Network Card
1 x PCIe x16 slot supporting (Slot #2, Top):• 1 x FHFL Single-Wide PCIe Card OR
• 1 x FHFL Double-Wide PCIe Card
• All x16 PCIe lanes sourced from PCIe Riser Slot #4 (No Switch Required)
• Requires PWRBRK_N (pin B30) signal for High Power PCIe Card throttling
I2C MUX used to avoid Address contention (see I2C block diagram)
8 Layer Stackup, Mid-Loss Material (0.062”)
5.91”
3.62”PCIe x16 slot 2
PCIe x16 slot 1
2x2 Pwr
Other 3U Server PCIe Riser Card Options
TX: TOPRX: BOT
Port0
TX: L3/BOTRX: L8
Port4
Port12
Port20 Port16
Port8PEX8796
Slot 4
Slot 3
Slot 2
Slot 1 TX: L3/BOTRX: L8
TX: L3/BOTRX: L8
TX: L3/BOTRX: L8
TX: TOPRX: L3/BOT
JTAGPCIe x16
PCIe x8PCIe x8
TX: BOTRX: TOP/L4
Slot 2
Slot 1
TX: TOPRX: L4
TX: TOPRX: L4
TX: BOTRX: TOP/L6
Slot 3
Slot 2
Slot 1
TX: BOTRX: L6
TX: BOTRX: L6
TX: BOTRX: TOP
JTAG PCIe x16TX: BOTRX: TOP/L10
Port0(not used)
TX: L3/BOTRX: L8
Port4
Port12
Port20 Port16
Port8PEX8796
Slot 4
Slot 3
Slot 2
Slot 1
TX: L3/BOTRX: L8
TX: L3/BOTRX: L8
TX: L3/BOTRX: L8
TX: L3/BOTRX: TOP
Slot 5
JTAG PCIe x16
3-Slot Active PCIe x16 Riser Card
2-Slot Passive PCIe x16 Riser Card 4-Slot Active PCIe x16 Riser Card
5-Slot Active PCIe x16 Riser Card
Provides PCIe Expansion System Flexibility!
Middle PCIe Riser Card connectionsSliver Connector to PCIe x8 Edge Connector Cable• Provides additional PCIe Lane connections from unused PCIe
Slots
• 2 Cables provide PCIe x16 Lanes to Riser PCIe Slot or Riser PCIe Switch
• Supports Middle Riser Options:
• 2-Slot Passive x16 Riser Card
• 3-Slot Active x16 Riser Card
• 5-Slot Active x16 Riser Card
• Typically used on the Intel® Xeon®-SP System SKU
• ZT Partnered with Tyco Electronics® on design
PCIe I2C Block Diagram
BMCAST2520
I2C
PCA9548A0xE0
I2C_0
I2C_1
I2C_2
I2C_3
I2C_4
PCA9617
PCA9617
PCA9617
PCA9617
I2C_1
I2C_2
I2C_1
I2C_2
Riser 1, PCIe x8
Riser 2, PCIe x8
P3V3 P1V8_PCH_STBY
0
0
0
0
2.2K 2.2K
0
P3V3_STBY
4.7K
334.7K
I2C_5 0
P3V3_STBY
4.7K
0
P3V3_STBY
4.7K
I2C_6
P3V3_STBY
I2C_1
* 0 based
Bus Naming:1-based pin 0-based int
P3V3_STBY
NI4.7K
0I2C I2C_0
I2C_1I2C_2I2C_3
0
P3V3_STBY
ISOLATOR
ISOLATOR
4.7K
P3V3
4.7K
Slot2,PCIe x16
Slot1,PCIe x16
PCA9548A0xE4
I2C_4
I2C_5 PCA9617
4.7K
0ISOLATOR
PEX87470x70
CLK Buffer0xD8
P3V30
4.7K
Riser 3
ISOLATORI2C_6
I2C_7
P3V3 P1V8
4.7K 4.7K
P3V3_STBY P3V3_STBY
4.7K
0
0
0
0
Tem p sensor0xF8(Topside)
FRU0xAC
22 2222
Tem p sensor0x98(Backside)
P3V3_STBY
Voltag e Monitor0x90
4.7K
22
P3V3_STBY
NI4.7K
0I2C I2C_0
I2C_1I2C_2I2C_3
0
P3V3_STBY
ISOLATOR
ISOLATOR
4.7K
P3V3
4.7K
Slot2,PCIe x16
Slot1,PCIe x16
PCA9548A0xE4
I2C_4
I2C_5 PCA9617
4.7K
0ISOLATOR
PEX87470x70
CLK Buffer0xD8
P3V30
4.7K
Riser 5
ISOLATORI2C_6
I2C_7
P3V3 P1V8
4.7K 4.7K
P3V3_STBY P3V3_STBY
4.7K
0
0
0
0
Tem p sensor0xF8(Topside)
FRU0xAC
22 2222
Tem p sensor0x98(Backside)
P3V3_STBY
Voltag e Monitor0x90
4.7K
22
I2C_7 NA
Riser 1, PCIe x8
Riser 2, PCIe x8
I2C I2C_0I2C_1I2C_2I2C_3
0
P3V3_STBY
4.7K
ISOLATORISOLATOR
P3V3_SL OT2
4.7K
Slot2,PCIe x16Slot1,PCIe x16
PCA9548A0xE4
I2C_4I2C_5
Temp sensor(EMC1413)
0xF8
FRU(24C64)0xAC
0 0
I2C_6
I2C_74.7K
P3V3_STBY
P3V3_STBY
4.7K
0
P3V3_SL OT1
I2C
4.7K
Riser4(x32)
P3V3_STBY
NI4.7K
0
Notes:• M.2 Slots labeled #1-4 in MB Silkscreen.• M.2 I2C Addresses (VPD: 0xA6; Mgmt: 0x3A)
0
P1V8_P0
4.7K
0
M.2 S0M.2 S1
0
0
P1V8_P0
4.7K
M.2 S3M.2 S2
TCA9548A0xE2
* 0 based
I2CI2C_0
I2C_1
I2C_2
I2C_3
I2C_[7:4] NA
Slot 3.4Slot 3.2
Slot 4.2
Slot 4.1
Slot 5.4Slot 5.2
Physical Slot #’s
Physical Slot #’s
Physical Slot #’s
All PCIe Connectors connect to the BMC I2C for PCIe Card telemetry.
• Use the PCIe designated pins for I2C (B5 & B6).
• To avoid I2C Address contention between PCIe Cards, an I2C MUX is needed on each riser
• The design must add appropriate voltage translation/isolation between different I2C voltage domains.
PCIe Clock BlockDiagramDifferential Clock Buffers used on the Risers to
fanout additional PCIe Clocks.
The Clock Buffers are set to PLL Bypass Mode by default (shown to provide best margins)
Included resistor stuffing Options to change the PLLBandwidth Mode if needed to adjust Clock Jitter performance:
• PLL High BW Mode
• PLL Low BW Mode
• PLL Bypass Mode (default)
Power Distribution Board Layout
PDB supports single 12V Power Domain from 3 x Project Olympus PSUs
• Uses Passive Droop Sharing
• No V_Sense or I_Share signals used
Power Delivery Requirements
• Hot Swap Controller for Power Monitoring and Isolation
• 12V Current Sense Ckts used for each branch
• Supports Power to entire System (MB, Risers, PCIe Cards)
• Fan Power also comes from PDB (none from MB)
Provides passthrough of PMDU signals between MB and RM
6 Layer Stackup (0.093”), 2oz copper planes
Different power harnesses can be used for different High-Power PCIe Cards
Power Harness for 2 x AMD® MI25 GPUs
MB Signal
PSU1
PSU2
PSU3
PSU1 Signal
PSU2 Signal
PSU3 Signal
MB
PWR
LED Signal
Spare PWR
PCIe Cards
PCIe Cards
PCIe Cards
Riser5 PWR
Riser4 PWR(Hi PWR)
Riser4 PWR(2-Slot)
Riser3 PWR
Fan PWR
7.48”
4.49”
PDB Power DistributionBlock Diagram
Power Distribution Board MB
Hot Swap Controller
(ADM1278)
P12V
_PSU
_SW
PSU10xB0
P12V_PSU 2x12
Riser 3 (2-Slot)
2x6
Riser 5 (2-Slot)
2x6
Riser 4 (2-Slot x32)
2x2
Fan Power(6 x 60mm)
2x12
PSU20xB0 2x12
PSU30xB0 2x12
Rev 1.2
GPUTwo 2x4
GPUTwo 2x4
GPUTwo 2x4
GPUTwo 2x4
Note: Motherboard has its own Hot Swap Controller & Switch Ckt.
1U Fan Power& HDD
PWR Conn
PWR Conn
HSC/SW
2x4
1U Fan PowerConn Not Used
Sense
Sense
Sense
Sense
Hot Swap Controller
(ADM1172)
P12V
_STB
Y
Gen5 RiserP3V3_CB VR
P3V3_STBY VR
P3V3_RISER4SW
PDB Misc Signal Block Diagram
1U
Co
nn
PSU2(0xB0)
C2020 MB PDBBMC
PSU1(0xB0)
PSU1_PRESENT_N
PSU1_PS_ON_NPSU1_ALERT_N
BMC_NODE_ID_[5:0]
I2C_PSU1_SCL/SDA
SMBUS MUXPCA9546A
(0xE0)I2C_PSU2_SCL/SDA
PSU1_ALERT_N
PSU1_BLADE_EN_N
PMDU
PSU1_THROTTLEBLADE_EN#BLADE_THROTTLENODE_ID[5:0]
BLADE_PRESENT#
PMBUS
PSU_ALERT#PS_ON#
PSU1_THROTTLE_N
GPIOF4
GPIOF3
Throttle Logic
PSU1_BLADE_EN_N
BMC_NODE_ID_[5:0]GPIX[5:0]
GPIOJ0
CPLD 1K
MM2
SCL/SDA
SC0/SD0SC1/SD1
PMBUS
PSU_ALERT#PS_ON#
PSU2_PS_ON_NPSU2_ALERT_N
BLADE_EN#BLADE_THROTTLENODE_ID[5:0]
PSU2_PRESENT_NBLADE_PRESENT#
PSU3(0xB0)
I2C_PSU3_SCL/SDAPMBUS
PSU_ALERT#PS_ON#
PSU3_PS_ON_NPSU3_ALERT_N
BLADE_EN#BLADE_THROTTLENODE_ID[5:0]NC
PSU3_PRESENT_NBLADE_PRESENT#
Rev 1.4
I2C I/OExpanderPCA9555
(0x40) NC
IO00
SCL/SDA
IO01PSU2_BLADE_EN_N
PSU3_BLADE_EN_N
IO03IO12IO02IO13IO10IO11
PSU3_PS_ON_BUF_NPSU2_PS_ON_BUF_N
NC
NC
NI
Jum
per
FRU0xA6
TempSensor0xF8
NIPS
U L
ED C
able
PSU_LED0PSU_LED1
PSU_LED0PSU_LED1
PSU_LED0PSU_LED1
P3V3_STBY
PSU1_G_LED
PSU2_G_LED
PSU3_G_LED
PSU1_Y_LED
PSU2_Y_LED
PSU3_Y_LED
RESET#
PSU3_Y_LED
PSU3_G_LED
PSU2_Y_LEDPSU2_G_LED
PSU1_Y_LEDPSU1_G_LED
EN_HSC_IN
RST_I2C_MUX_N
2U
Co
nn
MM8MM3
CPLD
MB HSCSN1402039 (0x80)
PDB HSCADM1278 (0x26)
PSU2_PS_ON_NPL3C
HSC_P12V_FAULT_N
HSC_P12V_CSOUT Comparator Circuit
PDB_CURRENT_ALERT_N
SMB_BMC_MM2_SCL/SDA
SMB_BMC_MM8_SCL/SDASMB_BMC_MM3_SCL/SDA
PL5A
Note: Cabled LEDs mount directly to front panel of Chassis.
SC2/SD2SC3/SD3
Not UsedNot Used
IO14IO05
IO04
IO15IO06IO16
IO17IO07
RISER5_OC_CLEARRISER5_OC_ALERT_NFAN_OC_CLEARFAN_OC_ALERT_N
RISER4_OC_CLEARRISER4_OC_ALERT_N
RISER3_OC_CLEARRISER3_OC_ALERT_N
Sen
seSe
nse
Sen
seSe
nse
PSU3_BLADE_EN_R_NPSU2_BLADE_EN_R_N
NI
NI
PSU2_ALERT_NGPIOM5
OD
OD
OD
Provides passthrough of PMDU signals between MB and RM
3U Server Slide Rails
Supports Slide Rails for smoother travel during servicing
Due to the weight of C2310 Server (90lbs), standard Project Olympus T-Pin Slide Rails will not allow the Chassis to pull in and out smoothly.
Width of Chassis prevents thicker rail slides (only 67% pull-out)
Top cover is split into two sections (Front and Rear).
Front Top Cover section allows access to most of the System hardware, leaving only the System Fans, PDB and Power Cables under the Rear Top Cover without in-Rack access.
Using Slide Rails from King Slide®
Plans for OCP Recognition of Project Olympus 3U Server
3U Server Base Spec submitted by Microsoft® (Siamak Tavallaei) – Targeting OCP Acceptance by Mid 2019
3U AMD® EPYC™ Server Product Spec & Design Collateral submitted by ZT – Targeting OCP Acceptance by Q3 20193U Intel® Xeon®-SP Server Product Spec & Design Collateral submitted by ZT– Targeting OCP Acceptance by Q3 2019
Both Products to be Available on Open Compute Marketplace 2H 2019
Visit us at Booth A17Or www.ztsystems.com/ocp