steve smith hpc pre-sales consultant dell emc•vertical, horizontal, right angle, straddle mount...

© Copyright 2017 by Gen-Z. All rights reserved.Internal Use - Confidential

Dell Customer Communication

Steve Smith

HPC Pre-Sales Consultant

Dell EMC



DisclaimerThis document is provided ‘as is’ with no warranties whatsoever, including any warranty ofmerchantability, noninfringement, fitness for any particular purpose, or any warranty otherwise arisingout of any proposal, specification, or sample. Gen-Z Consortium disclaims all liability for infringement ofproprietary rights, relating to use of information in this document. No license, express or implied, byestoppel or otherwise, to any intellectual property rights is granted herein.

Gen-Z is a trademark or registered trademark of the Gen-Z Consortium.

All other product names are trademarks, registered trademarks, or servicemarks of their respective owners.

All material is subject to change at any time at the discretion of the Gen-Z Consortium

http://genzconsortium.org/

http://genzconsortium.org/



About Gen-Z• The Gen-Z Consortium launched in October of 2016 to create an open, industry

standard for a high speed, low latency, scalable, memory centric fabric

• There are currently over 50 member companies covering all of the disciplines required to create an ecosystem based on this open standard

• Consortium members have released four specifications• Core Specification 1.0 (released 2/13/18)• Scalable Connector Specification 1.0• SFF 8639 2.5-inch Specification 1.0• SFF 8639 2.5-inch Compact Specification 1.0

• Gen-Z showed a demo of multiple servers using multiple memory pools at SC’17

• Released and draft specifications are available on www.GenZConsortium.orgfor public review and comment

3



Compute-Memory Balance is Degrading

0

1

2

3

4

5

6

7

8

2012 2013 2014 2015 2016 2017 2018 2019

Normalized Properties of Typical Server Processors

Cores Pins DDR Channels PCIe Lanes

>4000 pins8 DDR Channels64+ PCIe Lanes

64+ Cores

0

1

2

3

4

5

6

7

2012 2013 2014 2015 2016 2017 2018 2019

Memory & I/O Bandwidth Capacity per Core (GB/s)

DRAM Bandwidth/Core PCIe Bandwidth/Core

Added DDR Channels gave a bump in 2017, but core count growth offsets 8 DDR channels

Processor memory and I/O technologies … … are being stretched to their limits

x

x

x

x

x

x

x

x



NV Media technology candidates - 1-5 year spanSTT-MRAM

3D XPoint?

NRAM (CNT)

Stack of Carbon Nanotubes

Mechanism: Opening and closing of paths through CNT stack with bias voltage

Latency > DRAMIncluded as reference Latency ~ DRAM

Latency ~ DRAMLatency > DRAM Latency > DRAM



Memory and Storage are Converging

With memory/storage convergence, memory semantic operations

become predominant (volatile & non-

volatile)TimeCurrent Time



What’s the Solution?• High Performance

▪ High Bandwidth, Low Latency, Scalable

▪ Eliminates protocol translation cost / complexity / latency

▪ Eliminates software complexity / overhead / latency

• Reliable ▪ No stranded resources or single-point-of-failures

▪ Transparently bypass path and component failure

▪ Enables highly-resilient data (e.g., RAID / erasure codes)

• Secure▪ Provides strong hardware-enforced isolation and security

• Flexible▪ Multiple topologies, component types, etc.

▪ Supports multiple use cases using simple to robust designs

▪ Thorough yet easily extensible architecture

• Compatible▪ Use existing physical layers, unmodified OS support

• Economic▪ Lowers CAPEX / OPEX, unlocks / accelerates innovation

Gen-Z speaks the language of compute



Gen-Z Allows Memory Innovation

Processor

DRAM

DRAM

DRAM

DRAM

Memory Bus

Media Module

4-8 Memory Channels

17-25 GB/s / Channel

288pins / DIMM

Synchronous Interface

Media

Contr

oller



Gen-Z Allows Memory Innovation

Processor

DRAM

DRAM

DRAM

DRAM

Media Module

Gen-Z

Logic

Gen-Z MemorySemantic Fabric

Media

Contr

oller

Gen-Z

Logic

2-8 High-speed Serial Links

Low Latency, High performance

Split Memory Controller

Asynchro nous Interface

Processor is media agnostic



Gen-Z is Media Agnostic & Composable

Media

Contr

oller

Gen-Z

Logic

Compute Module

Gen-Z

Logic

Media Module

Gen-Z Fabric

SCM

SCM

SCM

SCM

Storage Class Memory

Compute Module

Gen-Z

Logic

Compute Module

Gen-Z Fabric GPU

or

FPGA

Accelerator

Media

Contr

oller

Compute ModuleMedia Module

SCM

SCM

SCM

SCM

Compute Module

GPU

or

FPGA

Gen-Z

Sw

itch

Media

Contr

oller

DRAM

DRAM

DRAM

DRAM

Media Module

Compute Module

CPU

CPU

Media

Contr

oller

Media

Media

Media

Media

Media Module

Multiple resources enabled by Universal Interconnect



High Performance: Latency

Traditional StorageSoftware Stack

Gen-Z w/PersistentMemory Software Stack

Application

OperatingSystem

File System

I/O Buffers

Drivers

StorageSubsystem

Controller

Cache

HDD/SDDMedia

Media Controller

SCM Media

25,000 instructions

5 Data Copies

0 Data Copies!

1-3 instructions!

Message Application

Libfabric

Gen-Z Switch 30-50 ns

Gen-Z Bridge sub-100 ns

Message Application

Libfabric

Gen-Z Bridge sub-100 ns

0 Data Copies!

▪ Ultra-low-latency messaging (Sub-250 ns one-way latency)▪ Load/Store byte-addressable access to DRAM or SCM

▪ Sub-100 ns load-to-use latency (DRAM media)

▪ Reduced CPU utilization = More workloads per core per CPU



Faster transit of small, low-latency traffic(interleaved with data access packets)

High Performance: ThroughputTraditional Network: 2 Lane Road

Gen-Z Fabric: Multilane Super Highway

H/W breaks large data transfers into smaller parts

Smaller, low-latency packets caught behind large data packets

Dedicate paths/links/VCs for different traffic classes

Traditional Networks

• Modest traffic prioritization

• Modest multi-path support

• Challenging to mix bulk data and low latency

Gen-Z

+ Hardware-driven Multi-Link (e.g., 8 links per node)

+ Hardware-driven Multi-Path

+ 32 Virtual Channels (VCs) per link

+ 256 byte max payload size (90+% efficiency)

+ 30-50 ns switch latency

+ Adaptive / Dispersive routing

+ Dynamic congestion management

+ Universal protocol (no protocol translation)

= Industry leading performance



Flexible: Multiple Topologies and Deployment Models• Low cost, low power, copper direct-attached interconnects for local systems

• Optical, long range, switched interconnects for scale out and HPC

Single Motherboard

• Copper, low cost

• PCIe or IEEE 802.3 PHYs

• P2P, Daisy-chain, switched

Chassis/Enclosure

• Copper, low cost

• PCIe or IEEE 802.3 PHYs

• P2P, Mesh, Torus, switched

Rack-Scale

• Copper or optical

• IEEE 802.3 PHYs

• Switched, Torus, Spine/Leaf

Cluster-Scale

• Optical

• IEEE 802.3 PHYs

• Switched, Fat Tree, Butterfly, Hyper-X, etc.

• Mixed deployment models for enterprise enclosure and rack scale

• PCIe PHY up to 32 GT/s, IEEE 802.3 electrical 25 to 112 GT/s (PAM 4)



Flexible: Universal Connector System• Vertical, horizontal, right angle, straddle mount

• Same connectors for memory, I/O, storage, etc.

• Cabled solutions: for copper & optical

• Eliminates “hard choices” • Universal connector eliminates industry fragmentation

• Simplifies supply chain—drives volume and lowers cost

• Any component, any slot, any time

• Any mix of static and hot-plug form factors

• Multi-connector option to provide added scalability

• 80W incremental power

• Incremental bandwidth

• Supports internal and external cable applications

• Enables modular system design

• Enables system disaggregation

• Eliminates expensive board materials

• Multipath—can bifurcate connector into multiple links

• Aggregate bandwidth, resiliency, no stranded resources

• Support multiple topologies—point-to-point, daisy-chain, mesh

• Supports multiple interconnect technologies—Gen-Z, PCIe, etc. Gen-Z members contributed mechanical & electrical specification to SNIA—see SFF-TA-1002Gen-Z Scalable Connector specification (final version is publicly available) covers remaining functionality.

Gen-Z 4C100-448GB/s

Gen-Z 2C50-224GB/s

Gen-Z 1C25-112GB/s



What’s Next for Gen-Z?

• The Consortium workgroups are developing our Core, Mechanical, Phy, Software & Management, and Compliance Specifications

• The Gen-Z consortium and its members will continue efforts to create Gen-Z IP and silicon in 2018

• Gen-Z will be showing demos at FMS’18 and SC’18

• Initial Gen-Z enabled products will be available in late 2019 and early 2020

• Join Gen-Z and help lead the IT industry into a new paradigm for composability

16



Gen-Z Resources

• Find educational material, membership information and links to other Gen-Z related information at www.GenZConsortium.org

• Follow us on LinkedIn• https://www.linkedin.com/company/gen-z-consortium/

• Follow us on Twitter• https://twitter.com/GenZConsortium

• View videos on our YouTube channel• https://www.youtube.com/channel/UCHV2opNzubAJBbURcw59QzQ

17

https://www.linkedin.com/company/gen-z-consortium/

https://twitter.com/GenZConsortium

https://www.youtube.com/channel/UCHV2opNzubAJBbURcw59QzQ



18

A broad-based industry consortium

www. GenZConsortium.org



www.genzconsortium.org

Communication at the Speed of Memory

steve smith hpc pre-sales consultant dell emc•vertical, horizontal, right angle, straddle mount...

Documents