interconnect your future with mellanox infiniband
TRANSCRIPT
Paving the Road to Exascale
April 2016
Interconnect Your Future
© 2016 Mellanox Technologies 2
Co-Design Architecture to Enable Exascale Performance
CPU-Centric Co-Design
Limited to Main CPU Usage
Results in Performance Limitation
Creating Synergies
Enables Higher Performance and Scale
Software
In-CPU
ComputingIn-Network
Computing
In-Storage
Computing
© 2016 Mellanox Technologies 3
Breaking the Application Latency Wall
Today: Network device latencies are on the order of 100 nanoseconds
Challenge: Enabling the next order of magnitude improvement in application performance
Solution: Creating synergies between software and hardware – intelligent interconnect
Intelligent Interconnect Paves the Road to Exascale Performance
10 years ago
~10
microsecond
~100
microsecond
NetworkCommunication
Framework
Today
~10
microsecond
Communication
Framework
~0.1
microsecond
Network
~1
microsecond
Communication
Framework
Future
~0.05
microsecond
Co-Design
Network
© 2016 Mellanox Technologies 4
High-Performance Intelligent 100Gb/s Interconnect Solutions
Transceivers
Active Optical and Copper Cables
(10 / 25 / 40 / 50 / 56 / 100Gb/s)VCSELs, Silicon Photonics and Copper
36 EDR (100Gb/s) Ports, <90ns Latency
Throughput of 7.2Tb/s
7.02 Billion msg/sec (195M msg/sec/port)
100Gb/s Adapter, 0.7us latency
150 million messages per second
(10 / 25 / 40 / 50 / 56 / 100Gb/s)
32 100GbE Ports, 64 25/50GbE Ports
(10 / 25 / 40 / 50 / 100GbE)
Throughput of 6.4Tb/s
© 2016 Mellanox Technologies 5
InfiniBand Architecture
InfiniBand Router Answers The Needs of Next Generation Scalable Data Centers
Native InfiniBand Connectivity Between
Different InfiniBand Subnets (Each
Subnet can Include 40K nodes)
Isolation Between Different InfiniBand
Networks (Each Network can be
Managed Separately)
Native InfiniBand Connectivity Between
Different Network Topologies (Fat-Tree,
Torus, Dragonfly, etc.)
© 2016 Mellanox Technologies 6
Introducing: InfiniBand Router Solution
SB7780 Router 1U
Supports up to 6 Different Subnets
© 2016 Mellanox Technologies 7
NCAR-Wyoming Supercomputing Center (NWSC) – “Cheyenne”
Cheyenne supercomputer system
5.34-petaflop SGI ICE XA Cluster
Intel “Broadwell” processors
More than 4K compute nodes
Mellanox EDR InfiniBand interconnect
Mellanox Unified Fabric Manager
Partial 9D Enhanced Hypercube interconnect topology
DDN SFA14KX systems
20 petabytes of usable file system space
IBM GPFS (General Parallel File System)
© 2016 Mellanox Technologies 8
The Performance Advantage of EDR 100G InfiniBand (28-80%)
28%
© 2016 Mellanox Technologies 9
Interconnect Architecture Comparison
Offload versus Onload (Non-Offload)
© 2016 Mellanox Technologies 10
Offload versus Onload (Non-Offload)
Two interconnect architectures exist – Offload-based and Onload-based
Offload Architecture
• The Interconnect manages and executes all network operations
• The interconnect is capable of including application acceleration engines
• Offloads the CPU and therefore free CPU cycles to be used by the applications
• Development requires large R&D investment
• Higher data center ROI
Onload architecture
• A CPU-centric approach – everything must be executed on and by the CPU
• The CPU is responsible for all network functions, the interconnect only pushes the data into the wire
• Cannot support acceleration engines, no support for RDMA, and network transport is done by the CPU
• Onload the CPU and reduces the CPU cycles available for the applications
• Does not require R&D investments or interconnect expertise
© 2016 Mellanox Technologies 11
Claims Used to “Market” Onload Architecture – “Too Many Cores”
Claim: There are many CPU cores, and the applications cannot use them all, so one can
dedicate some cores to manage the interconnect operations
Reality: False claim
• CPU vendors increase the CPU core count due to applications requirements!
• In cases where applications require less core, data center owners can buy the needed core count
• CPUs with less core are dramatically cheaper! Why would one spend more $ if not needed?
Intel Haswell CPU
10-Cores
Intel Haswell CPU
12-Cores
Intel Haswell CPU
14-CoresConclusions
CPU cost: $1502 CPU cost: $2170 CPU cost: $3003CPU cores cost more
than the interconnect!
CPU cores are not
free!
12-Core to 10-Core
Difference: $668
14-Core to 12-Core
Difference: $833
Dual Socket Server
Difference: $1336
Dual Socket Server
Difference: $1666
© 2016 Mellanox Technologies 12
Application Performance Comparison – WEIN2K
WIEN2k is a Quantum
Mechanical Simulation
Higher is better
InfiniBand Delivers 35% Higher Performance With Only 10 Nodes
© 2016 Mellanox Technologies 13
Application Performance Comparison – WEIN2K
WIEN2k is a Quantum
Mechanical Simulation
Higher is better
FDR and EDR InfiniBand Deliver Higher Performance
© 2016 Mellanox Technologies 14
Application Performance Comparison – Quantum ESPRESSO
Higher is better
InfiniBand Delivers 42% Higher Performance With Only 10 Nodes
Quantum ESPRESSO is
an electronic structure
and materials modeling
Simulation
© 2016 Mellanox Technologies 15
Application Performance Comparison – LS-DYNA
InfiniBand Delivers 48-63% Higher Performance With Only 12 Nodes
Omni-Path Does Not Scale Beyond 10 Nodes
A structural and fluid analysis software, used for automotive, aerospace, manufacturing simulations and more
Higher is better Higher is better
© 2016 Mellanox Technologies 16
Maximizing Data Center Return on Investment (ROI)
No Loss of CPU Efficiency (Offloads)
Compute systems cost: $1M
InfiniBand: $0.15M
Total cost: $1.15M
CPU overhead: 0%
Compute worth (ROI, higher is better): $1M
For compute worth of $0.7M, total cost would be $0.8M!
20% lower cost for the same performance!
30% Loss of CPU Efficiency (no-offloads)
Compute systems cost: $1M
Omni-Path: $0
Total cost: $1M
CPU overhead: 30%
Compute worth (ROI, higher is better): $0.7M
App App App App App App
AppApp
App App
AppApp
CPU CPU CPU CPU
Why do users prefer to pay
for Mellanox interconnect
even if Omni-Path would
have been provided
free of charge?
© 2016 Mellanox Technologies 17
Technology Roadmap – One-Generation Lead over the Competition
2000 202020102005
“Roadrunner”Mellanox Connected
1st3rd
TOP500 2003Virginia Tech (Apple)
2015
Terascale Petascale Exascale
10G 20G 40G 56G 100G 200G 400G
TrueScale
No-Offload Network
40G (InfiniBand)
Same PathScale Technology
Under Intel Logo
Infinipath
No-Offload Network
20G (InfiniBand)
Omni-Path
No-Offload Network
100G (Proprietary)
Same PathScale Technology
Under QLogic Logo
© 2016 Mellanox Technologies 18
Omni-Path Performance Claims?
ISC’15 July 2015
160M messages per Second
Linley Group Conference
February 2016
108M messages per second
IDF China April 2016
79-105M messages per Second
© 2016 Mellanox Technologies 19
InfiniBand Delivers Highest Applications Performance
Performance Advantage of InfiniBand Increases with System Size
InfiniBand the Only Interconnect that Scales with System Size
InfiniBand Enables Higher System Performance with 50% of the Servers
Multiple Application Cases Demonstrate 35-63% Performance Advantage
Thank You