accelerating hpc with ethernet

© 2013 Extreme Networks Inc. All rights reserved.

Senior Director, Product Line Management

December 10, 2013

Derek Granath

Accelerating High Performance

Computing with Ethernet


Agenda

High Speed, Low Latency Ethernet for HPC

Architecture Examples

HPC Interconnect Challenges and Alternatives

Transforming Data into Information


How Much Data?

3

The World Generates 2.5 Quintillion bytes each day…

or 57.5 Billion 32Gb iPad’s worth

each day


Image: NCSA’s Blue Waters sustained petascale supercomputing facility

Growth of Data

90% of the data in the world was created in the past

TWO YEARS Source: IBM, 2012


Data Analytics Challenge

5

User Generated Content

Social Interactions & Feeds

Spatial & GPS Coordinates

External Demographics

Business Data Feeds

HD Video, Audio, Images

Speech to Text

Product/Service Log

SMS/MMS

BIG DATA Sensors/RFID/Devices

Mobile Web

User Click Stream

Sentiment

Affiliate Networks

Search Marketing

Behavioral Targeting

Dynamic Funnels

A/B Testing

Dynamic Pricing

Web Logs

Offer History

WEB

Source: Contents of above graphic created in partnership with Teradata, Inc. http://tinyurl.com/mt4ltah

Big Data = Transactions + Interactions + Observations

CRM Segmentation

Offer Details

Customer Touches

Support Contacts

ERP

Purchase Details

Purchase Record

Payment Record

Increasing Data Variety and Complexity

Petabytes

Terabytes

Gigabytes

Megabytes

Interactions

Transactions

Observations


Structured and Unstructured Data

Example

6

Structured Data

• Fits neatly into traditional

database schemas

• Email metadata

• Call records

• Can be easily

• Stored

• Queried

• Analyzed

Unstructured Data

• Everything else…

• May contain patterns

• Also it might not!

• Video

• Audio

• Photos

• This Presentation!

• Doesn’t fit in fixed length fields


Utilities Weather impact analysis on

power generation Transmission monitoring Smart grid management

Retail 360° View of the Customer

Click-stream analysis

Real-time promotions

Law Enforcement Real-time multimodal surveillance

Situational awareness and threat detection

Cyber security detection

Transportation Weather and traffic

impact on logistics and

fuel consumption

Traffic congestion

Financial Services Fraud detection

Risk management

High Frequency Trading

360° View of the Customer

IT System log analysis Cybersecurity

Telecommunications CDR processing

Churn prediction

Geomapping / marketing

Network monitoring

7

Health & Life Sciences DNA sequencing

Epidemic early warning

ICU monitoring

Remote healthcare monitoring

HPC – What Can You Do with it?

7


HPC Architectural Evolution

– Clusters Dominate

8


Moore’s Law For HPC

Doubles Every 1.5 Years

Doubles Every 4 Years

Doubles Every 1.5 Years

Compute Data I/O

9


Interconnect Challenges

10

• Flexibility to grow big cheaply

• Deploy and re-deploy assets

Massive Scalability

• More servers, more storage, more applications

Manageability

• Proven, certified interoperability

• Standards-based technology

Eased Convergence

• Energy

• Operational

Efficiency

• Resilient architectures

Availability


Network Considerations for Big Data

11

Processing Time

Traffic Burstiness

Barrier Synchronized Computation

Buffering & Burst Handling

Das, Anupam et. al, "Transparent and Flexible Network Management for Big Data Processing in the Cloud."

Data Volume

Large long-lived flows


Next-Generation HPC clusters demand:

Latency

• Least time taken to complete a job o Lowest port-to-port latency in the I/O fabric

o Jumbo-frame support

o Short-reach optics and cables support

Throughput

• Non-blocking performance for any-to-any connectivity o Switch fabric performance

o Non-oversubscribed, wire-speed architecture

o Cut-through support

Bandwidth

• Ample bandwidth for multiple applications and jobs o Higher speeds and feeds: 10/40/100 Gigabit Ethernet

o Higher density per slot

o Link aggregation (LAG) support

Top 3 Requirements for HPC/SC

Interconnect Fabric


I/O Technology to Meet the Future

Demand

To bridge the gap, you need an I/O technology, that is:

• Minimal cost increase with speed migrations

• Reusable in terms of infrastructure and training Economical

• Short, medium, long range connectivity options

• I/O diversity and mix-n-match Flexible

• Bandwidth aggregation to multiply I/O

• Seamless migration to higher speeds and feeds Scalable

• Is resilient and time tested

• Provides required level of service up time Reliable


HPC I/O Technology Alternatives

Infiniband Fiber Channel Ethernet

Scalability

Flexibility

Reliability

Bandwidth

Latency

Economics


Ethernet and InfiniBand Dominate

Note: Gigabit Ethernet Category includes 10GbE

15


Ethernet Penetration in Top 500

212 (42.4%) of world’s top 500 fastest supercomputers use Ethernet

Source: Top500.org, November 2013


RoCE – RDMA over Converged Ethernet

• RoCE is a link layer protocol between two hosts in a broadcast domain

• Allows Remote Direct Memory Access (RDMA) similar to Infiniband to run over Ethernet

• RoCE replaces IB link layer with Ethernet

• Simpler than iWARP (Internet Wide Area RDMA Protocol)

• RoCE is marginally slower than IB which is sub μs, and latencies can approach 1-3 μs (micro-seconds), but less expensive and lower power to deploy than IB

• Ideal for High Performance Cluster Computing environments already familiar with Ethernet Technology, but need the speed and agility of IB

• No Support for IP (unlike iWARP): Need to use a head-end gateway to access closed- cluster environments

• Multicast RDMA is defined for RoCE (also unlike iWARP)

• Requires IEEE Data Center Bridging for PFC support on the network, a RoCE capable Ethernet adapter for hardware acceleration

17


Access

High Performance Interconnect Fabric

40G

10G

Back End

Sample Architecture

Clients

Front End

Compute Nodes

Cluster

Ma

ste

r N

od

es

Storage

Compute Nodes

Cluster

40G


HPC Design with Centralized Storage

Master

Nodes Master

Nodes

Compute Nodes Compute Nodes Storage Nodes

High-Performance Fabric

10GbE Data Path

40GbE Storage Path

Ac

ce

ss

Front End Front End

Back End

Ac

ce

ss


Cluster Interconnect Fabric Options

20

Fully Non-blocking Architecture • Six 32 x 40G switches

• 256 10G ports, fully non-blocking

• 64 10G ports to each server

• 16 40G uplinks per rack to spine

• Less than 1.8 microsecond latency

3:1 Over-subscribed • Ten 32 x 40G switches

• 768 10G ports to servers

• 8 40G uplinks per rack to spine

• Less than 1.8 microsecond

latency


Ethernet evolves –

That’s what it does!

21

Thank You

accelerating hpc with ethernet

Technology

extreme networks

hpc computedoubles

ethernet penetration

hpc architecture examples

generation hpc clusters

low latency ethernet

gigabit ethernet category

alternatives high speed