jayaram mudigonda, hp labs praveen yalagandula, hp labs

© Copyright 2010 Hewlett-Packard Development Company, L.P. 1© Copyright 2010 Hewlett-Packard Development Company, L.P.

Jayaram Mudigonda, HP Labs Praveen Yalagandula, HP LabsMohammad Al-Fares, UCSD Jeff Mogul, HP Labs

SPAIN:High BW Data-Center Ethernet

with Unmodified Switches

© Copyright 2010 Hewlett-Packard Development Company, L.P. 2

Datacenter Fabric

Traditional DatacenterInternet

Internet-facing applications: E-Mail, Web Servers, etc.


DC TrendsInformatio

n Explosion

Application Consolidatio

nVirtualization

HPC Applications


Datacenter Fabric

DC TrendsInternet

M R

M

R

RR

RR

R

R

M

M

M

M

M

M

Shuffle phase of Map – Reduce

RM


Datacenter Fabric

DC TrendsInternet

M R

M

R

RR

RR

R

R

M

M

M

M

M

M

Shuffle phase of Map – Reduce

RM

High bisection

bandwidth


Datacenter Fabric

DC TrendsInternet

Flat Network


DC Fabric Goals

High bisection BWFlat network

Low-cost


Ethernet: a good choiceCommodity Inexpensive

Speeds: 10G is here 40G/100G soonFlat-addressingSelf-configuring


But wait…


Spanning Tree Protocol (STP)

makes Ethernet hard

to scale!


Spanning Tree Protocol (STP) Roo

tBandwidth bottleneck

Unused links


Proposal 1: High-port core switch

A common current approach


Expensive Core

SwitchHigh BW or

Multiple Links


Proposal 2: L3

IP SubnettingVL2 [SIGCOMM’09]


L3 routers

ExpensiveNo non-IP protocols

(FCoE)


Proposal 3: Modify switches (HW/SW)

TRILL [IETF]

SEATTLE [SIGCOMM’08]PortLand [SIGCOMM’09]

Not deployable today!


SPAINUnmodified L2 switchesMulti-pathingArbitrary topologies


SPAIN Approach

Multi-pathing via VLANs

+ End-host driver to

spread load


A C B D

Multi-pathing via VLANsDefault

VLAN


SPAINUnmodified L2 switchesMulti-pathing via VLANsArbitrary topologiesMinor End-host modifs

Low-costHigh-BWDC FabricToday!


OutlineIntroductionSPAIN Components Offline computation End-host driverEvaluationSummary


Offline Computation Steps: 1. Discover topology 2. Compute paths 3. Layout paths as VLANs


Discover topologySNMP Queries

SPAIN


Compute pathsGoal: leverage redundancy; improve reliabilityChallenges: large graphs; more pathsmore resources


Compute pathsOnly consider paths between edge-switchesModified Dijkstra’s; Prefer edge-disjoint

paths


VLAN LayoutSimple scheme: Each Path as

VLAN


But…

VLAN ID = 12 bits 4096 VLANs!

IEEE 802.1Q:


Simple scheme: Each Path as VLANScales to only few switches

VLAN Layout


Our approach: 1 VLAN for a set of paths

VLAN Layout


Challenge: Minimize VLANs

NP-Hard for arbitrary topologies


Heuristics: 1. Greedy path packing 2. Parallel graph-coloring

VLAN Layout


# VLANs = 4VLAN Layout


SPAIN End-host Driver

A B

SPAIN


SPAIN


A BTopology & VLANs


21


A B

Flow TableAB, 1 : REDAB, 2 : BLUE

Flow Table2

1


ChallengesLink & switch failuresPathological flooding

InteroperabilityHost mobilityLoad-balanceEnd-host state


Failures

A B

Flow TableAB : RED

Flow Table


Pathological Flooding

A B

Flow TableAB : RED

Flow TableBA : GREEN

Does not know the

location of B


Solution:

Chirping


Chirping

A B

Flow TableAB : RED


Does not know the

location of B

C

Knows the location of B


Chirping

A B

Flow TableAB : RED BLUE



Evaluation

SimulationsReal testbed


SimulationsTopologies:CiscoD

C Core switches

Aggregation modulesm = 2

Access switches per modulea = 2



C Fat-Tree [Al-fares et al. SIGCOMM’08]

#ports/switchp = 4


Simulations

2D HyperX

k=4

Topologies:CiscoD


HyperX [Ahn et al. SC’09]





B-Cube [Guo et al. SIGCOMM’09]

#ports/switch (p) = 2

Levels (l) = 2






Metrics: #VLANs Link-Coverage Reliability Throughput


Num. of VLANs

CiscoDC (8,8) Fat-Tree (48)HyperX (16)B-Cube (48,2)

146

2880

256

2048

#switches38

576

971

2048

#VLANs


ThroughputCiscoDC Fat-TreeHyperXB-Cube

2x24x10.5x 1.6x

Improvement over STP


OpenCirrus Experiments


1G

10GRACK SWITCH (RS)

CORE SWITCH (CS)

80 blades

OpenCirrus Testbed


1G

10GRACK SWITCH (RS)

CORE SWITCH (CS)

80 blades

CS

S1

S2

S3

OpenCirrus Testbed


CS

S1

S2

S3

OpenCirrus Testbed

10G links that we added


CS

S1

S2

S3

OpenCirrus Testbed

4 VLANs


Shuffle-like experimentEvery server to all other

servers500MB data transfer


CS

S1

S2

S3

Spanning Tree Protocol(STP)


Link utilization in each direction

100%

100%

0%

Time


CS

S1

S2

S3

Spanning Tree Protocol(STP)

Over loade

d

Unused


CS

S1

S2

S3

No bottle-necks

SPAIN


Completion times

STP SPAIN

832 s

431 s

~50% reduction


Aggregate Goodput (Gbps)

STP SPAIN

35.6

66.7

87% improvement


Aggregate Goodput (Gbps)

0% 20% 50% 70% 100%

35.6 37.0 44.756.0

66.7

% SPAIN hosts

Incremental Deployability


CS

S1

S2

S3

Single Shortest Path(SSP) SEATTLE/TRILL

All flows

on RED

All flows on

GREEN

All flows

on GRAY

SEATTLE/TRILL on unmodified switches with

SPAIN


Comparison with SSP

SSP SPAIN

62.3 66.7Goodput (Gbps)

SSP SPAIN

513 431

Completion Time(s)

16% better 7% better


SPAIN Take-awayUnmodified L2 switchesMulti-pathing via VLANsArbitrary topologiesMinor End-host modifs

Low-costHigh-BWDC FabricToday!

jayaram mudigonda, hp labs praveen yalagandula, hp labs

Documents

hp labsspain

hp confidential13proposal

hp confidential8but

hp confidential7ethernet

hp confidential15proposal

hp confidential11proposal

hp confidential20spainu

load copyright