controlling congestion in new storage architectures

PRESENTATION TITLE GOES HERE

Controlling Congestion in New Storage Architectures

September 15, 2015

Today’s Presenters

Chad Hintz Ethernet Storage Forum

Board Member Solutions Architect - Cisco

David L. Fair SNIA Ethernet Storage Forum Chair

Intel

SNIA Legal Notice

!   The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted.

!   Member companies and individual members may use this material in presentations and literature under the following conditions:

!   Any slide or slides used must be reproduced in their entirety without modification !   The SNIA must be acknowledged as the source of any material used in the body of any

document containing material from these presentations. !   This presentation is a project of the SNIA Education Committee. !   Neither the author nor the presenter is an attorney and nothing in this

presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney.

!   The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.

3

Terms

!   STP-Spanning Tree Protocol !   An older network protocol that ensures a loop-free topology for

any bridged Ethernet local area network.

!   Spine/Leaf, CLOS, Fat-Tree, Multi Rooted Tree !   Network based on routing with all links active

!   VXLAN !   Standards based layer 2 overlay scheme over a layer 3 network

4

Agenda

!  Data Center “Fabrics” Current State

!  Current Congestion Control Mechanisms !  CONGA’s Design !  Why is CONGA important for IP based

Storage and Congestion Control? !  Q&A

5

Storage over Ethernet Needs and Concerns

6

Minimize chance of dropped traffic

In order frame delivery

Minimal oversubscription

Lossless fabric for FCoE (no Drop)

Reliability Performance

Data Center Fabric: Current State

7

STP- Spanning Tree

Data Center “Fabric” Journey

Blocking

STP- Spanning Tree Multi-Chassis

Etherchannel



Etherchannel


Spine-Leaf

LAYER 3


Etherchannel


MAN/WAN

VXLAN /EVPN

LAYER 3 W/ LAYER 2 OVERLAY

Spine-Leaf

Multi-rooted Tree(Spine/Leaf) = Ideal DC Network

12

1000s of server ports

Ideal DC network:

Giant switch

Ø  No internal bottlenecks è predictable Ø  Simplifies BW management


13


Ideal DC network:

Giant switch


Can’t build it L


14


Ideal DC network:

Giant switch


Multi-rooted tree


Can’t build it L ≈


15


Ideal DC network:

Giant switch


Multi-rooted tree


Can’t build it L ≈Possible

bottlenecks


16


Ideal DC network:

Giant switch


Multi-rooted tree


Can’t build it L ≈Possible

bottlenecks Need precise load balancing

Leaf-Spine DC Fabric

17

Approximates ideal giant switch

H1 H2 H3 H4 H5 H6 H7 H8 H9 H1 H2 H3 H4 H5 H6 H7 H8 H9

Leaf switches

Spine switches

≈

Leaf-Spine DC Fabric

18

Approximates ideal giant switch

H1 H2 H3 H4 H5 H6 H7 H8 H9 H1 H2 H3 H4 H5 H6 H7 H8 H9

Leaf switches

Spine switches

≈!   How close is Leaf-Spine to ideal giant switch?

!   What impacts its performance? !   Link speeds, oversubscription, buffering

Today: Equal Cost Multipath (ECMP) Routing

Pick among equal-cost paths by an algorithm(hash) Ø Randomized load balancing Ø Preserves packet order (Optimal for TCP)

19

Problems: -  Hash collisions -  No idea of congestion -  Flows mapped to one

path (Loss of link issues)

Impact of Link Speed

20

Three non-oversubscribed topologies:

20×10Gbps Downlinks

20×10Gbps Uplinks


5×40Gbps Uplinks


2×100Gbps Uplinks

How does Link Speed affect ECMP

Higher speed links improve ECMP efficiency

21

20×10Gbps Uplinks

2×100Gbps Uplinks

11×10Gbps flows (55% load)

1 2

1 2 20

Prob of 100% throughput = 3.27%

Prob of 100% throughput = 99.95%

http://simula.stanford.edu/~alizade/papers/conga-sigcomm14.pdf

Better


22

0 2 4 6 8

10 12 14 16 18 20

30 40 50 60 70 80 FCT

(no

rmal

ized

to

opti

mal

)

Load (%)

Avg FCT: Large (10MB,∞) background flows

OQ-Switch

20x10Gbps

5x40Gbps

2x100Gbps

http://simula.stanford.edu/~alizade/papers/conga-sigcomm14.pdf

Better


23

0 2 4 6 8

10 12 14 16 18 20

30 40 50 60 70 80 FCT

(no

rmal

ized

to

opti

mal

)

Load (%)

Avg FCT: Large (10MB,∞) background flows

OQ-Switch

20x10Gbps

5x40Gbps

2x100Gbps

!   40/100Gbps fabric: ~ same Flow Completion Time as Giant Switch (OQ)

!   10Gbps fabric: FCT up 40% worse than OQ

Storage over Spine-Leaf

!   New scale out storage is looking to have initiators and targets spread over multiple leaf switches

!   Concerns !   Multiple hops !   Potential for increased latency !   Oversubscription !   TCP Incast !   Potential buffering issues

24

Incast Issue with IP Based Storage

25

Initiators (Senders)

Target (receiver)

Spread over many paths

Single point of convergence

Incast events are most severe at receiver (iSCSI, other IP based storage)

Summary

!   40/100Gbps fabric + ECMP ≈ Giant switch; some performance loss with 10Gbps fabric

!   Oversubscription(incast) in IP Storage networks are very common and have cascading effect on performance and throughput

26

Current Congestion Control Mechanisms Hop-By-Hop

27

IEEE 802.1Qaz

Enhanced Transmission Selection (ETS)

!   Required when consolidating I/O – It’s a QoS problem

!   Prevents a single traffic class of “hogging” all the bandwidth and starving other classes

!   When a given load doesn’t fully utilize its allocated bandwidth, it is available to other classes

!   Helps accommodate for classes of a“bursty” nature

Ethernet Wire

FCoE

50% 50%

IEEE 802.1Qbb Priority Flow Control

!   PFC enables Flow Control on a Per-Priority basis

!   Therefore, we have the ability to have lossless and lossy priorities at the same time on the same wire

!   Allows FCoE to operate over a lossless priority independent of other priorities

!   Other traffic assigned to other CoS will continue to transmit and rely on upper layer protocols for retransmission

!   Not only for FCoE traffic

29

Ethernet Wire

FCoE

Adding in Spine-Leaf

!   Use IEEE ETS to guarantee bandwidth for traffic types

!   Use IEEE PFC to create lossless traffic for FCoE

!   Use Ethernet infrastructure for all kinds of storage

!   Improve scalability for all application needs and maintain high, consistent performance for all traffic types, not just storage

30

Problems ETS/PFC do not solve

!   Does not take in consideration Layer 3 links and ECMP in spine-leaf topology !   Limited to hop-by-hop links !   PFC designed for lossless

traffic, not typical IP-based storage

!   ETS guarantees bandwidth, does not alleviate congestion

31

The network paradigm as we know it…

Control and Data Plane

!   Two Models: !   Distributed Control and Data Plane-Traditional !   Centralized (Controller Based/SDN)

33

Control and Data Plane resides within Physical Device

Traditional

Software defined networking (SDN) definition: The physical separation of the network control plane from

the forwarding plane, and where a control plane controls several devices.

What is SDN? per ONF definition

https://www.opennetworking.org/sdn-resources/sdn-definition

In other words…

In the SDN paradigm, not all processing happens inside the same device

CONGA’s Design

37

CONGA in 1 Slide

38

L0 L1 L2

1.  Leaf switches (top-of-rack) track congestion to other leaves on different paths in near real-time

2.  Send traffic on least congested path(s)

Fast feedback loops between leaf switches, directly in dataplane

CONGA in 1 Slide

39

L0 L1 L2

1.  Leaf switches (top-of-rack) track congestion to other leaves on different paths in near real-time

2.  Send traffic on least congested path(s)

Fast feedback loops between leaf switches, directly in dataplane

Could this work with centralized control plane?

!   If control plane is separate then feedback can be sent in dataplane but then has to be computed in central control point (controller)

!   Latency for this along with constant change in network makes this not a valid option

40

VS.

CONGA in Depth

CONGA operates over a standard DC overlay (VXLAN) Ø Already broadly supported (VXLAN) to virtualize the

physical network

41 H1 H2 H3 H4 H5 H6 H7

L0 L1 L2

H9 H8

H1èH9 L0èL2 H1èH9 L0èL2

VXLAN encap.

CONGA-In Depth (VXLAN)

42

VXLAN Frame Format

H9

CONGA In Depth: Leaf-to-Leaf Congestion

43 H1 H2 H3 H4 H5 H6 H7

L1

H8

1 2 3 0

Track path-wise congestion metrics (3 bits) between each pair of leaf switches

Conges4on-‐From-‐Leaf Table @L2

Src Leaf

Path 0 1 2

L0 L1

3

L0èL2 Path=2 CE=5

5

L0 L2

L0èL2 Path=2 CE=0

L0èL2 Path=2 CE=5

L0èL2 Path=2 CE=0

pkt.CE ç max(pkt.CE, link.util)

H9

CONGA In Depth: Leaf-to-Leaf Congestion

44 H1 H2 H3 H4 H5 H6 H7

L1

H8

1 2 3 0

Track path-wise congestion metrics (3 bits) between each pair of leaf switches

Conges4on-‐To-‐Leaf Table @L0

Dest Leaf

Path 0 1 2

L1 L2

3

5

L0 L2

L0èL2 Path=2 CE=0

L2èL0 FB-‐Path=2 FB-‐Metric=5

51 1 43 7 2

CONGA-In Depth: LB Decisions

Send each packet on least congested path

45 H1 H2 H3 H4 H5 H6 H7

L0 L1 L2

H9 H8


Dest Leaf

Path 0 1 2

L1 L2

3

551 1 43 7 2

1 2 3 0

L0 è L1: p* = 3 L0 è L2: p* = 0 or 1

http://groups.csail.mit.edu/netmit/wordpress/wp-content/themes/netmit/papers/texcp-

hotnets04.pdf

CONGA-In Depth: LB Decisions

Send each packet on least congested path

46 H1 H2 H3 H4 H5 H6 H7

L0 L1 L2

H9 H8


Dest Leaf

Path 0 1 2

L1 L2

3

551 1 43 7 2

1 2 3 0

L0 è L1: p* = 3 L0 è L2: p* = 0 or 1

flowlet [Kandula et al 2004]

http://groups.csail.mit.edu/netmit/wordpress/wp-content/themes/netmit/papers/texcp-

hotnets04.pdf

CONGA-In Depth: Flowlet Switching

H1 H2

TCP flow

•  State-of-the-art ECMP hashes flows (5-tuples) to path to prevent reordering

TCP packets. •  Flowlet switching* routes bursts of

packets from the same flow independently.

•  No packet re-ordering

Gap ≥ |d1 – d2|

d1 d2

*Flowlet Switching (Kandula et al ’04) http://groups.csail.mit.edu/netmit/wordpress/wp-content/

themes/netmit/papers/texcp-hotnets04.pdf

Of Elephants and Mice

!   Two types of Flows in the data center !   Long Live Flows-”Elephant”

!   Data or (block) storage migrations, VM Migrations, MapReduce !   Flows that impact buffers !   Not many in Data Centers, but just a few of these could be impactful

!   Short Lived Flows-”Mice” !   Web requests, emails, small data requests !   Can be bursty

!   How they interact is key !   In tradition ECMP, multiple long live flows can be mapped to few

links !   If mice are mapped to same links it is detrimental to the

application performance

48



!   Data or storage migrations, VM Migrations, MapReduce !   Flows that impact buffers !   Not many in Data Centers, but just a few of these could be impactful





49



!   Data or storage migrations, VM Migrations, MapReduce !   Flows that impact buffers !   Not many in Data Centers, but just a few of these could be impactful





50

Need a new Metric to determine this impact

Application Flow Completion Time (FCT)

CONGA: Fabric Load Balancing Dynamic Flow Prioritization

Real traffic is a mix of large (elephant) and small (mice) flows.

Key Idea: Fabric detects initial few flowlets of each flow and

assigns them to a high priority class.



F1

F2

F3

Standard (single priority): Large flows severely impact

performance (latency & loss). for small flows





F1

F2

F3

Standard (single priority): Large flows severely impact

performance (latency & loss). for small flows

High Priority

Dynamic Flow Prioritization: Fabric automatically gives a higher priority to small flows.

Standard Priority



Why Is CONGA important for IP based Storage?

55

Storage Flows are Elephant Flows

!   Using Flowlet switching we can break long lived storage flows(block) into flowlets(bursts) and route across multiple paths !   No Packet reorder-in order delivery

!   Send traffic on least congested path using CONGA feedback loop

!   Loss of link in path is a loss of flowlet –Minimal disruption !   No TCP reset (ISCSI, NFS, APPs)

!   Just send small burst that was lost

!   Object and File based short flows(mice) get higher priority and are able complete flows faster

56

CONGA for “Elephant” flows (block)

57

0

0.2

0.4

0.6

0.8

1

1.2

1.4

10 20 30 40 50 60 70 80 90

FCT

(N

orm

. to

EC

MP

)

Load (%)

ECMP

CONGA

Mice flows (<100KB) Elephant flows (>10MB)

0

0.2

0.4

0.6

0.8

1

1.2

10 20 30 40 50 60 70 80 90

FCT

(N

orm

. to

EC

MP

)

Load (%)

ECMP

CONGA

CONGA up-to 35% better than ECMP for

elephants

CONGA for “Mice” flows (file or object)

58

0

0.2

0.4

0.6

0.8

1

1.2

1.4

10 20 30 40 50 60 70 80 90

FCT

(N

orm

. to

EC

MP

)

Load (%)

ECMP

CONGA


0

0.2

0.4

0.6

0.8

1

1.2

10 20 30 40 50 60 70 80 90

FCT

(N

orm

. to

EC

MP

)

Load (%)

ECMP

CONGA

CONGA up-to 40% better for mice

Single Fabric for Storage(block,file or object) and Data

59

0

0.2

0.4

0.6

0.8

1

1.2

1.4

10 20 30 40 50 60 70 80 90

FCT

(N

orm

. to

EC

MP

)

Load (%)

ECMP

CONGA


0

0.2

0.4

0.6

0.8

1

1.2

10 20 30 40 50 60 70 80 90

FCT

(N

orm

. to

EC

MP

)

Load (%)

ECMP

CONGA

CONGA up-to 35% better than ECMP for

elephants

CONGA up-to 40% better for mice

Link Failures with Minimal Loss

60

0 2 4 6 8

10 12 14 16 18 20

10 20 30 40 50 60 70

FCT

(N

orm

. to

Opt

imal

)

Load (%)

ECMP

CONGA

Overall Average FCT

Summary

61

CONGA with DCB meets the needs of Storage over Ethernet

62

Minimize chance of Dropped traffic

In Order Frame Delivery

Minimal Oversubscription/

Incast Issues

Lossless Fabric for FCoE (no Drop)

CONGA with DCB meets the needs of Storage over Ethernet

63

Routing flowlets over least Congestion

across path-link loss has minimal impact

Minimize chance of Dropped traffic

In Order Frame Delivery

Flowlet Switching with CONGA guarantees in

order delivery

Minimal Oversubscription/

Incast Issues

40G fabric with Enhanced ECMP and Mice/Elephant Flow detection/ separation

Lossless Fabric for FCoE (no Drop)

CONGA and DCB (PFC,ETS) can be

implemented together

After This Webcast

!   This webcast will be posted to the SNIA Ethernet Storage Forum (ESF) website and available on-demand ! http://www.snia.org/forums/esf/knowledge/webcasts

!   A full Q&A from this webcast, including answers to questions we couldn't get to today, will be posted to the SNIA-ESF blog ! http://sniaesfblog.org/

!   Follow us on Twitter @ SNIAESF

64

PRESENTATION TITLE GOES HERE Q&A

PRESENTATION TITLE GOES HERE Thank You

controlling congestion in new storage architectures

Documents