congestion management for ethernet-based lossless ... · pedro javier garcia1, jesus...
TRANSCRIPT
Congestion Management for Ethernet-based Lossless
DataCenter Networks Pedro Javier Garcia1, Jesus Escudero-Sahuquillo1,
Francisco J. Quiles1 and Jose Duato2
1: University of Castilla-La Mancha (UCLM)2: Technical University València (UPV)
DCN: 1-19-0012-00-ICne NENDICA
Abstract
This paper describes congestion phenomena in lossless data center networks and its nega- tive consequences. It explores proposed solutions, analyzing their pros and cons to determine which are suited to the requirements of modern data centers. Conclusions identify important issues that should be addressed in the future.
Agenda
IntroductionCongestion Dynamics in DCNsReducing In-Network and Incast CongestionCombining Congestion Management MechanismsConclusions
Agenda
IntroductionCongestion Dynamics in DCNsReducing In-Network and Incast CongestionCombining Congestion Management MechanismsConclusions
IntroductionOn-Line Data Intensive (OLDI) Services [Congdon18]
• Require immediate answers to requests that are coming in at a high rate.
• End-user experience is highly dependent upon the system responsiveness.
• The network becomes a significant component of overall DC latency when congestion occurs in the network.
Worker Worker ... Worker
Aggregator Aggregator ...
Worker Worker ... Worker
Aggregator
Aggregator
Deadline = 10 ms
Deadline = 50 ms
Deadline = 250 msRequest
• Todays DCNs require a flexible fabric for carrying in a convergent way traffic from different types of applications, storage of control.• Latency is a concern: Fabric design for DCNs must
minimize or eliminate packet loss, provide high throughput and maintain low latency.• These goals are crucial for applications of OLDI,
Deep Learning, NVMe over Fabrics and the Cloudified Central Offices.• However, congestion threatens these applications.
IntroductionData-Center Networks (DCNs)
• HoL-blocking dramatically degrades the network performance (e.g. PFC has not enough granularity and there is no congested flow identification) [Garcia05].• Classical e2e congestion
control for lossless networks is difficult to tune, reacts slowly, and may introduce oscillations and instability [Escudero11].
HSstarts
HSends
HS = traffic injected to Hot Spot destination
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0 1e+06 2e+06 3e+06 4e+06 5e+06Netw
ork
Thro
ughp
ut (
norm
aliz
ed)
Time (nanoseconds)
1QITh
VOQnet
64-node CLOS network, 4 hot-spots
IntroductionWhy congestion isolation is needed?
33%
33%
66%
33%
33%
33%
33%
66%
100%
Sw. 1
Sw. 2
Sw. 3
Sw. 4
Sw. 7
Sw. 6
Sw. 5
Sw. 8
Src. A
Sw. 933%
Src. B
Src. C
Src. D
Src. E
Dst. X
Dst. Y
Dst. Z33%
33%
33%
33%
33%
33 % Sending33 % Stopped
33 % Sending
Low-Order HoL-blocking
33%
33 % Sending33 % Stopped
33 % Sending
High-Order HoL-blocking
Congested flows (Dst. X)
Non-congested flows (Dst. Z)
Non-congested flows (Dst. Y)
IntroductionWhy congestion isolation is needed?
• We need a congestion isolation (CI) mechanism that reacts quickly when transient congestion situations appear, preventing network performance degradation caused by the HoL blocking.• We want a CI mechanism that complements other
technologies available in the DCNs, so that CI improves their performance, while the others reduce the CI complexity.
IntroductionWhy congestion isolation is needed?
Agenda
IntroductionCongestion Dynamics in DCNsReducing In-Network and Incast CongestionCombining Congestion Management MechanismsConclusions
Congestion
Inje
ctio
n ra
te a
t 100
% o
fth
e lin
k ba
ndw
idth
(ful
l rat
e)
Congestion
Inje
ctio
n ra
te a
t 100
% o
fth
e lin
k ba
ndw
idth
(ful
l rat
e)
Congestion (t0+T)
Inje
ctio
n ra
te a
t 100
% o
fth
e lin
k ba
ndw
idth
(ful
l rat
e)
Congestion (t0)Congestion (t0)
Inje
ctio
n ra
te a
t 100
% o
fth
e lin
k ba
ndw
idth
(ful
l rat
e)
Congestion (t0+T)
Congestion Dynamics in DCNsAppearance of Congestion
Speedup = 1 Speedup = 2
Speedup = 2 Speedup = 1.5
Congestion Dynamics in DCNsGrowth of Congestion Trees (from root to leaves)
Switch 1
Switch 3
Switch 5
Switch 2
Switch 4
Switch speedup = 1.5Packet flowsCongestion point
Switch 4
Switch speedup = 1.5Packet flowsCongestion point
Switch 3
Switch 2
Switch 1
Switch 5
Switch 6
Switch 7
Congestion Dynamics in DCNsGrowth of Congestion Trees (from leaves to root)
Congestion Dynamics in DCNsGrowth of Congestion Trees (Roots movement)
Switch 2
Switch 1
Switch 3
Switch 2
Switch 1
Switch 3
Switch speedup = 1.5Packet flows (start)Packet flows (after)Congestion point
Switch 4
Switch 3
Switch 2
Switch 1
Switch 5
Switch 6
Switch 7 Switch 8Y
X
Switch speedup = 1.5Packet flows addressed to XPacket flows addressed to YCongestion point
Congestion Dynamics in DCNsGrowth of Congestion Trees (in-network roots)
Switch speedup = 1.5Packet flows addressed to XPacket flows addressed to YCongestion point
Switch 1
Switch 2
Switch 3
Switch 4
Switch 5
Switch 6
Switch 7
Switch 8
Switch 9
X
Y
Congestion Dynamics in DCNsGrowth of Congestion Trees (Overlapping)
Switch 2
Switch 1
Switch 3
Switch speedup = 1.5Permanent packet flowsPacket flows disappearing firstCongestion point first appeared in the switch
Switch 2
Switch 1
Switch 3
Congestion Dynamics in DCNsGrowth of Congestion Trees (Vanishing)
Agenda
IntroductionCongestion Dynamics in DCNsReducing In-Network and Incast CongestionCombining Congestion Management MechanismsConclusions
Reducing CongestionIncast congestion reduction - ECMP
Switch 1
Switch 2
Switch 3
Switch 4
Switch 5
Switch 6
Switch 7
Switch 8
Switch 9
X
Y
Switch speedup = 1.5Packet flows addressed to XPacket flows addressed to YVictim flowCongestion point
Reducing CongestionIn-network congestion reduction - ECN
• These technologies may work together to eliminate loss in the cloud data center network.• Load-balancing and destination scheduling are end-to-
end solutions incurring in the RTT delays when congestion appear.• However, there is no time for loss in the network due to
congestion and congestion trees grow very quickly.• Transient congestion may still produce HoL blocking
that leads to increase latency, lower throughput and buffers overflow, significantly degrading performance.• Even using these mechanisms, we still need something
to deal with HOL Blocking locally and fast.
Reducing CongestionLimitations of current technologies [Escudero19]
Agenda
IntroductionCongestion Dynamics in DCNsReducing In-Network and Incast CongestionCombining Congestion Management MechanismsConclusions
Combining Congestion Management Mechanisms• CI is needed to react locally and very fast to
immediately eliminate HoL blocking.• Previous technologies reduce the use of PFC and
ECN, but their closed- and open-loop approach cause delays still happening.• Congestion trees appear suddenly, are difficult to
predict (even worse when load balancing is applied) and grow quickly.• New techniques can be applied in combination to
the previous technologies, improving their behavior.
Switch A
P1
P2
P3P3
Switch B
P2
P1
P4
CFQ
nCFQCongestion
Root
CIP
CFQ
LegendOutput port requested by thepacket on top.Congestion root.Congestion Isolation Packets (CIP).Packets from congested flows.Packets from non-congested flows.
CFQ
nCFQ
CFQ
nCFQ
nCFQ
CFQ
nCFQ
CFQ
nCFQ
P4
Combining Congestion Management MechanismsDynamic Virtual Lanes (DVL)
Agenda
IntroductionCongestion Dynamics in DCNsReducing In-Network and Incast CongestionCombining Congestion Management MechanismsConclusions
References
[Duato03] J. Duato, S. Yalamanchili, and L. M. Ni, Interconnection Networks: AnEngineering Approach. San Francisco, CA, USA: Morgan Kaufmann Publishers, 2003.[Garcia05] P. J. Garcia, J. Flich, J. Duato, I. Johnson, F. J. Quiles, and F. Naven, “Dynamic Evolution of Congestion Trees: Analysis and Impact on Switch Architecture,” in High Performance Embedded Architectures and Compilers, ser. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, Nov. 2005, pp. 266–285.[Congdon18] Paul Congdon, “IEEE 802 Nendica Report: The Lossless Network for Data Centers”, IEEE-SA Industry Connections White Paper, August 2018.
[Leiserson85] C. E. Leiserson, “Fat-trees: Universal networks for hardware-efficientsupercomputing,” IEEE Transactions on Computers, vol. C-34, pp. 892– 901, Oct 1985.[Escudero11] Jesús Escudero-Sahuquillo, Ernst Gunnar Gran, Pedro Javier García, JoseFlich, Tor Skeie, Olav Lysne, Francisco J. Quiles, José Duato: Combining Congested-FlowIsolation and Injection Throttling in HPC Interconnection Networks. ICPP 2011: 662-672[Escudero19] Jesús Escudero-Sahuquillo, Pedro Javier García, Francisco J. Quiles, José Duato: P802.1Qcz interworking with other data center technologies. IEEE 802.1 PlenaryMeeting, San Diego, CA, USA July 8, 2018(cz-escudero-sahuquillo-ci-internetworking-0718-v1.pdf)