Download - Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1
![Page 1: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/1.jpg)
Traffic shaping with OVS and SDN
Ramiro Voicu
Caltech
LHCOPN/LHCONE, Berkeley, June 2015
1
![Page 2: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/2.jpg)
SDN Controllers
• “Standard” SDN controller architecture– NB: RESTfull/JSON, NETCONF, proprietary, etc– SB: OF, SNMP, NETCONF,
NorthBound API
SDN Controller Core
SouthBound API
App1 App2 AppN
![Page 3: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/3.jpg)
Design considerations for SDN control plane
• Even at the site/cluster level the SDN control plane should support fault-tolerance and resilience– e.g. in case one or multiple controller instances fail the
entire local control plane must continue to operate
• Each NE must connect to at lest two controllers (redundancy)
• Scalability
![Page 4: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/4.jpg)
SDN Controllers
• NOX: – First SDN controller– C++– New protocols development– Open source by Nicira in 2008
• POX– Python-based version of NOX
• Ryu– Python– Integration with OpenStack– Clustering: No SPOF using Zookeeper
4
![Page 5: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/5.jpg)
ONOS: Open Network Operating System
• “A SDN network operating system for Service Providers and mission critical networks”
• Distributed core designed for High-Availability and performance (RAFT consensus algorithm)
• Developed in Java - a set of OSGi modules deployed in an OSGi container (Karaf) – similar to ODL
• Application Intents to NB which can be compiled on the fly in (Flow) Rules for SB– Intent: Request a service from the network without
knowing how the service will be performed5
![Page 6: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/6.jpg)
ONOS: Open Network Operating System
6
![Page 7: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/7.jpg)
OpenDaylight “Helium”
![Page 8: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/8.jpg)
OpenDaylight “Helium”
• Project under Linux Foundation umbrella• Well-established and very active community (very
likely the biggest)• Developed in Java - a set of OSGi modules deployed
in an OSGi container (Karaf) – similar to ONOS• Backed up by leading industry partners• Distributed clustering based on Raft consensus
protocol• OpenStack integration• “Helium” is the 2nd release of ODL and supports, apart
from SDN, NV (Network Virtualization) and NFV (Network Function Virtualization)
![Page 9: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/9.jpg)
OVS Open vSwitch“Open vSwitch is a production quality, multilayer virtual switch”
• OpenFlow protocol support (1.3)• Kernel and user space forwarding engines• Runs on all major Linux distributions used in HEP[*]• NIC bonding• Fine grained QoS support
• Ingress qdisc, HFSC, HTB• Used in a variety of hardware platforms (e.g. Pica8) and
software appliances like Mininet• Interoperates with OpenStack
• OVN (Open Virtual Network): Virtualized network “implemented on top of a tunnel-based (VXLAN, NVGRE, IPsec., etc) overlay network”
[*] For SL/CentOS/RH 6.x some patches had to be applied. Custom RPMs
![Page 10: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/10.jpg)
OVS Open vSwitch
Performance tests
• Compared the performance of hardware versus the OVS in two cases:
• Bridged network (the physical interface becomes a port in the OVS switch)
• Dynamic bandwidth adjustment
![Page 11: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/11.jpg)
Baseline performance tests (10Ge)• Two SanyBridge machines • 10Ge Mellanox cards (“back-to-back”)• Stock SLC 6.x kernel• Connected via Z9K
![Page 12: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/12.jpg)
• FDT nettest (memory to memory)
• 1 TCP Stream• 10Gbps
Baseline performance – hardware throughput (single stream)
![Page 13: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/13.jpg)
Baseline performance (single stream)
• FDT nettest (memory to memory)
• Line rate 10Gbps
• CPU utilization receiver • CPU utilization sender
95% idle 95% idle
1 TCP Stream
![Page 14: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/14.jpg)
• FDT nettest (memory to memory)• CPU Usage:
• Receiver: ~95% Idle• Sender: ~92% Idle (64 streams), 90% Idle (128
streams)• Similar results with 8, 16, 32, 64 and 128 TCP streams
Baseline performance (multiple stream)
• CPU utilization receiver • CPU utilization sender
95% idle 93% idle
64 TCP Streams
![Page 15: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/15.jpg)
Performance tests – OVS Setup• Same hardware• OVS 2.3.1 on stock SLC(6) kernel• Same eth interfaces added as OVS
interfaces• ovs-vsctl add port br0 eth5
![Page 16: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/16.jpg)
OVS performance
• FDT nettest (memory to memory)• Line rate 10Gbps• Slightly decreased performance for receiver
• CPU utilization receiver • CPU utilization sender
95% idle 90% idle
128 TCP Streams
![Page 17: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/17.jpg)
Conclusions:Baseline performance tests
• TCP tests with a range of multiple TCP streams varying from 1 up to 128
• Line rate in all the tests: 10Gbps
• CPU usage between 5% and 10% (normal scenario with <64 streams would be 95% CPU idle)
• OVS 2.3.1 with stock SL/CentOS/RH 6.x kernel
• OVS bridged interface achieved the same performance as the hardware (10Gbps)
• No CPU overhead for OVS in this scenario
![Page 18: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/18.jpg)
Ingress rate limit
• OVS ingress rate-limit • Adjust per interface
• Based on Linux kernel ”ingress qdisc”
OVS Dynamic bandwidth adjustmentIngress rate-limit
![Page 19: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/19.jpg)
OVS Dynamic bandwidth adjustmentIngress rate-limit
No policy10Gbps
10Mbps
100Mbps
1Gbps
2.5Gbps
5Gbps
7.5Gbps
9Gbps
10Gbpspolicy
![Page 20: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/20.jpg)
OVS Dynamic bandwidth adjustmentIngress rate-limit – Receiver CPU
1Gbps 2.5Gbps
5Gbps
7.5Gbps
9Gbps
10Gbpsw/ policy
No policy10Gbps
Almost the same CPU Usage as without ingress policy in place
![Page 21: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/21.jpg)
OVS Dynamic bandwidth adjustmentIngress rate-limit – Sender CPU
1Gbps 2.5Gbps
5Gbps
7.5Gbps
9Gbps
10Gbpsw/ policy
No policy10Gbps
Almost the same CPU Usage as without ingress policy in place
![Page 22: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/22.jpg)
OVS Dynamic bandwidth adjustment
• OVS egress rate-limit • Based on Linux kernel:
• HTB (Hierarchical Token Bucket)• HFSC (Hierarchical Fair-Service Curve)
Egress rate-limit
![Page 23: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/23.jpg)
OVS Dynamic bandwidth adjustment egress rate-limit
500Mbps
5Gbps
1Gbps
2.5Gbps
7Gbps
10GbpsNO policy
7.5Gbps
9Gbps
10Gbpspolicy
11Gbpspolicy
500Mbps
10GbpsNO policy
![Page 24: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/24.jpg)
5Gbps
2.5Gbps
10GbpsNO policy
7.5Gbps
10Gbpspolicy
11Gbpspolicy
10GbpsNO policy
4Gbps
Almost the same CPU Usage as without egress policy in place
OVS Dynamic bandwidth adjustment egress rate-limit
![Page 25: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/25.jpg)
OVS Dynamic bandwidth adjustment egress rate-limit– Sender CPU
5Gbps
2.5Gbps
10GbpsNO policy
7.5Gbps
10Gbpspolicy
11Gbpspolicy
10GbpsNO policy
4Gbps
Almost the same CPU Usage as without egress policy in place
![Page 26: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/26.jpg)
Conclusions:OVS Dynamic bandwidth adjustment
• Smooth egress traffic shaping up to 10Gbps, and up to 7Gbps for ingress
• Over long RTT the ingress traffic shaping may not perform well (needs more testing), especially above 7Gbps
• The CPU overhead is negligible when enforcing QoS
• More testing is needed: • Longer RTTs• 40Ge? (are there any storage nodes with 40Ge
yet)• Multiple QoS queues for egress• reliability over longer intervals
![Page 27: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/27.jpg)
All NEs controlled by a redundant SDN control plane
The entire SDN resides within the network layer
SDN-enabled site (1)
![Page 28: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/28.jpg)
OVS daemon runs on the end-host
May evolve in a site with SDN-controlled NEs
OVS is “ready” to be deployed today without any impact on the current operations
SDN-enabled site (2)
OVSdaemon
OVSdaemon
OVSdaemon
![Page 29: Traffic shaping with OVS and SDN Ramiro Voicu Caltech LHCOPN/LHCONE, Berkeley, June 2015 1](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d775503460f94a598c5/html5/thumbnails/29.jpg)
Possible OVS benefits
• The controller gets the “handle” all the way to the end-host
• Traffic shaping (egress) of outgoing flows may help performance in cases where upstream switch has smaller buffers
• A SDN controller may enforce QoS in non-OpenFlow clusters
• OVS 2.3.1 with stock SL/CentOS/RH 6.x kernel
• OVS bridged interface achieved the same performance as the hardware (10Gbps)
• No CPU overhead for OVS in this scenario