empowering edge software stack on arm with high ......containers enp9s0 eth1 eth0 flannel/calico...

© 2020 Arm Limited (or its affiliates)

Trevor Tao, Song ZhuMar/2020

Empowering Edge Software Stack on Arm with High Performance Container

Networking Technology

2 © 2020 Arm Limited (or its affiliates)

Agenda

• Introduction

• Arm Edge Software Stack Architecture

• High Performance CNIs(Calico, Cilium…) for Arm Edge Software Stack

• SRIOV and SmarNIC support for Container Networking

• Performance Tests

• Future Work(Provisional)

© 2019 Arm Limited

Introduction


Introduction• The purpose of edge computing and MEC is to bring real-time, high-bandwidth, low-latency access to

latency-dependent applications, distributed at the edge of the network. Arm CPUs are cost effective, consume low power and are customizable; making them a preferred choice for edge cloud service vendors.

• On the other side, a high performance, flexible and easy deployable container networking of edge software stack is the key to the success of using Arm platform on edge cloud since the widely used VNFs are deployed by container-based orchestration engine, such as Kubernetes, OpenShift or others.

https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/arm-and-telco-systems-partner-to-redefine-universal-cpe

https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/arm-and-telco-systems-partner-to-redefine-universal-cpe


Arm Edge Reference Stack

• Heterogeneous Architecture• VM, container, bare metal• Servers and customized Edge platforms• Virtualized NFs and Physical NFs• Accelerator interface

• Resource constraints• Kubernetes• SDN Controller for K8s

• HW Accelerations• Integrated accelerators• PCIe/CCIX attached accelerator (Smart NICs…)

5

Network Equipment Switch/GW

Edge Servers or Networking Edge Platform

Acceleration

Smart NICsFPGA/GPU

Integrated Accelerators

Controller

Flannel, Calico

Cilium

Lightweight App orchestration

Real Time Linux distribution

SR-IOV, DPDK

Infra Orchestration and Installer

Kubernetes

Containerized Compass

VPP,OVS

NetworkingSoftware

Apps Apps

Bare metal Container

Linux System Networking

OVN-K8s

Knative Kubeflow EdgeX

eBPF

CNCF Service Mesh

Docker


Arm Edge Stack Deployment Reference Cluster

Small deployment with Marvell Mcbin DBShot Boards

Arm server samples used in edge cloud

© 2019 Arm Limited

High Performance Container Networking for Arm Edge Software Stack


High Performance CNIs available for Arm Edge Stack

Things now available in Akraino IEC Arm edge stack as a ref:

8

IEC Arm Edge Stack

Calico Cilium Contiv-VPP OVN-K8s FlannelSRIOV

• pure IP networking fabric

• high-level network policy management by iptables

• Good scalability

• Support direct(non-overlay) and overlay(IPINIP, VxLAN) network connection

• Easy deployment

• Calico-VPP appears

• Linux-Native, API-Aware Networking and Security for Containers

• Linux eBPF based network policy, load balance and security which is believed to be with incredible performance

• L3 networking between hosts

• Good scalability too

• uses FD.io VPP to provide network connectivity between PODs

• Native DPDK interface support for phy NIC

• Native VPP ACL/NAT based network policy and access

• Good performance but with rather complex configuration

• Hard to debug

• OVS/OVN-controller based K8s networking solution

• Rather good performance with OVS inherited

• Use OVN logical switches/routers to connect Pods and for outside access

• No OVS-DPDK support now

• Direct physical interfaces(PF/VFs) support for Pods

• High performance with direct Linux kernel eth driver or DPDK PMD driver

• Usually co-work with other CNIs, such as Flannel, Calico by Multus or other glue CNI

• Need resource description or annotation when do the configuration for CNI and Pod setup

• Widely used and almost easiest deployment for a simple K8s networking

• Linux network bridge for pod connection and overlay based communication for inter-hosts access

• Easy to be integrated into other container networking solution, e.g., Cilium

• No good network policy support

Repo: https://gerrit.akraino.org/r/admin/repos/iec

https://wiki.akraino.org/display/AK/Integrated+Edge+Cloud+%28IEC%29+Blueprint+Family

https://fd.io/

https://gerrit.akraino.org/r/admin/repos/iec


Calico on Kubernetes

1. Pure IP networking

fabric

2. No encapsulation

needed when simple L2

connection available

for nodes

3. Easy deployment and

debug

4. Supporting Kubernetes

Network Policy by

iptables

5. Good scalability with

BGP based routing

K8s MASTER Node 10.0.0.2/24

Route Table

BIRD

Confd

Felix

Calico/node

Veth0

Pod A 192.168.1.1

Veth1

Route Item

K8s SLAVE Node 10.0.0.3/24

Route Table

BIRD

Confd

Felix

Calico/node

Kernel

Pod B 192.168.2.1

Veth1

Veth0

Kernel

Route Item

Route Table

192.168.1.1/32 Veth0

192.168.2.1/32 Eth0

10.0.0.0/24 Eth0

Route Table

192.168.2.1/32 Veth0

192.168.1.1/32 Eth0

10.0.0.0/24 Eth0

BGP route information

Src: 192.168.1.1Dest: 192.168.2.1

Src: 192.168.1.1Dest: 192.168.2.1

Src: 192.168.1.1Dest: 192.168.2.1


Cilium – API Aware Networking

Cilium: API Aware Netowrking & Network Security for Microservices using BPF & XDPCilium Architecture

BPF/XDP load balancing 10x performance over IPVS

API Aware(HTTP, Kafka, gRPC)

https://www.slideshare.net/ThomasGraf5/cilium-bringing-the-bpf-revolution-to-kubernetes-networking-and-security?from_action=save

https://docs.cilium.io/en/v1.3/concepts/

© 2019 Arm Limited

SRIOV and SmarNIC support for Container Networking


12

Kubernetes

CNI

Multus CNI Plugin

Flannel/Calico

Plugin

Flannel/Calico Linux Bridge

eth0

PF: enp8s0f0np0

VF0

SR-IOV CNI/Device Plugin

eth1

SR-IOV CNI/Device Plugin

eth1

VF1

SmartNIC

Pod

optional

Containers

enp9s0 enp9s1

enp9s0

eth1 eth0

Flannel/Calicobridge

eno2SmartNIC PS225

SR-IOVPlugin

Pod1

Pod2Kernel Driver

Kernel Driver

Uses only 1 of them

High Performance Networking with SmartNIC SRIOV Interfaces by SRIOV CNI data:

config.json: |{

"resourceList": [{"resourceName":

"ps225_sriov_netdevice","selectors": {

"vendors": ["14e4"],"devices": ["d800"],"drivers": ["bnxt_en"],"pfNames": ["enp8s0f0np0"]

}}, …

}]

}

Device Resource Description:

VFs need to be created

beforehand

For DPDK device driver(another Intel x710 nic as a sample):{

"resourceName": "intel_sriov_dpdk",

"selectors": {"vendors": ["8086"],"devices": ["154c"],"drivers": ["vfio-pci"],"pfNames": ["enp12s0f1"]

}}

apiVersion: v1kind: ConfigMapmetadata:name: sriovdp-confignamespace: kube-system


Podoptional

Containers

enp9s0

eth1 eth0


eno2enp9s1


eno2

VF0 VF1Host

Podoptional

Containers

eth1 eth0

enP8p1s0f2np0 VF0

SR-IOVPlugin

SR-IOVPlugin

IP:10.56.2.30/24Ro: 10.56.3.0/24GW: 10.56.2.1

10.56.3.30/24Ro: 10.56.2.0/24GW: 10.56.3.1

NIC

A72 CPUs

10.56.2.1/24 10.56.3.1/24

PF0 PF0

Sample: 2 Pods Connected byStingray PS225VFs in a single host

enP8p1s0f2np0 VF1

SmartNIC PS225

Networking Model


A Very Raw Thinking on the Possible Model for Container Networking with SmartNIC OVS offload support

14

Container

OVS-Bridge

veth_p

veth

Container

veth_p

veth

vSwitch Offload

PF/VF

NIC

NICNIC

OVS/OVN Control

Ovsdb-server

Ovs-vswitchd

Ovn-k8s-agent

KubernetesKubelet

Kube-ctrl-manager

OVN-KubeOvn-kube-controller

Ovn-k8s-agent-master

Container

vNIC

Container

vNIC

vSwitch Offload

NIC

NICNIC

OVS/OVN Control

Ovsdb-server

Ovs-vswitchd

Ovn-k8s-agent

OVN-KubeOvn-kube-controller

Ovn-k8s-agent-master

veth veth

User Space

Kernel

Model 1: Model 2:

User Space

Kernel

…

KubernetesKubelet

Kube-ctrl-manager

…

veth

OVS Flow

Offload

Partial OVS Flow

Offload

© 2019 Arm Limited

Performance Tests


13.3Gbps

15.8 Gbps

11.4 Gbps

0

2

4

6

8

10

12

14

16

18

Calico Pod-to-Pod Calico Node-to-Pod OVN-K8s Pod-to-Pod

Performance of Calico and OVN-Kubernetes(tested by iperf3)

In a single Arm server

Performance of Calico and OVN-Kubernetes


Pod

Containers

enp9s0

eth1 eth0


eno2

SmartNIC Stingray PS225

enp9s1


eno2VF0 VF1

Host

Pod

Containers

eth1 eth0

enP8p1s0f2np0 VF0

SR-IOVPlugin

SR-IOVPlugin

IP:10.56.217.13/24Ro: 10.56.217.0/24GW: 10.56.217.1

NIC

A72 CPUs

PF0 PF0

Perf Test1: 2 Pods Connection withStingray PS225 in the same host

enP8p1s0f2np0 VF1

IP:10.56.217.14/24Ro: 10.56.217.0/24GW: 10.56.217.1

IPerfclient

IPerfserver

Kernel Driver

Kernel Driver

Performance Test with SRIOV CNIContainer Networking by SRIOV Interface for Intra Host Communication


Pod

Containers

enp9s0

eth1 eth0


eno2

SmartNIC Stingray PS225

VF0Host1

Host2

enP8p1s0f2np0 VF0

SR-IOVPlugin

IP:10.56.217.13/24Ro: 10.56.217.0/24GW: 10.56.217.1

NIC

A72 CPUs

PF0

Perf Test2: 1 Pod Connected with another host via SmartNIC VF

IPerfclient

IPerfserver

Kernel Driver

Thunderx2-02

Thunderx2-04

enp137s0f0

PerformanceTest with SRIOV CNIContainer Networking by SRIOV Interface for Inter Host Communication


Performance Test Result

4.77

5.535.48 5.46

5.37

5.15

4.2

4.4

4.6

4.8

5

5.2

5.4

5.6

# of parallel threads

Initial Test Result(To be tuned)

1 2 3 4 5 6

Pod Communication Performance with SRIOV CNI in the same host via PS225 VFs

Gb

its/

sec

Note: Here we choose iperf instead of iperf3 due to its multi-threaded implementation


8.17

8.07

9.26

8.88

9.24 9.21

7.4

7.6

7.8

8

8.2

8.4

8.6

8.8

9

9.2

9.4

# of parallel threads

Axis Title

Perf Test 2 Result

1 2 3 4 5 6

Pod Communication Performance with SRIOV CNI in 2 hosts via PS225 VF(10GConnection)

Gb

its/

sec

Performance Test Result


Future Work(Provisional)

• Performance evaluation and tuning for Calico, Cilium CNI with Pod-2-Pod, Node-2-Pod cases on arm platform, mainly for cross-node. The possible tuning parameters include:➢ RPS/RFS

➢ Linux system netdev parameters

➢ RX/TX queue

➢ MTU

➢ eBPF/XDP

➢ Overlay/N-Overlay

• Service mesh integration with high performance CNIs, such as Cilium/Proxy

• Further DPDK incorporated container networking usage model and performance evaluation

• Further CI for high performance container networking solutions on arm with up-to-date processes

• More thinking and possible actual work on SmartNIC support for container networking with OVS offload

22

empowering edge software stack on arm with high ......containers enp9s0 eth1 eth0 flannel/calico...

Documents