using agilio smartnics for openstack networking acceleration
Post on 07-Jan-2017
207 Views
Preview:
TRANSCRIPT
© 2016 NETRONOME
Johann Tönsing August 9, 2016
Using Agilio™ SmartNICs for OpenStack Networking Acceleration
© 2016 NETRONOME 2
Agenda
• Overview of Traditional OpenStack Networking Options OVS (with/without connection tracking), Contrail vRouter, SR-IOV, VirtIO… • Issues: Flexibility, Performance, CPU Utilization
• Accelerating and Offloading OpenStack Networking using Agilio™ SmartNICs • Results: Features, Throughput, CPU Utilization • Implementation Details: Hardware and Software Architectures
• Evolution of OpenStack Networking • Supporting P4, C and eBPF Programmability • Integration Activities
• Conclusions
© 2016 NETRONOME
Traditional OpenStack Networking Options
3
Forwarding /Virtual Switching Technology
OpenStack Control Plane
Forwarding Overlays (Tunnels) Security Groups (Microsegmentation, Stateful Firewalling)
SR-IOV with regular NIC
Vendor specific plugin Limited E.g. MAC/VLANbased directing
None None
© 2016 NETRONOME
Traditional OpenStack Networking Options
3
Forwarding /Virtual Switching Technology
OpenStack Control Plane
Forwarding Overlays (Tunnels) Security Groups (Microsegmentation, Stateful Firewalling)
SR-IOV with regular NIC
Vendor specific plugin Limited E.g. MAC/VLANbased directing
None None
OVS
ML2 plugin (optional: SDN controller) OVN plugin
L2 / L3 VXLAN, GRE, (for OVN) GENEVE…Emerging: NSH (for NFV)
OVS 2.5+ can interface to Linux conntrack(Previously Linux bridge with iptables/nftables)
© 2016 NETRONOME
Traditional OpenStack Networking Options
3
Forwarding /Virtual Switching Technology
OpenStack Control Plane
Forwarding Overlays (Tunnels) Security Groups (Microsegmentation, Stateful Firewalling)
SR-IOV with regular NIC
Vendor specific plugin Limited E.g. MAC/VLANbased directing
None None
OVS
ML2 plugin (optional: SDN controller) OVN plugin
L2 / L3 VXLAN, GRE, (for OVN) GENEVE…Emerging: NSH (for NFV)
OVS 2.5+ can interface to Linux conntrack(Previously Linux bridge with iptables/nftables)
Contrail vRouterContrail plugin (via Contrail controller)
L2 / L3 MPLS in UDP, MPLS in GRE,VXLAN…
Built in, via flow table
© 2016 NETRONOME
Traditional OpenStack Networking Options
3
Forwarding /Virtual Switching Technology
OpenStack Control Plane
Forwarding Overlays (Tunnels) Security Groups (Microsegmentation, Stateful Firewalling)
SR-IOV with regular NIC
Vendor specific plugin Limited E.g. MAC/VLANbased directing
None None
OVS
ML2 plugin (optional: SDN controller) OVN plugin
L2 / L3 VXLAN, GRE, (for OVN) GENEVE…Emerging: NSH (for NFV)
OVS 2.5+ can interface to Linux conntrack(Previously Linux bridge with iptables/nftables)
Contrail vRouterContrail plugin (via Contrail controller)
L2 / L3 MPLS in UDP, MPLS in GRE,VXLAN…
Built in, via flow table
Additional features: load balancing, NAT etc.
© 2016 NETRONOME
OpenStack Networking Options Evaluated
4
Forwarding /Virtual Switching Technology
Traditional Approach Agilio™ SmartNIC Accelerated Approach
SR-IOV
Limited expressiveness to direct traffic to VMs (no support for general match/action rules, tunnel termination, stateful firewalling) High throughputNo VM migration support
Full OVS or vRouter virtual switching incl. tunnel termination, stateless/stateful firewalling
and SR-IOV based data delivery to VMs High throughputVirtIO integration, supporting VM migration
OVSandContrail vRouter
High expressiveness - match/action, tunnels, stateless/stateful firewalling etc. Limited throughput High CPU utilization (e.g. 50% of cores)
Same expressiveness - match/action, tunnels, stateless/stateful firewalling etc.
Higher throughput (~5x higher) Lower CPU utilization (~10x lower)
© 2016 NETRONOME 5
Agilio™ CX SmartNIC Family
• Optimized for standard server based cloud data centers • Low Profile Half Length PCIe form factor, power < 25W • Based on Netronome’s NFP-4xxx silicon (72 C programmable cores, 8 threads each) • 2GB DRAM for lookup tables / state tables (millions of entries) • Dataplane fully implemented in software
1x 40GbE 2x 40GbE2x 10GbE New: 2x 25GbE
Also available: Agilio™ LX 2x40G / 1x100G with dual PCIe interfaces, 120 cores, 8GB DRAM…
© 2016 NETRONOME 6
OVS Throughput vs. Packet Size
OVS L2 Forward to VMs
Packet Size
Mill
ions
of P
acke
ts p
er S
econ
d
OVS VXLAN + L2 Forward to VMs
Packet Size
Mill
ions
of P
acke
ts p
er S
econ
d
© 2016 NETRONOME 7
OVS Throughput vs. Number of Rules
5
10
15
20
25
30
OVS in Kernel Space
OVS in User Space on DPDK
100 Wildcard Rules
1000 Wildcard Rules
10000 Wildcard Rules
64000 Wildcard Rules
Mill
ions
of P
acke
ts p
er S
econ
d
12 CPU Cores
12 CPU Cores
OVS Offloaded to Agilio™ CX-4000
1 CPU Core
5X Throughput Improvement + 90% CPU Savings
OVS L2/L3 Forwarding to 8 VMs with 64K Flows
© 2016 NETRONOME 8
Tested Scenario: Server CPU Core Allocation
Unaccelerated OVS (Kernel / User Mode)
Agilio™ OVSOVS
© 2016 NETRONOME 9
Efficiency: Throughput Per Server CPU Core
Throughput with single server CPU corededicated to network related processing
Mill
ions
of P
acke
ts p
er S
econ
d • 50x Efficiency vs. Kernel OVS
• 20x Efficiency vs. User Mode (DPDK) OVS=> Replace 3-6 racks with 1 rack!
Throughput / efficiency similar for Agilio™ Contrail vRouter
Benefit for your use case: search for
“netronome.com roi calculator”
© 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API
1 Configuration via controller, CLI, or Callable API
(Nova, Neutron)
Execute Action
OVSKernel DP Match/Act
OVSKernel DP Match/Act
© 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API
1 Configuration via controller, CLI, or Callable API
(Nova, Neutron)
Execute Action
Open vSwitch Datapath
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
OVSKernel DP Match/Act
OVSKernel DP Match/Act
© 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API
1 Configuration via controller, CLI, or Callable API
(Nova, Neutron)
Execute Action
Open vSwitch Datapath
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
OVSKernel DP Match/Act
OVSKernel DP Match/Act
Miss
© 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API
1 Configuration via controller, CLI, or Callable API
(Nova, Neutron)
Execute Action
Open vSwitch Datapath
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
OVSKernel DP Match/Act
OVSKernel DP Match/Act
Miss
Miss
© 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute Action
Open vSwitch Datapath
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
OVSKernel DP Match/Act
OVSKernel DP Match/Act
Miss
Miss
© 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute Action
Open vSwitch Datapath
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
3 Offload datapath: copy match tables, sync stats
3
OVSKernel DP Match/Act
OVSKernel DP Match/Act
Miss
Miss
© 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute Action
Open vSwitch Datapath
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
3 Offload datapath: copy match tables, sync stats
3
4 Flow tracking: per-microflow state learning
4
Self Learning Exact MatchFlow Tracker
Miss
Hit
OVSKernel DP Match/Act
OVSKernel DP Match/Act
Miss
Miss
© 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute Action
Open vSwitch Datapath
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
3 Offload datapath: copy match tables, sync stats
3
Conn track
FTPSIP
4 Flow tracking: per-microflow state learning
4
Self Learning Exact MatchFlow Tracker
Miss
Hit
OVSKernel DP Match/Act
OVSKernel DP Match/Act
Miss
Miss
© 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute Action
Open vSwitch Datapath
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
Conn track
3 Offload datapath: copy match tables, sync stats
3
Conn track
FTPSIP
4 Flow tracking: per-microflow state learning
4
Self Learning Exact MatchFlow Tracker
Miss
Hit
OVSKernel DP Match/Act
OVSKernel DP Match/Act
Miss
Miss
© 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute Action
Open vSwitch Datapath
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
Conn track
3 Offload datapath: copy match tables, sync stats
3
5 Offload connection tracking: synchronize state
5
Conn track
FTPSIP
4 Flow tracking: per-microflow state learning
4
Self Learning Exact MatchFlow Tracker
Miss
Hit
OVSKernel DP Match/Act
OVSKernel DP Match/Act
Miss
Miss
© 2016 NETRONOME
Agilio™ vRouter Acceleration
11
vRouter Subsystem
vRouter Agent
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter CLI
1 Configuration via controller or CLI: BGP over XMPP
(Nova, Neutron)
Execute Action
vRouter Flow
Classify
vRouter ForwardingvRouter
Forwarding
© 2016 NETRONOME
vRouter Datapath
vRouter Flow
Classify
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
vRouter ForwardingvRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter Subsystem
vRouter Agent
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter CLI
1 Configuration via controller or CLI: BGP over XMPP
(Nova, Neutron)
Execute Action
vRouter Flow
Classify
vRouter ForwardingvRouter
Forwarding
© 2016 NETRONOME
vRouter Datapath
vRouter Flow
Classify
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
vRouter ForwardingvRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter Subsystem
vRouter Agent
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter CLI
1 Configuration via controller or CLI: BGP over XMPP
(Nova, Neutron)
Execute Action
vRouter Flow
Classify
vRouter ForwardingvRouter
Forwarding
2
2 Offload forwarding: pre-emptively copied
© 2016 NETRONOME
vRouter Datapath
vRouter Flow
Classify
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
vRouter ForwardingvRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter Subsystem
vRouter Agent
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter CLI
1 Configuration via controller or CLI: BGP over XMPP
(Nova, Neutron)
Execute Action
vRouter Flow
Classify
Miss
vRouter ForwardingvRouter
Forwarding
2
2 Offload forwarding: pre-emptively copied
© 2016 NETRONOME
vRouter Datapath
vRouter Flow
Classify
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
vRouter ForwardingvRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter Subsystem
vRouter Agent
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter CLI
1 Configuration via controller or CLI: BGP over XMPP
(Nova, Neutron)
Execute Action
vRouter Flow
Classify
Miss
Miss
vRouter ForwardingvRouter
Forwarding
2
2 Offload forwarding: pre-emptively copied
© 2016 NETRONOME
vRouter Datapath
vRouter Flow
Classify
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
vRouter ForwardingvRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter Subsystem
vRouter Agent
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter CLI
1 Configuration via controller or CLI: BGP over XMPP
3
3 Userspace agent populates kernel tables
(Nova, Neutron)
Execute Action
vRouter Flow
Classify
Miss
Miss
vRouter ForwardingvRouter
Forwarding
2
2 Offload forwarding: pre-emptively copied
© 2016 NETRONOME
vRouter Datapath
vRouter Flow
Classify
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
vRouter ForwardingvRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter Subsystem
vRouter Agent
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter CLI
1 Configuration via controller or CLI: BGP over XMPP
3
3 Userspace agent populates kernel tables
(Nova, Neutron)
Execute Action
4 Offload flows: copy tables, sync stats
4
vRouter Flow
Classify
Miss
Miss
vRouter ForwardingvRouter
Forwarding
2
2 Offload forwarding: pre-emptively copied
© 2016 NETRONOME
vRouter Datapath
vRouter Flow
Classify
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
vRouter ForwardingvRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter Subsystem
vRouter Agent
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter CLI
1 Configuration via controller or CLI: BGP over XMPP
3
3 Userspace agent populates kernel tables
(Nova, Neutron)
Execute Action
4 Offload flows: copy tables, sync stats
4
vRouter Flow
Classify
Miss
Miss
vRouter ForwardingvRouter
Forwarding
Notes: - Control protocol: BGP over XMPP - Tunnel protocols: VXLAN, MPLS over UDP / GRE - Distinct policy (flows - ACLs) and forwarding (IP/MPLS) - Flow table is used to implement stateful firewalling
2
2 Offload forwarding: pre-emptively copied
© 2016 NETRONOME
SmartNIC Firmware: Pre-programmed or Custom
• SmartNIC with dynamically downloadable firmware
© 2016 NETRONOME
SmartNIC Firmware: Pre-programmed or Custom
• SmartNIC with dynamically downloadable firmware
Contrail OVS
OpenStack ONOS ODL
Linux BSD
• OVS / Contrail / Linux eBPF datapath on host can be accelerated by SmartNICD
P D K
eBPF
© 2016 NETRONOME
SmartNIC Firmware: Pre-programmed or Custom
Compiler Debugger
Run-Time
app.P4 app.C
Editor
• Firmware can be developed in P4 and/or C • TBD how to integrate custom programmed
datapaths (implemented in P4 / C etc.) into OpenStack
• SmartNIC with dynamically downloadable firmware
Contrail OVS
OpenStack ONOS ODL
Linux BSD
• OVS / Contrail / Linux eBPF datapath on host can be accelerated by SmartNICD
P D K
eBPF
© 2016 NETRONOME 13
SmartNIC Datapath “Worker” Software Architecture
• Load balancer distributes each packet to next available thread for optimum throughput • Hardware assisted reordering ensures packet order is maintained • Flow tracker statefully learns / tracks millions of sessions • Matching performed using DRAM-backed tables - capacity > 500k entries • Actions efficiently performed in on-chip memory
Parse Match ActFlow Tracker Learn microflows
Cache action
Datapath Run to completion
Load Balance
C Plugin
Re- order
Pool of worker threads on flow processing cores
net or PCIE
net or PCIE
= Ring / Work Queue (multi producer / consumer)
© 2016 NETRONOME 14
Example: P4 “main” implementing a simple NIC
header_type eth_hdr { fields { dst : 48; src : 48; etype : 16; } } header eth_hdr eth;
parser start { return eth_parse; }
parser eth_parse { extract(eth); return ingress; }
action drop_act() { drop(); }
action fwd_act(port) { modify_field(standard_metadata.egress_spec, port); }
table in_tbl { reads { standard_metadata.ingress_port : exact; } actions { fwd_act; drop_act; } }
control ingress { apply(in_tbl); }
© 2016 NETRONOME
SmartNIC Firmware (P4/C)
Example of Fully Customized Datapath (P4 / C)
15
Run-Time Interface
Server (x86 - Linux)
PCIe
Agilio™ SmartNIC
Virtual Machine 1
VNF Kernel Mode
(C)
netdev
P4 / C Development Environment
Edit - Debug
Control App
Populate tables, display
statistics
Security µVNF (C)
Timestamp µVNF (C)
Latency Stats µVNF (C)
Virtual Machine 2
VNF User Mode
(C)
DPDK
Timestamp µVNF (C)
Latency Stats µVNF (C)
Match Protocol
Meter
Other
TCP
Concepts:
• P4 and C running on SmartNIC implements datapath - e.g. defines protocols, match / actionbehavior
• Datapath steers traffic to VNFsrunning on x86 server and on SmartNIC
© 2016 NETRONOME
SmartNIC Firmware (P4/C)
Example of Fully Customized Datapath (P4 / C)
15
Run-Time Interface
Server (x86 - Linux)
PCIe
Agilio™ SmartNIC
Virtual Machine 1
VNF Kernel Mode
(C)
netdev
P4 / C Development Environment
Edit - Debug
Control App
Populate tables, display
statistics
Security µVNF (C)
Timestamp µVNF (C)
Latency Stats µVNF (C)
Virtual Machine 2
VNF User Mode
(C)
DPDK
Timestamp µVNF (C)
Latency Stats µVNF (C)
Match Protocol
Meter
Other
TCP
Concepts:
• P4 and C running on SmartNIC implements datapath - e.g. defines protocols, match / actionbehavior
• Datapath steers traffic to VNFsrunning on x86 server and on SmartNIC
To Discuss: OpenStackIntegration
© 2016 NETRONOME
Extending OpenStack Networking - P4/C Plugins
16
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute Action
Open vSwitch Datapath
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
3 Offload datapath: copy match tables, sync stats
3
OVSKernel DP Match/Act
OVSKernel DP Match/Act
Miss
Miss
© 2016 NETRONOME
Extending OpenStack Networking - P4/C Plugins
16
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute Action
Open vSwitch Datapath
Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)
3 Offload datapath: copy match tables, sync stats
3
OVSKernel DP Match/Act
Datapath Extension or Plugin
P4 / C in Sandbox
DP Ext.
4 Datapath extension software
4
4
OVSKernel DP Match/Act
Miss
Miss
© 2016 NETRONOME
OVS “on” SmartNIC P4 Datapath
17
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API (Nova, Neutron)
Execute Action
OVSKernel DP Match/Act
P4 Matching
© 2016 NETRONOME
OVS “on” SmartNIC P4 Datapath
17
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API (Nova, Neutron)
Execute Action
P4 Generated Datapath
Execute P4
Action
OVSKernel DP Match/Act
P4 Matching
© 2016 NETRONOME
OVS “on” SmartNIC P4 Datapath
17
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API (Nova, Neutron)
Execute Action
P4 Generated Datapath
Execute P4
Action
OVSKernel DP Match/Act
P4 Matching
Fallback
© 2016 NETRONOME
OVS “on” SmartNIC P4 Datapath
17
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API (Nova, Neutron)
Execute Action
P4 Generated Datapath
Execute P4
Action
OVSKernel DP Match/Act
P4 Matching
Fallback
Fallback
© 2016 NETRONOME
P4 “into” OVS Datapath
18
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API (Nova, Neutron)
Execute P4 / OVS Action
P4 / OVS Matching
P4 / OVS Matching
© 2016 NETRONOME
P4 “into” OVS Datapath
18
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API (Nova, Neutron)
P4 Generated Datapath
Execute P4 / OVS Action
Execute P4 / OVS Action
P4 / OVS Matching
P4 / OVS Matching
© 2016 NETRONOME
P4 “into” OVS Datapath
18
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API (Nova, Neutron)
P4 Generated Datapath
Execute P4 / OVS Action
Execute P4 / OVS Action
P4 / OVS Matching
P4 / OVS Matching
Fallback
© 2016 NETRONOME
P4 “into” OVS Datapath
18
Open vSwitch Subsystem
OVS Agent
OpenFlow
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI Callable API (Nova, Neutron)
P4 Generated Datapath
Execute P4 / OVS Action
Execute P4 / OVS Action
P4 / OVS Matching
P4 / OVS Matching
Fallback
Fallback
© 2016 NETRONOME
P4 “instead of” OVS (Datapath and Control)
19
Open vSwitch Subsystem
P4 Agent
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
(Nova, Neutron)
P4 Generated Datapath
Execute P4
Action
Execute Action
P4 or eBPF or XDP
P4 Matching
Fallback
Fallback
1 Protocol(s) to be discussed (could become callable API)
© 2016 NETRONOME
P4 “instead of” OVS (Datapath and Control)
19
Open vSwitch Subsystem
P4 Agent
Virtual MachineVirtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV / VirtIO VFs
SR-IOV / VirtIO VFs
Agilio™SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
(Nova, Neutron)
P4 Generated Datapath
Execute P4
Action
Execute Action
P4 or eBPF or XDP
P4 Matching
Fallback
Fallback
1 Protocol(s) to be discussed (could become callable API)
Other open issues: - Downloading programs via OpenStack or other systems- Scheduling VMs to run on nodes with acceleration hardware (Nova)Acceleration discussion started at OpenStack Spring 2016 (Austin)
© 2016 NETRONOME 20
Integration / Open Sourcing Activities
Area ActivitiesLinux drivers VF netdev (kernel device
driver) upstreamed in kernel 4.5
Patches available for older kernels, or use VirtIO
Representative netdev (for fallback processing of traffic) proposals imminent
FreeBSD drivers Kernel device driver implemented
DPDK drivers Poll mode driver upstreamed in DPDK 2.2
Patches available for older DPDK versions
Open vSwitch accelerationintegration
Hooks for acceleration proposed in October 2014
Iteration in progress
OpenStack integration
Proposing plugins and agents to support virtual switching acceleration via RFE process
Integration for OVS in process - Mirantis, Ericsson CEE etc.
Integration for Contrail vRouter in process - Juniper etc.
© 2016 NETRONOME 20
Integration / Open Sourcing Activities
Area ActivitiesLinux drivers VF netdev (kernel device
driver) upstreamed in kernel 4.5
Patches available for older kernels, or use VirtIO
Representative netdev (for fallback processing of traffic) proposals imminent
FreeBSD drivers Kernel device driver implemented
DPDK drivers Poll mode driver upstreamed in DPDK 2.2
Patches available for older DPDK versions
Open vSwitch accelerationintegration
Hooks for acceleration proposed in October 2014
Iteration in progress
OpenStack integration
Proposing plugins and agents to support virtual switching acceleration via RFE process
Integration for OVS in process - Mirantis, Ericsson CEE etc.
Integration for Contrail vRouter in process - Juniper etc.
Also: P4 and protocol independent forwarding, switchdev, eBPF / XDP…
© 2016 NETRONOME 21
Next Steps
• Use Agilio™ SmartNICs with existing dataplanes • Use Agilio™ OVS (with / without Conntrack) • Use Agilio™ vRouter • Future: Agilio™ eBPF/XDP, etc.
• Program Agilio™ SmartNICs • Use APIs (on x86 servers) - with above dataplanes • Program in P4 and/or C (on SmartNIC / on x86)
• Improve performance + free up server resources!
© 2016 NETRONOME
More information: netronome.com and open-nfp.org
Thank You!
22
top related