overview of distributed virtual router (dvr) in openstack/neutron
TRANSCRIPT
Vivekanandan Narasimhan
Architectural Overview of Distributed Virtual Routers in
OpenStack Neutron
Paris OpenStack® Summit
Agenda Introduction
High level architecture and DVR Configuration
East-West Routing OVS Rules North-South Routing
API Changes and DB Extensions
Scheduling
Services
Plans for Kilo
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
Legacy Routing in Neutron
Network node provides:• IP forwarding
– Inter-subnet (east-west) traffic between VMs– Floating IP (north-south) traffic between
external and VM– Default SNAT (north-south) traffic from VM to
external• Metadata Agent
– access to Nova metadata service
Issues:• Performance bottleneck• Scalability limitations• Single Point of Failure
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron
Compute node
Network node
Compute node
VM1 VM2
VM1
VM2
VirtualPhysical
Router A
Rtr A
Paris OpenStack® Summit
Distributed Routing in Neutron
Compute nodes provide:• IP forwarding for local VMs
– Inter-subnet (east-west) traffic between VMs– Floating IP (north-south) traffic between external
and VM• Metadata Agent for local VMs
– access to Nova metadata service
Advantages:• Bypass network node improves performance• Scales with size of compute farm• Limited failure domain (per compute node)Limitations:• Default SNAT function is still centralized
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron
Compute node
Network node
Compute node
VM1 VM2
VM1
VM2
VirtualPhysical
Router A
Rtr ADefaul
tSNAT
Rtr A’’
Rtr A’
Paris OpenStack® Summit
High level requirements for DVR
• Provider feature– Tenants should not have to care
• Default router type set by global config knob– Distributed or Legacy
• Distributed and Legacy routers coexist in same cloud• Minimize overhead use of public IP addresses– one public IP per compute node
• Leverage existing code base• Ability to migrate from Legacy to Distributed
Paris OpenStack® Summit
DVR High Level Architecture and
Configuration
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
Network/Service Node(s)
Compute Node(s)
VMVM
VM
l3-agentovs-agent
Controller Node
Neutron controller/plugin
L3 Plugin
Network/Service Node(s)
l3-agentovs-agent
dhcp-agent
qr
qrqr
agent_mode=dvr
router_distributed=True
enable_distributed_routing=Trueagent_mode=dvr_snat
enable_distributed_routing=True
• One FIP namespace supports multiple routers
• Legacy OpenStack Routers (non-distributed) can exist on the network/service node
• Enhanced l3-agent works in different modes “dvr_snat”, “legacy” and “dvr”.
FIP
SNAT
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron
metadata
metadata
qrqr
DVR High level Architecture
Paris OpenStack® Summit
Configuring DVR
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
Plugin Configuration ( Default Router Option)“neutron.conf”router_distributed= True
L3 Agent Configuration and Modes
“l3_agent.ini”agent_mode=dvr_snat ( Network/Service Node)agent_mode=dvr (Compute Node only)agent_mode=legacy ( Network/Service Node)router_namespaces=True ( Enable namespace cleanup)(Optional)
L2/OVS Agent Configuration
“ml2_conf.ini”[ovs]
enable_tunneling = Truelocal_ip = <Local Data Network IP (or) TEP IP> [agents]tunnel_types = vxlanl2_population=Trueenable_distributed_routing = True
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron
DVR Configuration Overview
Paris OpenStack® Summit
br-int br-intbr-int
br-tun br-tun br-tunbr-ex
qr
VM1 VM2 VM3 VM4DHCP Agent
L3 Agent
OVS Agent
Q_DVR_MODE=legacy
External Network
Internal Network
Network Node Compute Node Compute Node
OVS Agent
Legacy Deployment
Paris OpenStack® Summit
br-int br-intbr-int
br-tun br-tun br-tunbr-ex
qr
VM1 VM2 VM3 VM4DHCP Agent
L3 Agent
OVS Agent
Q_DVR_MODE=dvr_snat Q_DVR_MODE=dvr Q_DVR_MODE=dvr
L3Agent
L3Agent
qr qrqr
snat
Network/Service Node Compute Node Compute Node
DVR Deployment without FIP
br-ex br-ex
Paris OpenStack® Summit
br-int br-int
br-tun br-tun
VM1 VM2 VM3 VM4
OVS Agent
Q_DVR_MODE=dvr_snat Q_DVR_MODE=dvr Q_DVR_MODE=dvr
L3Agent
L3Agent
qr qr
Enable
FIPEnable
FIP
fip fip
br-ex br-ex
DVR Deployment with FIP
br-int
br-tunbr-ex
qr
DHCP Agent
L3 Agent
qr
snat
Network/Service Node Compute Node Compute Node
Paris OpenStack® Summit
East-West Routing
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
L3 Agent
Rules
------
------
Rules
------
------
vm1 vm2
br-int
br-tun
qr
LMAC
East-West routing is accomplished through a combination of the following in each CN:•qr : a namespace that forwards traffic among routed subnets using route table entries.
•LMAC: a MAC address unique to each CN for use as source address in forwarding routed frames from one CN to another.
•A set of rules in the OVS bridges that prevent frames with src=gateway MAC from egressed tunnel bridge or the IR. These rules also cause peer bridges to substitute/restore gateway MACs with LMACs while routed frames traverse through br-tuns and underlay network.
br-ex
Compute Node entities for enabling the DVR East West
RoutingL3 Agent
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
vm1
br-int-cn1
br-tun-cn1
qr
vm2
br-int-cn2
br-tun-cn2
qr
Tenant has two VMs - vm1 that is in RED Net & vm2 that is in GREEN NetCN1 and CN2 are two compute nodes. qr represents a dvr owned by Tenant 1
Data Network
CN1 CN2
PING REQUEST from vm1 to vm2
vm1 ipqr red macvm1 mac vm2 ip
vm1 ipqr red mac vm2 ip
vm1 ipvm2 macqr grn mac vm2 ip
vm1 ipvm2 macdvr cn1 mac vm2 ip
vm1 ipvm2 macdvr cn1 mac vm2 ip
1
2
3
4
5
grn vni
grn 2 vlan
vm1 ipvm2 macqr grn mac vm2 ip6
vm1 mac
Flow of DVR routed packets (example)
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
Table 0: Dispatch from in_portfrom patch-int:goto table 1
default:drop
Table 1: DVR process TableIf packet is from dvr-router-intf-mac, replace SRCMAC with DVR-node-unique-mac and forward to table 2Any other packet, forward as is to table 2
Table 2: Local VLAN to TUN TableIf unicast frame, forward to table 20If broadcast/multicast frame, forward to table 21
Table 20: Unicast to Tunnel Table
Pre-populated L2-pop rules that bind dst-mac in frame to a single output tunnel
Frame out tosingle tunnel
Table 0: Local Switching Table from patch-tun:Is src-mac == DVR-Unique-MAC - goto table 1
default:forward packet to ports - NORMAL action
1
2to dvrrouter
Integration Bridge (br-int)
Tunnel Bridge (br-tun)4
5
6
3from dvr router
7
8
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron
Egress to Cloud
Paris OpenStack® Summit
Table 0: Dispatch from in_portfrom vxlan-port: forward to table 4
default: drop
Table 4: VXLAN Tun to Local Vlan Tablematch tun_id: Set local vlan-id, forward to table 9
default: drop
Table 9: DVR Learning blockerIf src_mac == DVR-Unique-MAC frame, don’t learn, just output to patch-int
default:forward to table 10
4
Integration Bridge (br-int)
Tunnel Bridge (br-tun)
1
2
3
Ingress from Cloud
Table 0: Local Switching Table from patch-tun:Is dvr_routed - goto table 1
default:forward packet to ports - NORMAL action
Table 1: DVR to LMAC Table match DVR-Unique-MAC: strip off dvr-unique-mac, add local dvr-router-intf-mac, output direct to vm-port
default:drop
5
vm1
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
6
North-South Routing
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
Overview of North-South Routing
External Network
vm1 vm3
br-int
br-tun
vm5
br-ex
• IP Rules
•Routing Entries
• Iptable based NATing rules
• /32 addresses for the floating
IPs
• One Floating IP Namespace
per CN
• Interface on external network
• Proxy ARPs enabled for hosted
floating IPs
qr
FIP
L3 Agent
Data Network
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron
Default SNAT Traffic flow
Paris OpenStack® Summit
API Changes and DB Extensions
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron
API ExtensionsAdds ‘distributed’ attributed to ‘router’ object
Router Create:
neutron router-create –distributed=True/False
Router Update:
neutron router-update –distributed=True/False
Can be set by admin user through the above APIGlobal default is set as “router_distributed” in neutron.confThe attribute is only visible to admin tenant in GET.
NOTE: Migrating or Converting a Legacy Router to Distributed is Work in Progress.
Paris OpenStack® Summit
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron
DB Changes for DVR
ml2_dvr_port_bindings
Paris OpenStack® Summit
Scheduling
Achitectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
br-int
br-tun
VM3 VM4
OVS Agent
Q_DVR_MODE=dvr_snat Q_DVR_MODE=dvr
L3Agent
qr
br-ex
Scheduling a “qr/snat” in Service/Compute Node
br-int
br-tunbr-ex
qr
DHCP Agent
L3 Agent
qr
snat
Schedule Events
Create a Router
Add one or more subnets with VMs
Set a default Gateway for the Router
Compute NodeService/Network Node
Paris OpenStack® Summit
br-int
br-tun
VM3 VM4
OVS Agent
Q_DVR_MODE=dvr_snat Q_DVR_MODE=dvr
L3Agent
qr
Enable
FIPEnable
FIP
fip
br-ex
br-int
br-tunbr-ex
qr
DHCP Agent
L3 Agent
qr
snat
Network/Service Node Compute Node
Scheduling a “fip” in Compute Node
Schedule Events
Make sure you have a router configured with External Network or gateway.
Create a Floating IP
Associate a Floating IP to a VM
Paris OpenStack® Summit
Un-Scheduling Routers and FIP Un Schedule Events
FIP Namespace cleanup
Last VM holding the FIP determines the FIP namespace cleanup Delete a FloatingIP Disassociate a Floating IP
Router Namespace cleanup
When no more DVR related ports ( Includes Compute, VIP, DHCP etc.,) are serviced by the routed subnet, the router namespaces are cleaned up.
Both l2 agent and l3 agent takes part in the clean up.
SNAT Namespace cleanup When a gateway is removed from a router the snat namespace will be
cleaned up.
NOTE: Namespace cleanup should be enabled “router_namespaces=True” in l3_agent.ini
Paris OpenStack® Summit
Services
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
Services support with DVR
Services Support in DVR as of Juno
LBaaS
FWaaS• North-South only ( External Traffic)• East-West ( Internal Traffic)
MetadataService
VPNaaS is still supported with Centralized Routers.
VPNaaS support for DVR is Work in Progress.
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
br-int br-int
br-tun br-tun
VM1 VM2 VM3 VM4
OVS Agent
Q_DVR_MODE=dvr_snat Q_DVR_MODE=dvr Q_DVR_MODE=dvr
L3Agent
L3Agent
qr qr
fip fip
br-ex br-ex
Firewall Firewall
LBaaS AgentVIP
br-int
br-tunbr-ex
qr
DHCP Agent
L3 Agent
qr
snat
Network/Service Node
Firewall
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron
Compute Node Compute Node
Firewall
Services deployment with DVR
Metadata
Metadata
Paris OpenStack® Summit
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron
DVR Enhancements in kilo
VPNaaS support for DVR Full migration support for DVR routers. HA for Service Node VLAN Support
Paris OpenStack® Summit
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron
Liberty Plans for DVR
L3 Agent Refactor Distributed DHCP Performance tuning. Distributed SNAT
Paris OpenStack® Summit
Questions?
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit