openstack neutron's distributed virtual router
DESCRIPTION
These are the slides that I used in a presentation given to the Colorado OpenStack meetup group on Wednesday, September 17, 2014.TRANSCRIPT
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Distributed Virtual RouterIntroduced in the Juno Release of OpenStack Neutron
Carl BaldwinDVR Illustrations courtesy of Jack McCann
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
DVR Architecture
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3
Network Service Node(s)
Neutron deployment architecture without DVR
neutron-server
API
auth
database
Compute hosts
DHCP agent
L3 agent
ML2 plug-in
message
queue
Metadata agent
ovs agent
Nova metadata
ovs agent
Open vSwitch
Open vSwitch
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4
Network Service Node(s)
Neutron deployment architecture with DVR
neutron-server
API
auth
database
Compute hosts
DHCP agent
L3 agent
ML2 plug-in
message
queue
Metadata agent
ovs agent
w/dvr agent
Nova metadata
ovs agent
Open vSwitch
Open vSwitch
enable_distibuted_routing = Trueprograms DVR flow handling
L3 agent agent_mode = dvr
external network
compute nodes on external network
Metadata agent
Nova metadata
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5
API extension
Adds ‘distributed’ attribute to ‘router’ object
• Can be set by admin user through the API• Global default is set as “router_distributed” in neutron.conf• Default is False
• The attribute is only visible to admin tenant in GET• Cannot be updated• Work in progress to allow update from False to True
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6
“VM1-1
DVR – East-West (subnet-to-subnet)
patch-tun
br-int
eth0
QRouter-X
S1.1
S2.1
“VM2-1
patch-tun
br-int
eth0
QRouter-X
S1.1
S2.1
same gw IP/MACon each node
no remote bcastin to routers
ARP for gwkept local
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
“VM1-1
DVR – North-South (floating IP)
br-int
eth0
QRouter-X
qr rfp-x
“VM2-1
br-ex
floating IP namespace
QRouter-Y
qr rfp-y
external-vlan
floating IP NAT in router ns
floating IP host routes pointing to QRproxy-arp for floating IPs on br-ex(future option: BGP route injection)
local addressing connects QR to FIP-NS
default route via FIP-NS
fpr-x
fpr-y
fg-u
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8
DVR – North-South (default SNAT)
br-int
eth0
snat-X
qr qg-x
br-ex
snat-Y
qr qg-x
external-vlan
default SNAT in snat namespacedefault route via br-ex
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9
Database
router_id string uuid
distributed boolean
router_extra_attributes
host string 255
mac_address string 32
dvr_host_macs
port_id string uuid
host string
router_id string uuid
vif_type string
vif_details string
vnic_type string
profile string
cap_port_filter boolean
driver string
segment string
status string
ml2_dvr_port_bindings - port binding for all theports associated to a DVR identified by router_id
router_id string uuid
l3_agent_id string uuid
host_id string
csnat_gw_port_id
string uuid
csnat_l3_agent_bindings
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
config file options
neutron.conf[DEFAULT]router_distributed = Falsedvr_base_mac = fa:16:3f:00:00:00
ovs_neutron_plugin.ini[agent]enable_distributed_routing = False # Make the l2 agent run in dvr mode
l3_agent.ini[DEFAULT]agent_mode = legacy # legacy, dvr, or dvr_snat
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11
How to Distribute the Router’s Internal Port
OVS Flow Handling
• ARP Requests to Router Port are Blocked from the Tunnel• These ARP requests should only be seen by the local port
• Source Mac is Mapped to Host Mac on Overlay Network• All traffic generated by the • A mac address is allocated for each compute host• Mapping must be done on both ends of the tunnel
• Destination Mac Blocked from Overlay• These should go to the local port• They would create mac ambiguity in the overly
• L2 Pre-Population is Required• “Prevent(s) multiple unicast of routed packets destined to remove VMs.”
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
DVR Limitations
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13
Default SNAT still centralized
eth1
br-ex
snat-Y
qr qg-x
external-vlan
“VM1-1
patch-tun
br-int
eth0
qrouter-X
S1.1
S2.1
patch-tun
br-int
eth0
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14
Floating IP Namespace
• Pros• Eliminates Need for Public Address/IR• Keeps IR Macs Off External Net
• Cons• Extra Complexity in L3 Agent• Still Consumes a Public Address / CN
eth0
QRouter-X
qr rfp-x
br-ex
floating IP namespace
QRouter-Y
qr rfp-y
external-vlan
fpr-x
fpr-y
fg-u
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15
Heavy L2 Integration
• Led to Initial Dependence on OVS and Tunnel Protocols• Mitigate Scope Creep
• Distributed Port Concept Needs to be Abstracted
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16
Firewall as a Service (FWaaS) Complexity
• External Net Connects to Hypervisors• FWaaS is Needed There Too.
• Asymmetric Routing Problem• E/W Routing
“VM1-1
patch-tun
br-int
eth0
QRouter-X
S1.1
S2.1
“VM2-1
patch-tun
br-int
eth0
QRouter-X
S1.1
S2.1
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Contributing DVR
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.18
Initial Development on Havana
• Stable Code Base• No Risk of External
Regression
• Very Large Effort to Integrate• Upstream Moves Quickly• Subject to Regression
• Comm. Standard Enforcment• Code Style• No Demand for Unit or Functional
Tests
Pros and Cons
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.19
If We had to do it Over Again…
Initial Development on Havana
• Contribute Smaller, More Focused Patches to Trunk• Start with Pure Refactoring Needed to Ease Development• Develop Unit Tests for Code that will be Modified• Move Gradually Beyond Refactoring to Other Improvements• Divide Implementation According to Themes• Develop Unit Tests (TDD) and Functional Tests to Prevent
Regression
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20
Divided in to 7 Patches
• Division According to Component• Patches Added Unused Code for Later
Patches• Indicates there are themes that cross patch boundaries
• Each Patch Had Multiple Active Authors• Indicates possibly more than one theme in the patch
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21
Handling Multiple Changes
• Dependency Order Not Linear• Should it be?
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22
Handling Multiple ChangesIt is Never Linear!!!
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23
Handling Multiple Changes
• Automatic Rebase Feature• Default behavior of “git review”
• Most of the Time it is Disruptive
• Sometimes it is Destructive
Enemy Number One!
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.24
Handling Multiple Changes
• Clobbered API Extension!!• Happened More than Once
• Use --no-rebase Always• Rebase on Merge Conflict• Work from the bottom up
• Merge Faster• Smaller, more focused patches• Continuous community involvement
Enemy Number One!