openstack neutron's distributed virtual router

© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Distributed Virtual RouterIntroduced in the Juno Release of OpenStack Neutron

Carl BaldwinDVR Illustrations courtesy of Jack McCann

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

DVR Architecture

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3

Network Service Node(s)

Neutron deployment architecture without DVR

neutron-server

API

auth

database

Compute hosts

DHCP agent

L3 agent

ML2 plug-in

message

queue

Metadata agent

ovs agent

Nova metadata

ovs agent

Open vSwitch

Open vSwitch

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4

Network Service Node(s)

Neutron deployment architecture with DVR

neutron-server

API

auth

database

Compute hosts

DHCP agent

L3 agent

ML2 plug-in

message

queue

Metadata agent

ovs agent

w/dvr agent

Nova metadata

ovs agent

Open vSwitch

Open vSwitch

enable_distibuted_routing = Trueprograms DVR flow handling

L3 agent agent_mode = dvr

external network

compute nodes on external network

Metadata agent

Nova metadata

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5

API extension

Adds ‘distributed’ attribute to ‘router’ object

• Can be set by admin user through the API• Global default is set as “router_distributed” in neutron.conf• Default is False

• The attribute is only visible to admin tenant in GET• Cannot be updated• Work in progress to allow update from False to True

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6

“VM1-1

DVR – East-West (subnet-to-subnet)

patch-tun

br-int

eth0

QRouter-X

S1.1

S2.1

“VM2-1

patch-tun

br-int

eth0

QRouter-X

S1.1

S2.1

same gw IP/MACon each node

no remote bcastin to routers

ARP for gwkept local

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7

“VM1-1

DVR – North-South (floating IP)

br-int

eth0

QRouter-X

qr rfp-x

“VM2-1

br-ex

floating IP namespace

QRouter-Y

qr rfp-y

external-vlan

floating IP NAT in router ns

floating IP host routes pointing to QRproxy-arp for floating IPs on br-ex(future option: BGP route injection)

local addressing connects QR to FIP-NS

default route via FIP-NS

fpr-x

fpr-y

fg-u

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8

DVR – North-South (default SNAT)

br-int

eth0

snat-X

qr qg-x

br-ex

snat-Y

qr qg-x

external-vlan

default SNAT in snat namespacedefault route via br-ex

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9

Database

router_id string uuid

distributed boolean

router_extra_attributes

host string 255

mac_address string 32

dvr_host_macs

port_id string uuid

host string

router_id string uuid

vif_type string

vif_details string

vnic_type string

profile string

cap_port_filter boolean

driver string

segment string

status string

ml2_dvr_port_bindings - port binding for all theports associated to a DVR identified by router_id

router_id string uuid

l3_agent_id string uuid

host_id string

csnat_gw_port_id

string uuid

csnat_l3_agent_bindings

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10

config file options

neutron.conf[DEFAULT]router_distributed = Falsedvr_base_mac = fa:16:3f:00:00:00

ovs_neutron_plugin.ini[agent]enable_distributed_routing = False # Make the l2 agent run in dvr mode

l3_agent.ini[DEFAULT]agent_mode = legacy # legacy, dvr, or dvr_snat

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11

How to Distribute the Router’s Internal Port

OVS Flow Handling

• ARP Requests to Router Port are Blocked from the Tunnel• These ARP requests should only be seen by the local port

• Source Mac is Mapped to Host Mac on Overlay Network• All traffic generated by the • A mac address is allocated for each compute host• Mapping must be done on both ends of the tunnel

• Destination Mac Blocked from Overlay• These should go to the local port• They would create mac ambiguity in the overly

• L2 Pre-Population is Required• “Prevent(s) multiple unicast of routed packets destined to remove VMs.”

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

DVR Limitations

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13

Default SNAT still centralized

eth1

br-ex

snat-Y

qr qg-x

external-vlan

“VM1-1

patch-tun

br-int

eth0

qrouter-X

S1.1

S2.1

patch-tun

br-int

eth0

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14

Floating IP Namespace

• Pros• Eliminates Need for Public Address/IR• Keeps IR Macs Off External Net

• Cons• Extra Complexity in L3 Agent• Still Consumes a Public Address / CN

eth0

QRouter-X

qr rfp-x

br-ex

floating IP namespace

QRouter-Y

qr rfp-y

external-vlan

fpr-x

fpr-y

fg-u

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15

Heavy L2 Integration

• Led to Initial Dependence on OVS and Tunnel Protocols• Mitigate Scope Creep

• Distributed Port Concept Needs to be Abstracted

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16

Firewall as a Service (FWaaS) Complexity

• External Net Connects to Hypervisors• FWaaS is Needed There Too.

• Asymmetric Routing Problem• E/W Routing

“VM1-1

patch-tun

br-int

eth0

QRouter-X

S1.1

S2.1

“VM2-1

patch-tun

br-int

eth0

QRouter-X

S1.1

S2.1

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Contributing DVR

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.18

Initial Development on Havana

• Stable Code Base• No Risk of External

Regression

• Very Large Effort to Integrate• Upstream Moves Quickly• Subject to Regression

• Comm. Standard Enforcment• Code Style• No Demand for Unit or Functional

Tests

Pros and Cons

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.19

If We had to do it Over Again…

Initial Development on Havana

• Contribute Smaller, More Focused Patches to Trunk• Start with Pure Refactoring Needed to Ease Development• Develop Unit Tests for Code that will be Modified• Move Gradually Beyond Refactoring to Other Improvements• Divide Implementation According to Themes• Develop Unit Tests (TDD) and Functional Tests to Prevent

Regression

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20

Divided in to 7 Patches

• Division According to Component• Patches Added Unused Code for Later

Patches• Indicates there are themes that cross patch boundaries

• Each Patch Had Multiple Active Authors• Indicates possibly more than one theme in the patch

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21

Handling Multiple Changes

• Dependency Order Not Linear• Should it be?

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22

Handling Multiple ChangesIt is Never Linear!!!

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23

Handling Multiple Changes

• Automatic Rebase Feature• Default behavior of “git review”

• Most of the Time it is Disruptive

• Sometimes it is Destructive

Enemy Number One!

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.24

Handling Multiple Changes

• Clobbered API Extension!!• Happened More than Once

• Use --no-rebase Always• Rebase on Merge Conflict• Work from the bottom up

• Merge Faster• Smaller, more focused patches• Continuous community involvement

Enemy Number One!