2013/06/24 1

Linux Native VXLAN Integration

Toshiaki Hatano

Verio Inc.

2013/06/24 2

• Toshiaki Hatano

• Network Engineer, and Technical Account Manager at Verio

• Employee of NTT Communicationso  a leading telecommunication company in Japan

About me

2013/06/24 3

• We’re using CloudStack

• As core component of our Public Cloud Service

CloudStack and Us

Cloudn•

• We’re providing bothBasic and Advanced zone.

• Planning to provide VPC.

2013/06/24 4

• Advanced Zoneo have more functionality

• NAT, FW, LB, VPN• VPC

o Isolation required• For each guest network• For each VPC tier

• Isolation Method: VLANo VLAN ID is limited

• Only 4096 • Should be identical in a zone

o # of Domains are limited by VLAN• A domain require at least one VID

Problem: VLAN ID limit

VPC

Public Network

Virtual

Router

VPCTier

VM VM VM VM

VPCTier

GuestNetwork

VM VM

Virtual

Router

Isolated

AdvancedZone

2013/06/24 5

• VXLAN• VLAN like Layer 2 encapsulation over UDP• being standardized in IETF• 16M isolated network

• Why?• Open source implementation exists in Linux kernel• Work in distributed manner, just like VLAN

• Learning bridge• 1:N tunneling

• UDP encapsulation• No need of expensive network device to support

VXLAN and Why?

2013/06/24 6

VXLAN 1:N tunnel

Host

VM

vxlanYethX

brethX-Y

vnet

Underlying Network

VMVM

(not associated with VXLAN Y)

(1)(2)

① If multicast or broadcast or Unicast but host (Src) doesn’t know mapping VXLAN uses Multicast Host (Dst) learn mapping between VM and Host (Src)

② If Unicast and Host (Src) learned mapping between VM and Host (Dst) VXLAN uses Unicast

*1

*1: If underlying Network supports IGMP/MLD snooping and/or Multicast routing.

2013/06/24 7

• Initial target• KVM hypervisor with “Bridge” (not Open vSwitch)

• Only for Guest Network

• Share logic/UI-flow with VLAN as much as possible1. Assign VNI range for zone while zone creation

2. Allocate VNI for network while network creation

3. Automatically create VXLAN interface and connect it to bridgewhen first VM in network created

• To handle difference• Add isolation method “VXLAN”• Add Guru “VxlanGuestNetworkGuru”• Add code like “if( isolationmethod == “VXLAN” ) …”

to every code assuming VLAN, outside Guru

Implementation strategy

2013/06/24 8

CloudStack KVM VLAN – bridging Overview

KVMHost

Public Network

Internet

cloudbrX

ethX

VRvnetX

vnetX

brethX-Y

VMvnetX brethX-Y

KVMHost

Guest Network (VLAN encap)

ethX.Y ethX.Y

VM

vnetXethX ethX

2013/06/24 9

CloudStack KVM VXLAN– bridging Overview

KVM

Public Network

Internet

cloudbrX

ethX

VRvnetX

vnetX

brethX-Y

VMvnetX brethX-Y

KVM

VXLAN encapsulated

VM

vnetX

cloudbrX cloudbrXethX ethX

vxlanY vxlanY

2013/06/24 10

Requirement:

KVM/Bridge (not Open vSwitch)

Linux kernel 3.7 or later

VXLAN kernel module and iproute2 supporting

Recent Linux distribution satisfy this.

Fedora 17

Ubuntu 13

Etc.

User flow – (1) Setup KVM

2013/06/24 11

User flow – (2) Adding Zone

2013/06/24 12

User flow – (2) Adding Zone

* UI is Mockup

vNet

2013/06/24 13

User flow – (3) Adding Network

* UI is Mockup

vNet

2013/06/24 14

Packet capture

KVM 1

vxlanX

KVM 2 KVM 3vxlanX vxlanX

eth eth eth

VM 1

VR

VM 2 VM 3

1) Ping from VM1 to VM2(captured from vxlanX on KVM1)

2) Ping from VM1 to broadcast address(captured from vxlanX on KVM1)

bcastping.pcap

unicastping.pcap

2013/06/24 15

unicastping.pcap

Left: Outer packet Right: Decode inner frame

2013/06/24 16

bcastping.pcap

Left: Outer packet Right: Decode inner frame

2013/06/24 17

• We’re adding new network isolation method: “VXLAN”

• The goal is to provide bigger substitute of VLAN• And make as little change in UI/UX as possible

Summary

Special Thanks:Jamie Gritton: Verio Inc.Junji Arakawa: NTT Communications Corp.

2013/06/24 18

QUESTIONS?

Design Doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Feature+Proposal+-+Linux+native+VXLAN+support+on+KVM+hypervisor