edge virtual bridging: introduction and implementation in ... › images › stories › pdf ›...

© 2009 IBM Corporation

Edge Virtual Bridging: Introduction and Implementation in Linux (Open-LLDP)

Thomas Richter– IBM Research and Development, Linux Technology Center 7-Nov-2012

2

Agenda

■ Virtualization & Bridges/Switches

■ Network Administration Issues

■ IEEE Standard 802.1 T M T M

Qbg

■ Principle of Operation

■ Open-LLDP Design & Configuration

■ Current & Future Work, Related Work

■ References, Acknowledgments, Trademarks

IBM Presentation Template Full Version

Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012

3

Virtualization & Switches Today (KVM on Linux)


Traffic between VMs within Host

■ Stays inside the host

■ No external traffic analysis in regards to– Security policies (Firewalls, Virus Scans, etc)– Network profiling

■ “Host oriented” approach– Vswitch & SR-IOV switch configuration maintained per host

Switch Locations

Virtual Bridge

SR-IOV (VF/PF)

VM VM VM

vir-br0

Host A

NIC

Switch

Intranet/Internet

Host B

NIC

vir-br0

External Switch

Sys

tem

Adm

in D

om

ain

Net

wor

k A

dmin

Dom

ain

VM


4

Network Administration Issues

■ Large number of virtual and physical switches in data centers

■ Host virtual switch offers filtering, ACLs, bandwidth limitations, QoS

– Not integrated in external switch management

– Outside scope of network administrators

– Needs manual configuration/verification per host

■ Migration of VMs

– Network policies and port profiles have to move with VM

– Manually ensure target is correctly configured

IEEE 802.1 Qbg enables

– Configuration and management of bridge services for VMs

– Multiple VMs to share a switch port for relay

“Network oriented” approach: consolidate/automate virtual and physical switch administration



5

IEEE 802.1 Qbg (Edge Virtual Bridging)■ All switching done externally

■ VEPA:– Frame relay (forwarding)– Inbound replicate received multi cast packets

■ Switch– Outbound port same as inbound port

■ Simplify physical and virtual switch management


VM VM VM

VEB

Host A

NIC

Switch

Host B

NIC

VEPA

VM VM

VSI TypeDatabase

■ Pros:– All traffic forwarded to external switch, host internal VM traffic visible and accountable– Reduces network configuration required by host administrator– No modifications of Ethernet frames

■ Cons:– Additional networking traffic and latency– Requires switch support/configuration– Simultaneous support for VEB and VEPA on same switch port not supported

Hairpin Mode


6

VM Attachment and Macvtap Device Options


Host B

App

macvtap0 macvtap1

Macvtap

■ Combines tun/tap and macvlan devices

■ Modes:

(1)Bridged: destination MAC address lookup on all macvtap devices defined on NIC

(2)Vepa:Traffic forwarded to external switch

(3)Private: Same as vepa, but ingress traffic blocked

(4)Passthrough: Only 1 macvtap device allowed per NIC (“exclusive” use)

NIC

/dev/tapX I/f to User Space (tuntap)

Virtual I/f with new MAC address

App

3

1

24


7

Principle of Operation

(1)Switch announces 802.1Qbg support on configured ports• Access to VSI Type database

(2)Lldpad on Linux host receives switch announcement• Negotiates switch port into “reflective relay”

(3)VM definition contains network information• Used by one or more VMs• Identified by unique ID

(4)Host sends <ID, MAC, VLAN-ID, ...> to switch• Lldpad for LLDP communication with switch• Libvirtd for VM network attachment

(5)Switch receives <ID, MAC, VLAN-ID, ...> on port• Confirms or denies VM connection to network• Enforces ACL and QoS

(6)Libvirtd starts VM

Switch Edge defines port characteristics: ACL, QoS, etc

VM Edge defines connection settings: VLAN-ID, MAC, UUID

Switch

Host B

Eth0

VM VM

VSI TypeDatabase *

1

Eth0.4

macvtap0 macvtap1

4 5

2

3

Lldpad

Libvirtd

A

B

A

B


VM Definition (ID, MAC, VLAN-ID)

6

* ID, ACL, QoS

8

IEEE 802.1 Qbg Protocols


■ Open-LLDP has been enhanced to support IEEE 802.1 Qbg– Draft 0.2 support available, ratified standard support under work– git://open-lldp.org/lldp/open-lldp

■ EVB Edge Virtual Bridge Protocol– Data Unit carried in LLDP messages– Exchanges information about “reflective relay” mode with switch port

■ CDCP Channel Discovery and Control Protocol– Data Unit carried in LLDP messages– Negotiates service-channels between host and switch port

■ ECP Edge Control Protocol– Simple data carrier protocol with retry and confirmation

■ VDP Virtual station interface Discovery and configuration Protocol– Payload of ECP– Negotiation of VM network data with switch port


9

EVB

■ Part of LLDP Messages

– LLDP sends network interface characteristics to neighbors

– 3 Agents for different bridge types (via multi cast MACs 00:80:C2:00:00:0x)

• Nearest bridge, nearest customer bridge, nearest notpmr bridge

■ EVB DU

– Send to nearest customer bridge only

– Exchange information about role, state, max number of retries and wait time

– Host requests reflective relay

– Switch accepts/denies request

SwitchTPMRSVLANSVLANHost.

Nearest Bridge

Nearest no TPMR BridgeNearest Customer Bridge


10

ECP

■ LLDP and DCB/DCBX are unacknowledged protocols

■ ECP protocol– Provides acknowledgment, signaling, re-transmit, sequence numbering– TLV formatted payload– Transmit arbitrary payload– Send to neareast customer bridge MAC (Ether type 0x8890)

TLVTLV

ECP-Buffer

Station/Host Bridge

TLVTLV

ECP-BufferTLV

TLV

TLVTLV

ECP-Buffer

Time1 ULP request sendTLV

TLV

TLV

3 set timer4 push to ULP

5 ACK

5a xmit timer?7 timer expired: re-transmit

TLV

ECP-Buffer

8 timer active: drop contents, incr seqno & continue

2 transmit data

Octets

ValueXx bits

Length9 bits

Type7 bits

1 2 3 N

Sample TLV

6 xmit timer?


11

VDP

■ Protocol to create, renew and destroy VSI associations

– VSI data consists

• VSI Type ID, VSI Manager ID, VM/IF UUID, <MAC,VLAN>

– Host usually initiator, switch responds with ack/nack

• Pre-associate (RR): send VSI data for switch to check

• Associate: Send VSI data to established association

• Dis-associate: Send VSI data to terminate association

– Associations renewed in regular intervals (keep alive)

– Associations can be terminated by switch

• Reboot, port disabled, etc Keep-Alive

Dis-associate

Association

Ack

Ack

Ack

BridgeStation

Time


12

CDCP

■ Share network link for simultaneous VEB/VEPA modes

– Divide link to logical channels using VLAN Tag (Service-VLAN)

• Assign logical channel to VM/VEBVEPA

– Sender inserts 802.1Q VLAN header, receiver removes it

– Requires Q-in-Q support for NIC and switch

■ Not yet implemented

VM

VM

VMVEB

VEPA

VMVM

VMVM

Mul

ti C

hann

el N

IC

Mu

lti C

hann

el S

w

Host

Server Edge

Switch Edge

8100 VLAN #

VLAN Tag

Add Remove


13

Example Configuration (with Switch Support for Qbg)LLDPAD Configuration File

eth2 : { tlvid00000001 : { info = "04001B217B3D24"; }; tlvid00000002 : { info = "03001B217B3D24"; }; Tlvid001b3f00 : /* EVB OUI */ { enableTx = true; fmode = "reflectiverelay"; capabilities = "vdp,ecp,rte"; }; adminStatus = 3; vdp : { enableTx = true; }};

VSI Type Database File

<vsi-type> <id> 123 </id> <version> 1 </version> <managerid> 1 </managerid> <vlanid> 4 </vlanid> <name>Thomas4</name> <bandwidth> <txrate> <txcommitedrate>512</txcommitedrate> <txburst>64</txburst> </txrate> <rxrate> <rxcommitedrate>1024</rxcommitedrate> <rxburst>128</rxburst </rxrate> </bandwidth> </vsi-type>

VM Configuration File (Networking Section)

<interface type='direct'> <mac address='08:18:21:63:be:e8'/> <source dev='eth2.4' mode='vepa'/> <virtualport type='802.1Qbg'> <parameters typeid='123' versionid='1' managerid='1' instanceid='a1412857-60f7-4ce1-e95a-2164943f53db'/> </virtualport> <target dev='macvtap0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/></interface>


14

Related and Future Work

Future:

■ Support for ratified standard (still at draft 0.2)

■ Support for Bonding (active-backup) – Other modes to be done

■ Support for SR-IOV NICs– Kernel and iproute2 support under work (Intel)

■ Support for SNMP

Related:

■ Lldpd (from Vincent Bernat: http://vincent.bernat.im): IEEE 802.1 Qbg not in plan

■ Ladvd (from Sten Spans: http://code.google.com/p/ladvd): IEEE 802.1 Qbg not in plan

■ OpenLLDP (from http://openlldp.sourceforge.net): ?


http://vincent.bernat.im/

http://code.google.com/p/ladvd

http://openlldp.sourceforge.net/

15

Questions?Questions?



16

References

(1)Blade Network Technology, Broadcom, Brocade, Citrix, Emulex, Exterme Networks, HP, IBM, Intel, Juniper Networks, Qlogic: “Standardizing Data Center Server-Network Edge Virtualizing”,http://www.extremenetworks.com/libraries/whitepapers/VEPA-EVB_whitepaper.pdf, Oct 2010

(2)IEEE Organization: “http://www.ieee802.org/1/pages/802.1bg.html

(3)Stuart Miniman: “Edge Virtual Bridging”, http://wikibon.org/wiki/v/Edge_Virtual_Bridging, 27 Feb 2012

(4)Vivek Kashyap: “Network Security in the Cloud and Datacenter”, Linux Foundation Collaboration Summit, 7 Apr 2011, San Francisco, Ca, USA

(5)Vivek Kashyap, Arnd Bergman, Stefan Berger, Gerhard Stenzel, Jens Osterkamp: “Automating Virtual Machine Network Profiles”, Linux Symposium, Ottawa, Canada, 13-16 Jul 2010, pp 147-152



http://www.extremenetworks.com/libraries/whitepapers/VEPA-EVB_whitepaper.pdf

http://www.ieee802.org/1/pages/802.1bg.html

http://wikibon.org/wiki/v/Edge_Virtual_Bridging

17

Acknowledgments

■ John Fastabend, Intel, Maintainer of open-lldp

■ Kishore Karolil, Florin Stelian, IBM Systems Networking for switch support

■ Vivek Kashyap, Gerhard Stenzel, Dirk Herrendörfer, Mijo Safradin, Sridhar Sumadrala, IBM

Linux Technology Center, Data Center Networking



18

Trademarks

■ This work represents the view of the author and does not necessarily represent the view of IBM.

■ IBM is a registered trademark of International Business Machines Corporation in the United States and/or other countries.

■ UNIX is a registered trademark of The Open Group in the United States and other countries .

■ Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

■ Other company, product, and service names may be trademarks or service marks of others.



19

Glossary

CDCP: Channel Discovery and Control Protocol

CEE: Converged Enhanced Ethernet

CNA: Converged Network Adapter

DCB: Data Center Bridging

DCBX: Data Center Bridging Extensions

ECP: Edge Control Protocol

EVB: Edge Virtual Bridge

LLDP: Link Layer Discovery Protocol (IEEE802.1AB)

SR-IOV: Single Resource Input/Output Virtualization

VDP: Virtual station interface Discovery and control Protocol

VEB: Virtual Ethernet Bridge

VEPA: Virtual Ethernet Port Aggregation

VM: Virtual Machine

VSI: Virtual Station Interface

Vswitch: Virtual Switch



20

BACKUPBACKUP



21

Example: VM Creation


VSIDatabaseNetwork Admin

creates VSI Types #

System Admin creates VM network withMAC, VLAN-ID, VSI

Switch12 Switch loads database

3

LLDPAD

LIBVIRTDLIBVIRTDLIBVIRTD

VIRT-MANAGER

VM

APP

EthX

EthX.4/Macvtap

4User starts VMSystem Admin

creates VM network withMAC, VLAN-ID, VSI *

5

VM Definitions

LLDPAD negotiates VSIData with Switch Port

86Associate VM VSIData with Switch Port

VM Definitions

Host

7

Libvirt creates & starts VM

VM communicates

* VSI: VSI Type ID,VSI Type Version ID, VSI Manager ID, VSI-IF UUID

# VSI Types: VSI Type ID,VSI Type Version ID, ACL, QoS, etcCourtesy of V. Kashyap [4] page 7

Hairpin Mode


22

Example: VM Migration


Switch

VIRT-MANAGER

LLDPAD

LIBVIRTDLIBVIRTDLIBVIRTD VM

APP

EthX

EthX.Y/Macvtap

Source Host

8

System Admin Migrates VM

LLDPAD

LIBVIRTDLIBVIRTDLIBVIRTD VM

APP

EthX

Target Host

1

2

SwitchVSI

Database

3

4

56 7VSI Dis-associate

Resume VM

Stop and move VM

Retrieve VSI Data

VSI Pre-associate

VSI Associate

EthX.Y/Macvtap

Courtesy of V. Kashyap [4] page 9

Hairpin ModeHairpin Mode


23

Bonding Support

■ Support for Bonding

– Mode: Active-backup

■ “Edge Relay” on bond interface

■ Switches are interconnected for VSI data exchange

Host

eth3

eth2

bond0Bond0.4

Switch2

Switch1VM

libvirtd EVBECP/VDP

lldpad

IntranetInternet


24

BACKUP SR-IOV NIC

■ SR-IOV NICS using 1 physical function and several virtual functions

■ Allows one PCIex device to appear as multiple independent PCIex devices– PF can be configured and managed– VF can just move data

■ VF are independent PCIex devices with limited functionality– Appear as individual network interfaces– Requires kernel support– Can be assigned to VMs

■ Integrated internal switch

■ References– http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-sr-iov-

technology-paper.html


Eth0

Eth1 EthnEth3Eth2 ....

Physical Function (PF)

Virtual Functions (VF)

NIC

Cable

Integrated switch


25

VM Network Attachment to Host NIC


Macvtap

■ TAP Interface to Ethernet NIC

■ Has MAC address of VM

■ Forwards frames from VM to NIC

■ Bypass of virtual switch in host

Host B

VM VM

Eth0

macvtap0 macvtap1

EthYEthX

SR-IOV NIC

VFInternal SwitchIn VEPA PF

Macvtap

■ SR-IOV with VF and VEPA mode

Host B

Eth0

VM VM

Eth0.4

macvtap0 macvtap1

NIC

EthY.4EthX.4


edge virtual bridging: introduction and implementation in ... › images › stories › pdf ›...

Documents