overview of xenserver distributed virtual switch/controller and troubleshooting network issues

Overview of XenServer Distributed Virtual Switch/Controller and Troubleshooting Network IssuesBlaine A. Anaya XenServer Escalation Engineer05/24/2011

Overview

Agenda

XenServer Networking Architecture / vSwitch Architecture

Troubleshooting the Network

Overview

Agenda



Performance Testing

Networking Terminology

XenServer Networking Terminology

PIF- Physical Interface Object – directly correlates to a physical interface

VIF- Virtual Interface Object- directly correlates to a virtual interface in a VM

Bridge- Represents a network and is where PIFs and VIFs are plugged in

Dom0- Short form of Domain 0 the control domain in XenServer that manages network and storage connections for virtual machines

Bond- is the association of two network interface cards to make them appear as one

Trunk – a switch port designated to carry traffic for more than one VLAN

XenServer Networking

DomUDomUDomUDomU

Xen HypervisorXen Hypervisor

Dom0Dom0

ToolstackToolstack AppApp AppApp AppAppAppApp

Native Driver /

PIF

Native Driver /

PIF

netfront/ VIF

netfront/ VIF

netfront/VIF

netfront/VIF

Guest OSGuest OS Guest OSGuest OS

BridgeBridge

Host Machine (Hardware)Host Machine (Hardware)Host Machine (Hardware)Host Machine (Hardware)

netback/0 netback/0

netback/1 netback/1 netback/2 netback/2

netback/3 netback/3

XenServer Networking Configurations- Linux Stack

Linux NIC Drivers

Linux NIC Drivers

Linux Config Files

Linux Config Files

XenServer PoolDB

XenServer PoolDB

Network Card

XAPIXAPI

Command Line

XenCenter

xsconsole

XenServer Network Terminology

Internal Switches

PIF (eth0)

VIF

VIF

VIF

Virtual Machine

Virtual Machine

Network 0 (xenbr0)

Private(xapi1)

Network Card


Internal Switches

PIF (eth1)

PIF (eth0)

VIF

VIF

VIF

Virtual Machine

Virtual Machine

Network 1 (xenbr1)

Network 0 (xenbr0)

Network Card

Network Card


PIF (bond0)

PIF

VIF

VIF

Virtual Machine

Virtual Machine

Network Card

Network Card

VIF

Bond 0+1 (xapi2)

PIF (eth0)

PIF (eth1)

Bonding Type (Balance SLB)

Virtual Machine

Network Card

Network Card

Virtual Machine

Bond

0:00 SEC0:10 SEC0:20 SEC0:30 SEC

Stacked Switches

Virtual Machine

Distributed vSwitch

Open Virtual Switch for XenServer

VM

HypervisorHypervisor

VM VMVMVM


VM VMVMVM


Visibility· Resource control · Isolation · Security

VMVM

• Open Source Virtual Switch maintained at www.openvswitch.org• Rich layer 2 feature set (in contrast to others on the market)• Ships with XenServer 5.6 FP1 as a post-install configuration option

http://www.openvswitch.org/

Distributed Virtual Switch Controller


HypervisorHypervisorHypervisorHypervisorHypervisorHypervisor

VMVM VM VM VM VM VM VM VM VM VM

DVS Controller is a XenServer Virtual Appliance that controls multiple Open vSwitches

Distributed Virtual Switch

HypervisorHypervisorHypervisorHypervisorHypervisorHypervisor

Built-in policy-based ACLs move with VMs

DVS

VMVM VM VM VM VM VM VM VM VMVM

Virtual Interface (VIF) {MAC, IP} ACLspermit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 123

Virtual Interface (VIF) {MAC, IP} ACLspermit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 123

Enabling the vSwitch

Distributed Virtual Switch

[root@vswitch1-baa-r222 ~]# xe-switch-network-backend openvswitchCleaning up old ifcfg files Remove... ifcfg-bond0 Remove... ifcfg-bond1 Remove... ifcfg-eth0 Remove... ifcfg-eth1 Remove... ifcfg-eth2 Remove... ifcfg-eth3 Remove... ifcfg-eth4 Remove... ifcfg-eth5 Remove... ifcfg-xapi2 Remove... ifcfg-xapi4 Remove... ifcfg-xenbr0 Remove... ifcfg-xenbr3Enabling openvswitch daemonConfigure system for openvswitch networkingYou *MUST* now reboot your system

#xe-switch-network-backend openvswitch (Command must be ran on each individual host)

vSwitch Architecture – Process Level View

Distributed Virtual Switch DVS Controller

OVS

Flow Table

Flow Table Cache

vSwitchNetwork A

Flow Table

Flow Table Cache

vSwitchNetwork B

ovsdb-server vswitchd

OpenFlowJSON-RPC

PIF PIF

VIF

VIF

VIF

VIF

XenServer Networking Configurations- vSwitch

Linux NIC Drivers

Linux NIC Drivers

vSwitch Config

vSwitch Config

XenServer PoolDB

XenServer PoolDB

Network Card

XAPIXAPI

Command Line

XenCenter

xsconsole

DVSCWeb Interface

Overview

Agenda



Troubleshooting The Network

Symptoms Issue

• Intermittent Packet Loss/ Dropped Connections • Physical Connection/Switch Configuration, Bonding

• Physical Connection/Switch Configuration, Change in Hardware, Configuration Conflict.

• Network Appears Disconnected

• Bond Fails To Pass Traffic When One Leg is Disconnected

• Physical Connection/Switch Configuration, Bond Mode

• Using Command Line Interface (CLI)

• Off-line using a system status report• BareGrep Pro• Xenoscope



1.Check switch port configuration – Physical – Layers1-3 (Cables,NICs,Switch/Router connections)

2.Verify enabled network backend (Linux Bridge/vSwitch)

4.Use “brctl show” to see bridge/bond association.

3.Use ifconfig –a to see bonds, physical interface statistics, bridges.

5.Verify bonding configuration

6.Use ethtool for NIC settings, driver and firmware versions.

7.Use xe network-list, xe pif-list, to check XAPI configuration.

Common Configuration Items to Check


/etc/xensource/network.conf

/etc/sysconfig/network-scripts

/proc/net/bonding/bond0

/etc/sysconfig/iptables

Linux Bridge/vSwitch Enabled


# brctl show# Shows the bridges and the interfaces plugged into them

[root@vswitch1-baa-r222 ~]# brctl showbridge name bridge id STP enabled interfacesxapi2 0000.001517868b8f no bond1 eth5 eth4xapi4 0000.001d09699d86 no bond0 eth1 eth2 vif5.0 vif6.0xenbr0 0000.001d09699d84 no eth0xenbr3 0000.001517868b8c no eth3

Linux Bridge Enabled


#brctl showmacs <brname>#Shows a list of learned MAC addresses for this bridge.

[root@localhost ~]# brctl showmacs xenbr0port no mac addr is local? ageing timer 1 00:00:0c:07:ac:3c no 1.83 1 00:0c:29:3a:12:79 no 120.59 1 00:0c:29:fa:8e:e8 no 26.52



# netstat -np# Provides information on connections and processes.

[root@vswitch1-baa-r222 ~]# netstat -npActive Internet connections (w/o servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp 0 0 127.0.0.1:37259 127.0.0.1:443 ESTABLISHED 2645/stunneltcp 0 0 127.0.0.1:36806 127.0.0.1:80 ESTABLISHED 6280/stunneltcp 0 52 10.12.45.209:22 10.54.75.163:63296 ESTABLISHED 31145/5tcp 0 0 127.0.0.1:443 127.0.0.1:37259 ESTABLISHED 6280/stunneltcp 0 0 10.12.45.209:443 10.12.45.114:39105 ESTABLISHED 6280/stunneltcp 0 0 10.12.45.209:34969 10.12.45.194:6633 ESTABLISHED 5304/ovs-vswitchd



# netstat -s# Provides summary statistics for each protocol.

[root@vswitch1-baa-r222 ~]# netstat -sIp: 17340461 total packets received 9190 with invalid addresses 0 forwarded 0 incoming packets discarded 12463755 incoming packets delivered 14230986 requests sent out 8 dropped because of missing route

Tcp: 69504 active connections openings 126760 passive connection openings 0 failed connection attempts 229 connection resets received 17 connections established 12462000 segments received 13220998 segments send out 3144 segments retransmited 0 bad segments received. 416 resets sent



#ethtool –k <interface>#Provides information on current offload settings

[root@vswitch1-baa-r222 ~]# ethtool -k eth0Offload parameters for eth0:rx-checksumming: ontx-checksumming: onscatter-gather: ontcp-segmentation-offload: onudp-fragmentation-offload: offgeneric-segmentation-offload: ongeneric-receive-offload: offlarge-receive-offload: off



#ethtool –i <interface>#Provides information on driver/firmware versions for network cards

[root@vswitch1-baa-r222 ~]# ethtool -i eth0driver: bnx2version: 2.0.8efirmware-version: bc 2.9.1bus-info: 0000:04:00.0

vSwitch Enabled


#ovs-appctl bond/list#Shows Bridge, Bond, Slave Association

[root@vswitch1-baa-r222 ~]# ovs-appctl bond/listbridge bond slavesXapi2 bond1 eth4, eth5Xapi4 bond0 eth2, eth1

Disclaimer: Using OVS command line options for configuration purposes is not supported. The vSwitch should only be configured using XenCenter, xe CLI, xsconsole, and the Distributed vSwitch Controller.The commands shared here are for data collection and diagnostic purposes only.

vSwitch Enabled


#ovs-appctl bond/show bond0 #Shows bond members, up/down delay, and next rebalance time.

[root@vswitch1-baa-r222 ~]# ovs-appctl bond/show bond0updelay: 31000 msdowndelay: 200 msnext rebalance: 4314 msslave eth2: enabled

active slavehash 123: 1 kB load

86:43:b2:1a:f2:d0slave eth1: enabled

vSwitch Enabled


#ovs-appctl fdb/show <bridge_name> #Shows MAC Table/VLAN information for the bridge

[root@vswitch1-baa-r222 ~]# ovs-appctl fdb/show xapi4 port VLAN MAC Age 3 0 00:1d:09:2c:c4:c9 58 3 0 0a:34:ee:08:53:06 47 3 0 6a:e8:14:89:5c:af 42 3 0 ba:89:bf:f5:b8:ab 35 3 0 00:16:c8:d8:f1:11 27

vSwitch Enabled


#ovs-ofctl dump-flows <bridge_name> #Shows FlowTable – (ACLs applied from controller)[root@vswitch1-baa-r222 ~]# ovs-ofctl dump-flows xapi4 | grep dropMay 02 15:49:07|00001|ofctl|INFO|connecting to unix:/var/run/openvswitch/xapi4.mgmtcookie=0x0, duration_sec=171s, duration_nsec=25000000ns, table_id=1, priority=32763, n_packets=0, n_bytes=0, tcp,dl_dst=86:43:b2:1a:f2:d0,nw_dst=10.12.45.151,tp_src=80,actions=dropcookie=0x0, duration_sec=171s, duration_nsec=25000000ns, table_id=1, priority=65529, n_packets=15, n_bytes=930, tcp,in_port=4,dl_src=86:43:b2:1a:f2:d0,nw_src=10.12.45.78,tp_dst=80,actions=drop

vSwitch Enabled


#ovs-dpctl dump-flows <bridge_name> #Shows FlowCache – (ACLs applied from controller)[root@vswitch1-baa-r222 ~]# ovs-dpctl dump-flows xapi4 | grep mac86:43

tunnel00000000:in_port0004:vlan65535:pcp0 mac86:43:b2:1a:f2:d0->00:00:0c:07:ac:3c type0800 proto6 tos0 ip10.12.45.78->69.147.112.160 port4284->80, packets:1, bytes:62, used:2.160s, actions:drop

vSwitch Enabled


#ovs-appctl vlog/list#Show current logging levels[root@vswitch1-baa-r222 ~]# ovs-appctl vlog/list console syslog file ------- ------ ------bridge EMER ERR INFOvswitchd EMER ERR INFOxenserver EMER ERR INFOofproto EMER ERR INFOsflow EMER ERR INFOjsonrpc EMER ERR INFOfail_open EMER ERR INFOnetflow EMER ERR INFOovsdb_error EMER ERR INFO

vSwitch Enabled


#vlog/set module[:facility[:level]] #Modify vswitch logging level

Sets the logging level for module in facility to level:

•Module may be any valid module name (as displayed by the --list action on ovs-appctl(8)), or the special name ANY to set the logging levels for all modules.

• Facility may be syslog, console, or file to set the levels for logging to the system log, the console, or a file respectively, or ANY to set the logging levels for both facilities. If it is omitted, facility defaults to ANY.

Note: The log level for the file facility has no effect unless ovs-vswitchd was invoked with the --log-file option.

•Level must be one of emer, err, warn, info, or dbg, designating the minimum severity of a message for it to be logged. If it is omitted, level defaults to dbg.

Status Report and BareGrepPro

Troubleshooting The Network – Off-Line

Status Report and Xenoscope

Troubleshooting The Network – Off-Line

• CTX127885 - Introduction to XenServer Networking

• CTX123489 - XenServer VLAN Networking

• CTX124421 - Understanding Network Interface Card Bonds in XenServer

• CTX127970 - Distributing Guest Traffic Over Physical CPUs in XenServer

• CTX127065- XenServer Virtual Machine Performance Utility

• CTX123477 - How to Move a XenServer Pool to a Different IP Subnet

• CTX125358 - How to Identify the Network Adapters on XenServer

• CTX101810 - Communication Ports Used By Citrix Technologies

Useful Networking CTX Articles

Before you leave…

• Recommended related breakout sessions: • Session: YN203: Managing VM networking across the datacenter with XenServer distributed

virtual switching Date: Wednesday May 25th Time: 4:30-5:15 Room: Moscone 2003-2005

• Session surveys are available online at www.citrixsummit.com starting Thursday, May 26• Provide your feedback and pick up a complimentary gift at the registration desk

• Download presentations starting Friday, June 3, from your My Organizer Tool located in your My Synergy Microsite event account

overview of xenserver distributed virtual switch/controller and troubleshooting network issues

Documents

virtual switchcontroller

virtual machinesbond

xenserver virtual appliance

network interface cards

old ifcfg files

switch port

overview of xenserver

configuration conflict