brocade-vcs-gateway-vmware-dp

53-1003610-02January 2015

Brocade VCS Gateway forVMware NSXDeployment Guide

© 2015, Brocade Communications Systems, Inc. All Rights Reserved.

ADX, Brocade, Brocade Assurance, the B-wing symbol, DCX, Fabric OS, HyperEdge, ICX, MLX, MyBrocade, OpenScript, The EffortlessNetwork, VCS, VDX, Vplane, and Vyatta are registered trademarks, and Fabric Vision and vADX are trademarks of BrocadeCommunications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned may betrademarks of others.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning anyequipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this documentat any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not becurrently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained inthis document may require an export license from the United States government.

The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to theaccuracy of this document or any loss, cost, liability, or damages arising from the information contained herein or the computer programs thataccompany it.

The product described by this document may contain open source software covered by the GNU General Public License or other opensource license agreements. To find out which open source software is included in Brocade products, view the licensing terms applicable tothe open source software, and obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.

http://www.brocade.com/support/oscd

Contents

Preface..................................................................................................................................... 4Glossary of Terms.............................................................................................4Document History..............................................................................................5Related Documents...........................................................................................5

Background.............................................................................................................................. 6Virtual Extensible Local Area Network (VXLAN)...............................................6

VXLAN Segments and Packet Format..................................................7VXLAN Unicast Packet Path................................................................. 7Layer 2 VXLAN Gateway...................................................................... 8

Brocade VCS Gateway for VMware NSX overview.......................................... 9VMware NSX.....................................................................................................9

Transport Components....................................................................... 10Logical Components........................................................................... 12NSX Network Views............................................................................ 12OVSDB................................................................................................13

Brocade VCS Gateway and VMware NSX......................................................13Control flow between Brocade VCS gateway fabric and VMware

NSX............................................................................................... 13Data flow in a Brocade VCS Gateway and VMware NSX based

topology......................................................................................... 14

Deployment............................................................................................................................ 16Supported Brocade Hardware and Software.................................................. 16VMware NSX requirements.............................................................................16Deployment Topologies.................................................................................. 17

Top of Rack (ToR) Topology...............................................................17One-Arm Gateway Topology...............................................................18

Limitations and Restrictions............................................................................ 19

Configuration..........................................................................................................................20Topology used in Configuration...................................................................... 20Configuration Steps for the VMware NSX solution with VCS VXLAN

Gateway.................................................................................................... 21Set up NSX Controller Cluster............................................................ 21Set up the Service Node..................................................................... 22Brocade VCS VXLAN Gateway Configuration.................................... 22Hypervisor (KVM) Configuration......................................................... 24NSX Manager Configuration............................................................... 26

Verification............................................................................................................................. 42

Brocade VCS Gateway for VMware NSX Deployment Guide 353-1003610-02

Preface

● Glossary of Terms...........................................................................................................4● Document History............................................................................................................5● Related Documents.........................................................................................................5

This document is a deployment guide for implementing the Brocade VCS VXLAN gateway for VMwareNSX. It is written for technology decision-makers, architects, systems engineers, NOC engineers andother experts.

This document provides an overview of VXLAN technology, describes the Brocade VCS VXLANgateway solution for VMware NSX, depicts two common use cases for deploying the Brocade VCSVXLAN Gateway and provides the VXLAN gateway related configuration and verification proceduresfor a sample solution.

Step-by-step examples to prepare, perform, and verify the deployments are explained. It is assumedthat the reader is familiar with establishing console access and entering commands using the BrocadeCLI. For information about the Brocade CLI, refer to theNetwork OS Administration Guide.

For details on configuration and deployment of VMware NSX, please refer to VMware documentation.

Glossary of Terms

Terms Definition

VXLAN Virtual Extensible Local Area Network - An overlaytechnology for carrying layer 2 Ethernet frames withinLayer 3 IP packets

VNI VXLAN Network Identifier - A 24-bit field in the VXLANheader uniquely identifying a Layer 2 broadcast domainin a VXLAN based network

VTEP VXLAN Tunnel End Point - an entity which originatesand/or terminates VXLAN tunnels. A VTEPencapsulates as well as decapsulates Ethernet framesusing VXLAN

L2 VXLAN Gateway Layer 2 VXLAN gateway - A VTEP endpoint whichprovides encapsulation of VLAN traffic from a physicalnetwork to VXLAN based virtual network anddecapsulation of VXLAN traffic from the virtual networkto VLAN traffic in a physical network

OVSDB Open vSwitch Database Management Protocol - aprotocol for interacting with Open vSwitch database forthe purposes of managing and configuring OpenvSwitch instances. Defined in RFC 7047

VMware NSX/NVP Network Virtualization and Security Platform fromVMware

ToR Top of Rack

TRILL Transparent Interconnection of Lots of Links - A Layer 2overlay technology used in Brocade VCS for carryingEthernet frames

Preface

4 Brocade VCS Gateway for VMware NSX Deployment Guide53-1003610-02

Document History

Date Version Description

2014-12-04 1.0 Deployment guide for NOS 5.0.1a

2015-01 2.0 Updated document

Related DocumentsSee the following URLs for related documents and information:

• Brocade VCS Gateway for VMware NSX Solution Brief:

http://www.brocade.com/downloads/documents/solution_briefs/partners_solution_briefs/brocade-vcs-gateway-vmware-nsx-sb.pdf

http://www.vmware.com/files/pdf/products/nsx/Brocade-NSX-SolutionBrief.pdf• Brocade VCS Gateway and VMware NSX Video on Demand (VoD):

http://www.vmware.com/products/nsx/resources.html• Brocade VDX 6740 Switches:

http://www.brocade.com/products/all/switches/product-details/vdx-6740-switch/index• Brocade VDX 6940 Switches:

http://www.brocade.com/products/all/switches/product-details/vdx-6940-switch/index.page• Configuration guide for Brocade VCS VXLAN Gateway for NSX (NOS 5.0.1a):

http://www.brocade.com/downloads/documents/html_product_manuals/NOS_501_LAYER2/GUID-773FF857-F62D-4A4A-89AA-B2301BF3ED8E.html

Document History




http://www.vmware.com/files/pdf/products/nsx/Brocade-NSX-SolutionBrief.pdf

http://www.vmware.com/products/nsx/resources.html

http://www.brocade.com/products/all/switches/product-details/vdx-6740-switch/index

http://www.brocade.com/products/all/switches/product-details/vdx-6940-switch/index



Background

● Virtual Extensible Local Area Network (VXLAN).............................................................6● Brocade VCS Gateway for VMware NSX overview........................................................ 9● VMware NSX...................................................................................................................9● Brocade VCS Gateway and VMware NSX....................................................................13

Brocade VCS Gateway for VMware NSX solution interconnects Ethernet VLAN based physicaldevices with VXLAN based virtual overlay networks, providing data center operators a unified networkoperations model for all application types. This section provides details about VXLAN, Brocade VCSVXLAN gateway and VMware NSX that should be understood before deploying and configuring thesolution.

Virtual Extensible Local Area Network (VXLAN)Server virtualization allows many Virtual Machines (VMs) per physical device, with each VM beingassigned a unique MAC address. The traditional Ethernet frame provides a 12-bit field for identifyingthe VLAN ID, which supports up to 4096 unique VLAN IDs. This limit may be reached with a largenumber of VMs in a single Ethernet network when the VMS have to be segregated via VLANs.

In addition, a virtualized infrastructure is often used to support multiple tenants, and each tenantrequires network separation to support administrative, security policy, and application requirements.Also, with multiple tenants sharing a single physical infrastructure, a mechanism is required to ensurethe uniqueness of MAC addresses and VLAN IDs.

VXLAN is an overlay technology that addresses these requirements by encapsulating MAC traffic fromindividual VMs and sending the encapsulated traffic over a logical tunnel. Brocade VCS Fabrictechnology with VXLAN supports the creation of large numbers of virtual domains above existingnetworks. This enables organizations to efficiently use their current infrastructure while leveraging thebenefits of VXLAN to support multi-tenancy and large-scale deployment of applications and VMs.

VXLAN based overlay networks for datacenter deployments provide the following benefits as opposedto a "traditional" datacenter:

• Extends the number of broadcast domains beyond the 4094 supported by the IEEE 802.1Q 12-bitVLAN identifier. This provides a larger number of logical network segments required for multi-tenantcloud deployments

• Creates a logical network overlay on top of a Layer 3 physical layer, extending Layer 2 broadcastdomains across Layer 3 boundaries

• Provides decoupling of the virtual topology provided by the VXLAN tunnels from the physicaltopology of the network

• Provides Layer 3 benefits, such as load balancing on redundant links leading to higher networkutilization. This is in contrast to a traditional L2 Ethernet network with Spanning Tree Protocol (STP)where redundant links may be blocked

• Isolating the physical network from the addressing of the virtual networks, thus avoiding issues suchas MAC table size in physical switches

• Support for virtual machine mobility independent of the physical network

Background


VXLAN Segments and Packet FormatA VXLAN segment is functionally equivalent to a traditional Layer 2 domain. Each VXLAN segment isidentified by a 24-bit VXLAN Network Identifier (VNI) within the VXLAN packet, illustrated Figure 1 .

FIGURE 1 VXLAN Packet Format

The original Ethernet frame consists of the source and destination MAC addresses, Ethernet type, andan optional IEEE 802.1q header (VLAN ID). This frame is encapsulated using VXLAN, which adds thefollowing additional headers.

VXLAN Header: This is an 8-byte (64-bit) field that includes the following important fields:

• Flags: 8 bits in length, where the 5th bit (I flag) is set to 1 to indicate a valid VNI. The remaining 7 bits(R bits) are reserved fields and are set to zero

• VNI: 24-bit value that provides a unique identifier for the individual VXLAN segment. VMs in differentVXLAN segments cannot communicate with each other. The 24-bit VNI provides a unique identifierfor up to 16 million VXLAN segments within a single administrative domain.

Outer UDP Header: The source port in the outer UDP header is dynamically assigned by theoriginating VTEP and the destination port is typically the well-known UDP port 4789 but may varybetween implementations.

Outer IP Header: The outer IP header has a source IP address of the source VTEP associated with theinner frame source (traffic endpoint like a virtual machine or a physical server). The outer destination IPaddress is the IP address of the destination VTEP corresponding to the inner frame destination.

Outer Ethernet Header: The outer Ethernet header has a source MAC address of the VTEPassociated with the inner frame source. The destination MAC address is the MAC address of the routingnext-hop to reach the destination VTEP. The outer Ethernet header may be tagged with an IEEE 802.1qfor the transport network.

VXLAN Unicast Packet PathA sample unicast packet flow is shown in Figure 2 with the packet contents at each point (1-5).

1. VM-A sends an IP packet to VM-B with a source IP address 9.9.9.1 and a destination IP address of9.9.9.2 with an Ethernet destination mac of VM-B (original Ethernet frame shown in green)

2. VTEP-A (Host-A) encapsulates the original Ethernet frame along with the IP payload within a VXLANheader (shown in yellow), using its VTEP source IP address and a destination IP address of VTEP-B(Host-B) in the outer IP header (outer Ethernet and outer IP header shown in blue)

3. The packet is routed through the IP network to VTEP-B (145.13.1.1)4. VTEP-B (Host-B) receives the packet and removes the VXLAN header and forwards the original

Ethernet frame5. VM-B receives the original Ethernet frame

VXLAN Segments and Packet Format


FIGURE 2 VXLAN Unicast Packet Path

Layer 2 VXLAN GatewayFrame encapsulation and decapsulation is performed by a VXLAN tunnel endpoint (VTEP). A VTEPoriginates and terminates VXLAN tunnels. A Layer 2 VXLAN gateway is a VTEP endpoint whichprovides encapsulation of VLAN traffic from a physical network to a VXLAN based virtual network, anddecapsulation of VXLAN traffic from the virtual network to VLAN traffic in the physical network.Brocade VCS VXLAN gateway solution for VMWare NSX, includes a VTEP providing L2 VXLANgateway functionality.

A Layer 2 VXLAN gateway device has at least one interface connected to a VXLAN segment andanother to a native Ethernet Layer 2 segment. The gateway provides bridging between the nativeEthernet segments and VXLAN virtual networks.

FIGURE 3 L2 VXLAN Gateway for VMware NSX with Brocade VCS

Figure 3 shows a Brocade VCS fabric with a VDX 6740 configured as a L2 VXLAN gateway forVMware NSX.

Brocade VCS must communicate with the NSX controller to create tunnels with VXLAN-aware enddevices like OVS in the Hypervisors. The NSX controller function can comprise a cluster of controllers.

Layer 2 VXLAN Gateway


The communication between the VXLAN gateway and the NSX controller occurs over the managementinterface.

Brocade VCS Gateway for VMware NSX overviewBrocade VCS Gateway for NSX provides a solution that unifies both virtual and physical infrastructurefor a seamless transition to cloud environments. By unifying the best of both worlds-physical and virtual,Brocade VCS Gateway for NSX enables physical devices to connect to an NSX virtual platform (virtualdevices). Brocade VCS Gateway for NSX is integrated with VMware NSX and enables the entire VCSfabric to function as a VXLAN gateway, eliminating the need for specific network placement. And theNSX Controller sees the VCS fabric as a single logical gateway. This provides management simplicityand resiliency. Users can leverage existing infrastructure while gaining the benefits of VXLAN to supportmulti-tenancy and large-scale deployment of (distributed) applications and Virtual Machines. Given thatthe Brocade VCS Gateway for NSX is integrated with the VMware NSX Controller, it offers the benefitsof agility with self-service provisioning, flexible network architecture, and scale-out modularity with VCSFabric, multitenancy and an easily managed unified solution for physical and virtual assets.

FIGURE 4 Brocade VCS Gateway for VMware NSX

VMware NSXVMware NSX manages the vSwitches in hypervisors and uses VXLAN to interconnect the virtualnetworks together to create a logical network spanning across Layer 3 physical/underlay networks .NSX also provides centralized northbound APIs to provision and configure many isolated logical virtualnetworks that run on top of a single physical network. NSX manages virtual networks by instantiatingand controlling the VXLAN tunnels connecting the logical segments.

In addition, NSX can interconnect third-party devices like Brocade VCS to the virtual networks toleverage services like L2 VXLAN gateway described in this document. In order to work with VMwareNSX, devices like Brocade VCS must provide a VXLAN tunnel endpoint (VTEP) and conform to NSX

Brocade VCS Gateway for VMware NSX overview


southbound API requirements. This API uses the OVSDB protocol for information exchange andprovides its own schema. NSX manages the tunnels on the devices; other Layer 2 and Layer 3configuration is outside NSX scope

Figure 5 shows NSX communication with Open vSwitches (OVS) and the Brocade VCS based L2VXLAN gateway for NSX, using OVSDB protocol. Also shown are the VXLAN tunnels formed betweenthe OVS (virtual infrastructure) and the Brocade VCS gateway (physical infrastructure).

FIGURE 5 Integrating with VMware NSX Using a VDX Gateway

The NSX components are divided into two categories:

1. Transport Components2. Logical Components

Transport ComponentsNSX Manager

Transport Components


NSX manager provides a centralized management plane. It provides a web-based GUI managementdashboard for user friendly human interaction with the VMware NSX controller cluster API, for systemsetup, administration and troubleshooting.

NSX Controller

The VMware NSX controller cluster is a system of x86 machines responsible for the programmaticdeployment of virtual networks. The controller cluster accepts API requests from northboundmanagement platforms (e.g. vCloud, OpenStack), calculates the virtual network topology, and programsthe hypervisor vSwitches and Gateways with the appropriate real-time configuration and forwardingstate. As the computing environment dynamically changes, the controller cluster updates the necessarycomponents to keep the virtual network state in lock-step with the virtual computing state.

The NSX controller cluster has visibility to all virtual machines and network services provisioned withNSX. With this knowledge, the NSX controller cluster can program all NSX components with the virtualnetwork topology. The NSX controller cluster is out-of-band, and does not handle data packets.

NSX controller interacts with the Brocade VCS Gateway to get the topology information, VXLAN tables,MAC addresses and VTEP L2 gateway location. This information is used by NSX to provide VNI-to-VLAN mapping and MAC address reachability to the vSwitches as well as the VCS fabric. Additionaldetails of information exchange between NSX and Brocade VCS gateway is provided in a 'BrocadeVCS Gateway and VMware NSX' section of this document.

Hypervisor vSwitch

Each hypervisor has an in-kernel vSwitch with a programmable data plane and configuration database.The controller cluster programs each hypervisor vSwitch with configuration and forwarding states, tomatch the desired virtual network topology to which the virtual machines are attached. As any givenvirtual network spans multiple hypervisors, the controller dynamically programs IP encapsulationtunnels (such as VXLAN) between hypervisors, decoupling the VM address space and virtual networksfrom the physical network fabric.

Layer 2 VTEP Gateway

Some applications within NSX might need to connect to services on non-virtual hosts within the datacenter, such as IP storage. For this requirement, NSX offers L2 Gateway services where L2 Gatewaynodes like Brocade VCS based L2 VXLAN gateway, can bridge between NSX virtual networks andVLANs on a physical network.

Gateway nodes provide a Gateway service, implementing the same programmable vSwitch ashypervisors, and managed by the controller cluster. The NSX Manager defines the Gateway servicesvia API requests to the controller cluster, which calculates the topology and programs Gateway nodeswith the necessary tunnels (VXLAN) and forwarding state, thereby attaching the NSX virtual networks tothe appropriate Gateway service.

Service Node

Service Nodes are x86 machines managed by the NSX Controller dedicated to performing additionalCPU intensive packet processing services such as handling broadcast, unknown unicast, multicast(BUM) frames - offloading that work from hypervisor hosts. The handling of BUM frames by a scale-outcluster of Service Nodes provides scalable network virtualization on any network. Service node has atunnel established from each Hypervisor and the Gateway, and uses these tunnels to forward thepackets.

Physical Server

A physical server is a non-virtual host in a data center and is configured to serve regular VLAN traffic.This server does not have capabilities to handle VXLAN traffic (cannot be a VTEP point). Example, adatabase server.

Background


Logical ComponentsLogical Switch (LSW)

A NSX logical switch creates a logically abstracted segment to which applications or tenant machinescan be wired. Each logical segment represents a layer 2 domain of communication.

With VMware NSX a logical switch is mapped to a unique VXLAN. When mapped to a VXLAN thevirtual machine traffic is encapsulated and is sent out over the physical IP network. The NSX controlleris a central control point for logical switches. Its function is to maintain state information of all virtualmachines, hosts, logical switches and VXLANs on the network.

Logical Switch Ports (LSP)

Each Logical Switch includes one or more Logical Switch Ports. Each logical port includes anAttachment that describes either the VM interface or physical network that acts as a source/sink oftraffic sent in and out of that logical port. Below are the types of attachments deployed in this guide.

Vif Attachment attaches the virtual interface (VIF) of a VM to a logical port on a logical switch

L2 Gateway Attachment connects NSX logical network to a physical L2 segment. A L2 GatewayAttachment connects a logical switch port to a physical network interface (and optionally maps it to aVLAN) exposed via an NSX Gateway Service.

NSX Network ViewsNSX provides two different views of the network, Logical Network view and Transport Network view,which are described in the following sections.

NSX Logical Network View

The Logical Network view is a representation of network resources presented to VMs andadministrators that is independent of the underlying physical devices. In a multi-tenant cloud, eachtenant has its own logical network view and does not see the logical network views of other tenants.The logical network view consists of logical ports, switches, and routers that interconnect VMs andconnect VMs to the external physical network.

VM administrators perform VM network provisioning, configuration, and monitoring using the logicalnetwork view, without regard for the physical network and the placement of VMs. The VMadministrator simply connects VMs to logical switches and logical routers. NSX then manages anychanges to physical network connectivity that is required to support VM migration or upgrades tonetwork hardware.

NSX Transport Network View

The Transport Network view represents the actual devices in the physical network. These physicaldevices, including hypervisor servers and network devices (gateways), are referred to as transportnodes. Each transport node runs an instance of OVS. When a VM migrates, NSX notifies the transportnodes to ensure stable logical network operation. The data center administrator works with thetransport network view, connecting hypervisors to the transport network, deploying transport nodessuch as hypervisors and gateways, and connecting them to the physical network.

The NSX Dashboard displays the logical components on the left and the transport components on theright. (See Figure 6 )

Logical Components


FIGURE 6 NSX Dashboard (Network Components Tab)

OVSDBOpen vSwitch (OVS) is the networking component inside the hypervisors. OVSDB is a protocol that hasthe capabilities to enables NSX orchestration with other physical and virtual networking devices.Hypervisor vSwitches and L2 VTEP Gateway devices like Brocade VCS fabric run an API that usesOVSDB protocol for information exchange. Brocade VCS fabric based Layer 2 VXLAN gatewayadheres to the API requirement of NSX and capable of running OVSDB protocol and can establishtunnels with the VXLAN capable hypervisor vSwitches or/and other Gateways.

Brocade VCS Gateway and VMware NSX

Control flow between Brocade VCS gateway fabric and VMware NSXFigure 7 illustrates a high level flow of control messages between Brocade VCS Gateway and VMwareNSX.

OVSDB


FIGURE 7 Control Flow between VCS Gateway and NSX

Steps in the above diagram are explained below. The corresponding configuration steps are describedin detail in the Configuration section of this document.

1. Brocade VCS fabric is configured as a VXLAN L2 gateway for VMware NSX2. NSX manager is configured with the Brocade VCS gateway information3. Brocade VCS fabric provides NSX with information about itself and the physical ports in the VCS

fabric4. The NSX administrator associates a physical port on Brocade VCS with a VTEP L2 gateway service5. The NSX administrator creates a Logical Switch (LSW) and associates it with a VNI and a transport

zone6. The NSX administrator also creates a Logical Switch Port (LSP) associated with the Logical Switch.

The LSP is associated with the data VLAN and the VTEP L2 gateway service configured for theBrocade VCS gateway earlier

7. NSX then pushes VNI-to-VLAN mapping to the VCS fabric. It also provides MAC addresses and theassociated VTEPs in the virtual infrastructure to the VCS fabric to instantiate the VXLAN gatewayfunctionality

8. VCS uses the VNI-to-VLAN mapping to bridge VXLAN and VLAN traffic. VCS also installs the MACaddresses and VTEP tunnels associated with the virtual network VTEPS in its MAC address table

9. VCS provides the MAC address associated with the mapped VLANs to NSX. NSX uses thisinformation to populate MAC address tables in the virtual infrastructure

Data flow in a Brocade VCS Gateway and VMware NSX based topologyFigure 8 is a sample deployment of Brocade VCS based VXLAN gateway for VMware NSX. Thissection illustrates the flow of data traffic between a virtual machine (VM-A) and a physical server(Host-B).

Data flow in a Brocade VCS Gateway and VMware NSX based topology


FIGURE 8 Sample topology illustrating data traffic flow with Brocade VCS Gateway and VMware NSX

In Figure 8 , VLAN 10 is used as Transport VLAN to carry the tunnels between Host-A, NSX ServiceNode and the Brocade VCS Gateway fabric. The communication between NSX components (NSXManager, NSX Controller and NSX Service Node - shown in dark blue), Hypervisor vSwitch (Host-A -shown in gray) and VXLAN Gateway node (Brocade VCS based VXLAN Gateway- shown in blue) takesplace via Management LAN (shown in red). Service Node requires connectivity over the data plane tothe Brocade VCS Gateway fabric and the hypervisor Host-A to establish tunnels. However, the Servicenode does not need to be in the same VLAN as Host-A and the VCS fabric.

This example assumes that NSX Controller, NSX Manager and NSX Service Nodes have been installedand configured. Also, Brocade VCS Gateway Services are configured for data VLAN 30 andconnectivity to the physical server over physical edge ports are in the data VLAN 30. These proceduresare described in the Configuration section.

When VM-A sends traffic, it will be encapsulated by Host-A in a VXLAN header (with correspondingVNI) and sent out on tunnel established over VLAN 10 to the VCS Gateway. Tunnels are establishedbetween Service node, Brocade VCS Gateway fabric and all the hypervisor vSwitches. In the abovefigure, 33.32.31.x network is used for originating/terminating the VTEP tunnels.

Using NSX controller, VLAN 30 is mapped with a VNI. Whenever a packet arrives over VLAN 30 onHost-A, the corresponding VNI mapping is included in the VXLAN header and the packet is sent out ofthe corresponding tunnel.

The Brocade VCS Gateway fabric upon receiving the VXLAN packet, strips off the VXLAN header andforwards the packet over physical port belonging to VLAN 30. When a packet is received from thephysical server (destined to VM-A), the gateway adds a VXLAN header and sends it over the tunnel toHost-A. Host-A removes VXLAN header and forwards the packet to VM-A.

Background


Deployment

● Supported Brocade Hardware and Software................................................................ 16● VMware NSX requirements...........................................................................................16● Deployment Topologies................................................................................................ 17● Limitations and Restrictions.......................................................................................... 19

The major building blocks in deploying the Brocade VCS VXLAN gateway for VMware NSX solutionare -

1. VXLAN gateway for VMware NSX-capable Brocade VDX switches (See Supported Hardware andSoftware)

2. VMware NSX Components

a. NSX Controllerb. NSX Service Nodec. NSX Managerd. OVS in Hypervisors

Supported Brocade Hardware and SoftwareThe following are the VDX switches that support the VXLAN gateway:

• Brocade VDX 6740• Brocade VDX 6740(T)• Brocade VDX 6740(T)-1G• Brocade VDX 6940• Brocade VDX 2740

The VXLAN gateway is supported by Brocade NOS Release 4.1.2 and later. The latest recommendedsoftware release at the time of this publication is NOS 5.0.1a.

The following hypervisors are supported and verified with the VMware NSX and Brocade VCS VXLANgateway solution: Xen 6.2, VMware ESXi 5.5 with NSX vSwitch, KVM version 1.0 on Ubuntu 12.04.

VMware NSX requirementsThe required VMware NSX components include the following:

• NSX Controller• NSX Service Node• NSX Manager

The deployments described in this document are verified using VMware NSX version 4.2.0.

Deployment


Deployment TopologiesThe following are the common VCS fabric topologies for implementing Brocade VCS based L2 VXLANgateway for NSX:

• Top of Rack (ToR)• One-arm gateway

Top of Rack (ToR) TopologyIn this topology, the Brocade VCS VXLAN gateway is positioned at the Top of Rack and the VXLAN(virtual) and VLAN (physical) servers are directly connected to the Brocade VCS gateway fabric. Thegateway terminates the tunnels for the VTEP endpoints in the virtual network and forwards the trafficdestined to the physical VLAN based server. When the VCS fabric receives VLAN traffic from the Layer2 physical server destined for the VXLAN server, the gateway encapsulates the packets and forwardsthe traffic to VTEP endpoints in the virtual network.

FIGURE 9 ToR Topology

In this example, after configuring the hypervisor vSwitch, service node, NSX Controller, and NSXManager, a VXLAN tunnel is established from Host-A to the Brocade VCS gateway fabric between theVRRP virtual IP address (33.32.31.13) of the VCS fabric and the VTEP IP of the hypervisor vSwitch(33.32.31.1). (The procedure to configure the NSX components and the Brocade VCS gateway fabric isdescribed in Configuration section of this document).

If the Guest OS running on the hypervisor sends traffic destined to Host-B, it gets VXLAN encapsulatedand reaches the VCS fabric through the VXLAN tunnel, where the traffic is forwarded to the destinationon the physical network. In the same way, traffic destined to a VM located on the hypervisor reachesthe gateway using normal Layer 2 forwarding, and the gateway forwards the traffic to the hypervisorusing the established VXLAN tunnel.

Deployment Topologies


The hypervisor vSwitches and the VCS gateway forward broadcast, unicast, and multicast (BUM)traffic to the service node, using the VXLAN tunnels established between them.

One-Arm Gateway TopologyIn a One-Arm Gateway topology, the VXLAN L2 gateway functionality is performed in a separate VCSfabric. The topology is used to provide VXLAN gateway functionality in an existing deployment.

As shown in Figure 10 , the access and aggregation fabrics forward Layer 2 traffic, for the data VLAN(VLAN 30) and the transport VLAN (VLAN 10) between the server infrastructure (both physical andvirtual) and the one-arm VCS fabric will perform the L2 VXLAN gateway functionality.

Specific to this topology, the Gateway fabric is connected to the VCS Access-Aggregation fabric. TheAccess-Aggregation fabric carries both VLAN 10 and VLAN 30 traffic. Both VXLAN and regular trafficis handed over by the Access-Aggregation fabric to the Gateway fabric for bridging.

FIGURE 10 One-Arm Gateway Topology

After configuring the hypervisor vSwitch, service node, NSX Controller, and NSX Manager, a VXLANtunnel is established from Host-A (Hypervisor) to the VCS gateway fabric between the VRRP virtual IPaddress (33.32.31.13) and the VTEP IP of Host-A vSwitch (33.32.31.1). After the tunnel is established,Host-A sends VXLAN encapsulated traffic destined to the physical server (Host-B in Figure 10)through the VXLAN tunnel up to the VCS gateway fabric, where the VXLAN traffic is decapsulated andbridged to VLAN 30. This traffic in VLAN 30 is switched back to the Layer 2 server through theAccess-Aggregation VCS fabrics. In the same way, traffic destined to VM1 from Host-B reaches thegateway fabric using normal Layer2 forwarding and the gateway fabrics encapsulates the traffic inVXLAN and forwards the traffic to Host-A over the established Tunnel.

One-Arm Gateway Topology


Limitations and RestrictionsThe following are some of the limitations to keep in mind when deploying NSX VXLAN gateway withBrocade VCS (as of NOS 5.0.1a):

• VXLAN functionality is currently supported on VDX 6940, 6740, 6740-T and VDX 2740.• Management of Brocade VCS VXLAN gateway requires VMware NSX.• VXLAN Gateway functionality is only supported in Logical Chassis mode.• A maximum of 4 RBridges are supported in a VXLAN enabled VCS Cluster. VXLAN Gateway should

be enabled on all the RBridges of the VCS Cluster.• Only 1 VTEP Gateway is supported in a VXLAN enabled VCS Cluster.• Only one VLAN to VNI mapping is allowed.• VXLAN GW for VMware NSX and VF Extension cannot be enabled in the same VCS fabric.• Service and Transport VF cannot be attached to VXLAN GW.• When multiple VMware NSX Service Nodes are setup, only one node will be used for handling BUM

traffic. If the reachability to that service node fails, another service node will not be picked, leading toloss of BUM traffic.

• Tunnel interfaces cannot be used as SPAN (Switch port Analyzer) destination.• Only Ingress ACL can be applied on tunnels.• Ingress/Egress QoS policies cannot be applied to tunnels.• Unicast/Multicast routing between VXLAN and VLAN/VXLAN is not supported.

Please refer to release notes for additional limitations and scale numbers.

Limitations and Restrictions


Configuration

● Topology used in Configuration.................................................................................... 20● Configuration Steps for the VMware NSX solution with VCS VXLAN Gateway............21

Topology used in ConfigurationThis section illustrates the sample One-Arm topology used to describe the configuration. Details of thistopology are described in the Deployment Topologies section of this document.

FIGURE 11 One-arm Topology used in Configuration

The sample deployment provides an Access-Aggregation fabric with four leaf VDX switches, and twospine switches, and a pair of VDX 6740(T) switches acting as a one-arm gateway to integrate the VCSfabric with the VXLAN overlay network.

NOTEVXLAN gateway functionality is only supported in Logical Chassis mode.

The following platforms are used in the sample topology:

• VXLAN gateway fabric:

Configuration


‐ RB239: GW1, VDX-6740(T)‐ RB240: GW2, VDX-6740(T)

• Spine switches:

‐ RB23: AGGR1-8770-1, VDX-8770‐ RB100: AGGR1-8770-2, VDX-8770

• Leaf switches:

‐ RB201: ACC1-6720-1, VDX-6720‐ RB110: ACC1-6720-1, VDX-6720‐ RB3: ACC2-6720-2, VDX-6720‐ RB4: ACC2-6720-1, VDX-6720

The major components for this deployment include the NSX components from VMware and VDX 6740switch, which provides the VXLAN gateway functionality. The following sections describe theconfiguration and verification steps in detail.

NOTEThe example deployment was tested using NSX version 4.2.0. This section illustrates a sample NSXdeployment which has been tested. Please refer to VMware NSX documentation for specific detailsabout NSX configurations.

Configuration Steps for the VMware NSX solution with VCS VXLANGateway

This section details the configuration of the One-Arm Gateway topology, illustrated in Figure 11 on page20. The configuration is generally independent of the type of deployment and can be applied, withadjustments for the topology, for the other deployment topologies.

Set up NSX Controller ClusterThe NSX Controller is a server running a Linux-based ISO image, which can be obtained from VMWare.The procedure here shows configuration of one controller only, but multiple controllers for a cluster canbe configured. Refer VMware documentation for details. Once ISO image is installed on the server,follow the below procedure to bring up the service node.

1. Set up the hostname.2. Set the IP address for the eth1 connected to the transport network.3. Specify additional cluster information.4. Add the NSX Controller to the controller cluster.

Steps 1 to 4 are illustrated below:Node# set hostname NSXController1 -> Name of the controllerNSXController1#set network interface breth0 ip config static 10.18.124.194 255.255.252.0 -> Management IPNSXController1# show network interface -> verify the config is acceptedInterface Address/Netmask MTU Admin-Status Link-Status breth0 10.18.124.194/22 1500 UP UP eth1 1500 UP UP eth0 1500 UP UP NSXController1#add network default-route <default-route IP> NSXController1#add network dns-server <IP>NSXController1#add network ntp-server <NTP-server-IP>NSXController1#set control-cluster management-address 10.18.124.194

Configuration Steps for the VMware NSX solution with VCS VXLAN Gateway


NSXController1#set control-cluster role switch_manager listen-ip 10.18.124.194NSXController1#set control-cluster role api_provider listen-ip 10.18.124.194NSXController1#join control-cluster 10.18.124.194 -> Master Controller IPNSXController1#show control-cluster status

Set up the Service NodeA service node is a server running a Linux-based ISO image, which can be obtained from VMWare.Once ISO image is installed on the server, follow the below procedure to bring up the service node.

Step 1: Set the hostname.

Step 2: Set the IP address for eth1 connected to the transport network.

Step 3: Add to the controller cluster (switch manager).

Step 4: Add the additional server information.

Steps 1 to 4 are illustrated below:Node# set hostname ServiceNode1 -> Name of the service nodeServiceNode1#set network interface breth0 ip config static 10.18.124.196 255.255.252.0 -> Management IPServiceNode1#set network interface breth1 ip config static 33.32.31.10 255.255.255.240 -> VTEP Endpoint ServiceNode1#add switch manager 10.18.124.194 -> Controller IPServiceNode1#show network interfaces -> verify the config is acceptedServiceNode1 # show network interfaceInterface Address/Netmask MTU Admin-Status Link-Status breth1 33.32.31.10/28 1500 UP UP breth0 10.18.124.196/22 1500 UP UP eth1 1500 UP UP eth0 1500 UP UP ServiceNode1#add network default-route <default-route IP> ServiceNode1#add network dns-server <IP>ServiceNode1#add network ntp-server <NTP-server-IP>ServiceNode1#show switch certificate

Step 5: Copy the certificate from the output of the show switch certificate command and saveit.

The certificate is used when adding the service node to the NSX Manager.

Brocade VCS VXLAN Gateway ConfigurationThis section describes how to configure the NSX controller connection properties for implementingNSX gateway functionality in a VCS fabric using a VDX 6740(T) as the gateway. The configurationsupports multiple named NSX controller connection profiles.

This example for the GW1 switch includes VRRP-E configuration, which has not changed since earlierNOS releases. The distributed overlay gateway functionality depends on VRRP-E for multi homing. Bydefault, the VRRP-E virtual MAC is derived based on the following value:02:e0:52:<2-byte-ip-hash>:<1-byte-vrid>The NSX gateway functionality requires the virtual MAC to be a function of only the VRID. The 2-bytehash of the virtual IP should be set to zero (0), and the virtual MAC should be like the following:02:e0:52:00:00:<1-byte-vrid>A new configuration has been introduced under VRRP-E group to enable this gateway compatibilitymode (described in Step 2 of this section).

Set up the Service Node


Procedure SummaryThe following is a summary of the steps required for the gateway configuration:

Step 1: Configure VCS fabric virtual IP address.

Step 2: Configure VRRP-E.

Step 3: Generate and save the switch certificate.

Step 4: Configure the VXLAN gateway.

Step 5: Configure the NSX controller information on the VCS VXLAN gateway.

The following are the detailed procedures and examples.

Detailed ProcedureStep 1: Configure VCS fabric virtual IP address.

Configure the VCS fabric virtual IP address on the NSX gateway for establishing the connection with theNSX controller:GW1(config)# vcs virtual ip address <IPv4ADDRESS>This address should be in the same subnet as the management IP address.

Step 2: Configure VRRP-E.

Configure VRRP-E onthe VXLAN gateway fabric RBridges.rbridge-id 239 protocol vrrp-extended interface Ve 10 ip proxy-arp ip arp-aging-timeout 0 ip address 33.32.31.11/28 no shutdown vrrp-extended-group 110 virtual-mac 02e0.5200.00xx <-Specify the vMac address for VRRP-E virtual-ip 33.32.31.13 <-VRRP-Virtual IP enable no preempt-mode short-path-forwarding !rbridge-id 240 protocol vrrp-extended interface Ve 10 ip proxy-arp ip arp-aging-timeout 0 ip address 33.32.31.12/28 no shutdown vrrp-extended-group 110 virtual-mac 02e0.5200.00xx <-Specify the vMac address for VRRP-E virtual-ip 33.32.31.13 <-VRRP-Virtual IP enable no preempt-mode short-path-forwardingStep 3: Generate and save the switch certificate.

Generate the switch certificate using the following command:GW1#nsx-controller client-cert generateVerify certificate generation using the following command:GW1#show nsx-controller client-certCopy and save the certificate to be used when setting up the NSX Manager.

Step 4: Configure the VXLAN gateway.

Configuration


To set up a VXLAN gateway, configure the source IP addresses of the tunnels, which are not userconfigurable. In this example, VLAN 30 is mapped to the VNI on the NSX controller.[no] overlay-gateway <name> [no] attach rbridge-id {add|remove} <rb-range> [no] ip interface ve <id> vrrp-extended-group <gid> [no] attach vlan <vid> [mac <mac>] ! multiple of such lines [no] enable statistics direction {tx|rx|both} vlan {add|remove} <vid-range> [no] monitor session <sid> remote-endpoint {<addr>|any} direction {tx|rx|both} vlan {add|remove} <vid-range> [no] activate!Configure the VDX switch (GW1) using the following commands:sw0#show running-config overlay-gatewayoverlay-gateway demoattach rbridge-id add 239, 240attach vlan 30ip interface Ve 10 vrrp-extended-group 10 activateStep 5: Configure the NSX controller information on the VCS VXLAN gateway.

To configure the NSX controller information in VCS, enter the following commands:[no] nsx-controller <name> ip address <address> [ port <port> ] [ method <method> ] [no] reconnect-interval <secs> [no] activateConfigure the VCS fabric using the following commands:GW1# show running-config nsx-controller nsx-controller demo ip address 10.18.124.194 reconnect-interval 20 activate

Hypervisor (KVM) ConfigurationThis section illustrates the steps to setup a KVM Hypervisor based server, with OVS on the hypervisorbeing managed by NSX. The KVM hypervisor used in this case runs on Ubuntu 12.04 along withlibvirt, and Open vSwitch package from VMware. Xen or VMware ESXi hypervisor can also be used inplace of the KVM hypervisor, but necessary NSX packages are required to enable the VXLANfunctionality.

Procedure SummaryThe following is a summary of the steps required for the gateway configuration:

Step 1: Install Ubuntu OS 12.04 on bare metal server.

Step 2: Install KVM/libvirt/virtinst/virt-manager packages in Ubuntu for managing the virtualmachine and network.

Step 3: Install OVS packages from VMware in Ubuntu.

Step 4: Bring up the Guest VM and attach the Guest VM to the OVS integration bridge.

Step 5: Create the appropriate bridges and attach the eth interfaces in OVS.

Step 6: Configure OVS on the KVM Hypervisor.

The following are the detailed procedures and examples.

Detailed ProcedureStep 1: Install Ubuntu OS 12.04 on bare metal server.

Hypervisor (KVM) Configuration


Install Ubuntu 12.04 on the bare metal server to work as the host OS.

Step 2: Install KVM/libvirt/virtinst/virt-manager packages in Ubuntu for managing the virtualmachine and network.

• Install KVM in Ubuntu (refer KVM documentation for installation notes)• The KVM version used for the deployment here is as follows:

root@ubuntu12_124175:~# kvm --versionQEMU emulator version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 Fabrice Bellard

• Install libvirt/virtinst/virt-manager and all dependencies on Ubuntu server• The libvirt package version used for the deployment here is as follows:

root@ubuntu12_124175:~# libvirtd --version libvirtd (libvirt) 0.10.1

Step 3: Install OVS packages from VMware in Ubuntu.

• Download and install the OVS archive from VMware to the Ubuntu server (refer VMwaredocumentation for details on installation of OVS)

• The OVS package version used for the deployment here is as follows:root@ubuntu12_124175:~# ovs-vsctl --versionovs-vsctl (Open vSwitch) 2.0.0.3017

Step 4: Bring up the Guest VM and attach the Guest VM to the OVS integration bridge.

• In order to complete this step, KVM and libvirt packages must be installed in the Ubuntu server.• Open the "virt-manager" and add a guest VM and attach it to the default Network.• Edit the Domain XML file and attach the VM to the integration bridge "br-int" in OVS:

root@ubuntu12_124175:~#virsh edit “vmName”• Find the XML section for each interface the VM has, as "network" parameters shown below:

...<interface type='network'> <mac address='52:54:00:71:b1:b6'/> <source network='default'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/></interface>...

• Change the parameters as shown above to the values shown below. This step attaches the GuestVM to the 'br-int' bridge on OVS:...<interface type='bridge'> <mac address='52:54:00:71:b1:b6'/> <source bridge=’br-int’/> <virtualport type='openvswitch'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/></interface>...

• Save the xml file and power up the VM. If this is done properly you should see VM vnet interfacesgetting attached to the OVS in "ovs-vsctl show":root@ubuntu12_124175:~#ovs-vsctl showBridge br-int Port br-int Interface br-int type: internal Port vnet0 Interface vnet0ovs_version: “1.11.0.24839”

Step 5: Create the appropriate bridges and attach the eth interfaces in OVS.

• Use the following commands to create a bridge and add a port:ovs-vsctl add-br <bridgeName> ovs-vsctl add-port <bridgeName> <portName>

• Enter the following commands:root@ubuntu12_124175:~#ovs-vsctl add-br br0 root@ubuntu12_124175:~#ovs-vsctl add-port br0 eth0root@ubuntu12_124175:~#ovs-vsctl add-br br1root@ubuntu12_124175:~#ovs-vsctl add-port br1 eth1

Configuration


• The following command displays the interfaces and bridges associated with them:root@ubuntu12_124175:~#ovs-vsctl showBridge “br1”Port “br1”Interface “br1”type: internalPort “eth1”Interface “eth1”Bridge “br0”Port “br0”Interface “br0”type: internalPort “eth0”Interface “br0”type: internalovs_version: “1.11.0.24839”

• Assign the appropriate IP addresses to the bridge interfaces by entering the following commands:root@ubuntu12_124175:~#ifconfig eth0 0 && ifconfig br0 10.18.124.174 netmask 255.255.255.0root@ubuntu12_124175:~#ifconfig eth1 0 && ifconfig br1 33.32.31.1 netmask 255.255.255.0

• 10.18.124.174 on br0 is the management IP address and 33.32.31.1 on br1 is used as VTEP IP forthe hypervisor.

• Assign default gateway address:root@ubuntu12_124175:~#route add default gw <ip-address>

Step 6: Configure OVS on the KVM Hypervisor.

• Create OVS certificates for NSX Controller authentication• Specify the NSX Controller IP address in OVS

root@ubuntu12_124175:~#ovs-vsctl get-sslroot@ubuntu12_124175:~#ovs-vsctl set-manager ssl: 10.18.124.194

NSX Manager ConfigurationConfiguring NSX Manager requires creating the transport components, which include the following:

• Step 1: Add a controller cluster from NSX Manager• Step 2: Add a Transport Zone• Step 3: Add the Service Node• Step 4: Add the Hypervisor• Step 5: Add the Brocade VCS Gateway• Step 6: Add Gateway Services• Step 7: Create Logical Switch• Step 8: Configure Logical Switch Ports

The following provides details about the configuration with examples.

Detailed ProcedureStep 1: Add a controller cluster from NSX Manager

• After the initial login, NSX Manager indicates that it is currently not connected to any cluster. (See Figure 12 )

• Click 'Add Cluster' from Admin > Clusters to add the NSX cluster to the NSX Manager.

NSX Manager Configuration


FIGURE 12 Controller Cluster Administration

• Specify the NSX Controller IP address of the Master NSX Controller, username and password. Click'Connect'

FIGURE 13 Specify NSX Controller Cluster Information

• Give a Name to the Controller Cluster and click 'Save'

Configuration


FIGURE 14 Add name to the NSX Controller Cluster

Step 2: Add a Transport Zone

In VMware NSX, a transport zone corresponds to the underlying physical network used to interconnecttransport nodes (hypervisors, service nodes, and gateways). A simple VMware NSX deployment has asingle transport zone that represents the physical network connectivity within the data center. This

Configuration


example uses a single transport zone because all transport nodes connect to the same underlyingnetwork.

• On the NSX Manager Dashboard, in the Summary of Transport Components section, click Add onthe Zones row. (See Figure 15 )

FIGURE 15 Add a Transport Zone

• After adding the transport zone, verify the Zone from the Network Components tab (See Figure 16 )

FIGURE 16 Verify the added Transport Zone

Step 3: Add the Service Node

1. To add the service node to the NSX manager, on the Dashboard (see Figure 17 ), in the Summary ofTransport Components Section, click Add on the Service Node row.

• To see the Add button in this row, move the slider bar to the right, or click Expand Panel.

Configuration


FIGURE 17 Add Service Node

2. In the Create Service Node dialog box (Figure 18 ), verify that the type is Service Node and clickNext.

FIGURE 18 Specify Service Node Type

3. TypeServiceNode1in the Display Name field and click Next.

Configuration


FIGURE 19 Specify the Display Name of the Service Node

4. Leave the other fields with the default values, as listed below (Figure 20 ):

• Disabled: Management Rendezvous Server• Enabled: Admin Status and Tunnel Keep-Alive Spray

FIGURE 20 Specify Service Node properties

5. Verify that the Credential Type is Security Certificate.

Configuration


FIGURE 21 Specify the Security Certificate

6. Copy the SSL certificate from the NSXCLI of the service node, by entering the following commandfrom the NSX service node system prompt:ServiceNode1 # show switch certificate

7. Paste the SSL certificate into the Security Certificate Text box (Figure 22 ); then click Next.

• This is the certificate referred to in the "Set up the Service Node" section.

NOTEMake sure to include the -- Begin Certificate -- and -- End Certificate - Lines.

8. To add a transport connector, Click Add Connector, and complete the following sub-steps:

FIGURE 22 Add a transport connector

Configuration


• Choose VXLAN as the Transport Type.• Choose the Transport Zone, Zone1, from the pull-down list.• Add the IP address on the transport:

Network, 33.32.31.10 <= This IP will be the tunnel end point of the Service Node.

• Click OK.

FIGURE 23 Specify Transport Connector properties

• To register the service node, Click Save & View.• Verify the service node status.

FIGURE 24 View the Service Node status

Step 4: Add the Hypervisor

1. In the Summary of Transport Components section, click Add on the Hypervisors row. (See Figure25 )

Configuration


FIGURE 25 Add Hypervisor

2. Type 'br-int' in the Integration Bridge Id field; then click Next3. Copy the SSL certificate from the KVM Hypervisor, by entering the following command in the

Ubuntu server (see Section 4.2.4):root@ubuntu12_124175:~# cat /etc/openvswitch/ovsclient-cert.pem

4. Paste the SSL certificate into the Security Certificate Textbox; then click Next5. Click Add Connector; then choose the VXLAN as the Transport Type

FIGURE 26 Create Transport Connector

6. Choose the Transport Zone - Zone1, from the pull-down list and add the IP address on thetransport. This IP address will be the tunnel end point; then click OK

7. Click Hypervisors number under registered tab and expand the Transport node to see thecomponent status

Configuration


FIGURE 27 View Hypervisor status

• The components on the Status table are green when everything is configured correctly. (See Figure 27 )

8. Verify that the correct VXLAN Connector IP addresses are defined in the Transport Connectors table.(See Figure 28 )

FIGURE 28 Verify the Transport Connector for the Hypervisor

Step 5: Add the Brocade VCS Gateway

1. Select Gateway from the Transport Node Types selection list2. Enter the Name for the Gateway as 'Gateway2' in the Display Name field (Figure 29 )

FIGURE 29 Create Gateway and specify Transport Node Type

Configuration


• Make a note of the gateway name, which is required for other configuration steps.• Click Next. Set the next screen with default values.

FIGURE 30 Specify Gateway properties

3. Enable the VTEP Enabled checkbox (Figure 31 )

FIGURE 31 Specify Gateway properties

4. Copy and paste the Switch Public certificate to the Security Certificate field on the Credentials page.(Figure 32 ) Refer Step 3 of Section 4.2.3 for certificate details.

Configuration


FIGURE 32 Specify the Security Certificate for the Gateway

5. Select VXLAN as the Transport Type on the Edit Transport Connector page (Figure 33 ) and enterthe IP address of the VRRP-E, as configured on the switch

FIGURE 33 Edit Transport Connector for the Gateway

Step 6: Add Gateway Services

1. Enter a Name for the Gateway Service.

• Make a note of the gateway name, which is required for other configuration. In this example, it isset as 'VTEP-L2-DutA4'

Configuration


FIGURE 34 Create Gateway Service

2. Add Gateway device:

• In this example, choose Gateway2, which was configured in the previous section.• Enter the Port ID of a port that has been exported from the switch (Figure 35 ). Any VCS

Gateway port can be selected in this step, as Brocade VCS creates 1:1 mapping betweenVLANs and VNIs.

FIGURE 35 Specify the Gateway and the Port ID

Step 7: Create Logical Switch

1. Create a Logical Switch from the Dashboard. (Figure 36 )

Configuration


FIGURE 36 Create Logical Switch

2. Select VXLAN from the Transport Type selection list, and add the VNI, which is the unique VXLANnumber (Figure 37 )

FIGURE 37 Create Transport Zone Binding

Step 8: Configure Logical Switch Ports

1. Add a Logical Switch Port

• Click 'Add' against Switch Ports in NSX Manager.• Associate the Logical Switch Port with the Logical switch created in Step 7.

Configuration


FIGURE 38 Associate Logical Switch Port with Logical Switch

2. Select VTEP L2 Gateway from the Attachment Type selection list, select the correct L2 GatewayService (VTEP-L2-DutA4) configured earlier, and enter the data VLAN number (Figure 39 )

FIGURE 39 Edit Logical Switch Port

3. Create logical switch ports for the Guest VMs

• 'Add' Switch Ports in NSX Manager• Associate the Logical Switch Port with the Logical switch created in Step 7 (as shown in Step 1

above)• Select the Attachment Type as 'VIF'• Specify the Hypervisor from the dropdown (as configured in Step 4 - Add the Hypervisor of this

Section)

Configuration


• Select the VIF interface of the Guest VM created in Step 4.2.3 based on the Mac address of thevnet0 interface

• Leave all other options at the default settings

FIGURE 40 Attachment

4. Verify that the status is Up for the Link and Fabric (Figure 41 )

FIGURE 41 Status

• Make sure that the transport connector for the service node is VXLAN Connector. (Figure 42 )

FIGURE 42 Transport Connector

Configuration


Verification

This section provides the commands to use for verifying the configuration and operation of a VCSgateway fabric.

Verifying Controller StatusTo verify the controller status, use the following command:show nsx-controller [<name>]The following is an example of the show output on GW1:GW1# show nsx-controller name Controller1NSX controller cluster "Controller1"Seed IP address 10.18.124.194, port 6632, method SSLReconnect interval 20 secs, Max retries 100Admin state up, Number of connections 1Number of tunnels 6, Number of MACs 11Connection details: ID 5c137c9f-3a8d-40a4-a649-df5f3462856e, Connected IP address 10.18.124.194, port 6632, method SSL Reconnect interval 20000 millis, Number of retries 0 (max 100) Last connect time: Fri Jan 31 01:10:41 2014 Last disconnect time: Fri Jan 31 01:10:01 2014

Verifying Gateway StatusTo verify gateway status, enter the following command:show overlay-gateway [name <name>] {brief|stats|vlan stats}The following is an example of the show output on GW1:GW1# show overlay-gateway Overlay Gateway "VXLAN", ID 1, rbridge-ids 239-240Admin state upIP address 33.32.31.13 ( ve10, Vrid 110 ), Vrf default-vrfNumber of tunnels 6Packet count: RX 17909 TX 1247Byte count : RX (NA) TX 356626

Verifying Tunnel StatusTo verify tunnel status, enter the following command:show tunnel [ mode <mode> | overlay-gateway <name> | src-ip <src-addr> | dst-ip <dst-addr> ] briefThe following is an example of the show output on GW1:

Tunnel 1 in the following output has source IP as the VRRP-E virtual IP and destination IP as theService node IP address. Tunnel 2 has same source as Tunnel 1, but destination is Hypervisorrunning KVM.GW1# show tunnel brief Tunnel 1, mode VXLAN, rbridge-ids 239-240Admin state up, Oper state upSource IP 33.32.31.13, Vrf default-vrfDestination IP 33.32.31.10 Tunnel 2, mode VXLAN, rbridge-ids 239-240Admin state up, Oper state up

Verification


Source IP 33.32.31.13, Vrf default-vrfDestination IP 33.32.31.1

Viewing Tunnel DetailsTo view tunnel details, enter the following command:Show tunnel <id>The following is an example of the show output on GW1:GW1# show tunnel 1Tunnel 1, mode VXLAN, rbridge-ids 239-240Ifindex 2080374798, Admin state up, Oper state upOverlay gateway "VXLAN", ID 1Source IP 33.32.31.13 ( ve10, Vrid 110 ), Vrf default-vrfDestination IP 33.32.31.10Active next hop on rbridge 239: IP: 33.32.31.10, Vrf: default-vrf Egress L3 port: Ve 10, Outer SMAC: 0027.f886.bb36 Outer DMAC: e41f.1343.97d2 Egress L2 Port: Po 11, Outer ctag: 10 BUM forwarder: noActive next hop on rbridge 240: IP: 33.32.31.10, Vrf: default-vrf Egress L3 port: Ve 10, Outer SMAC: 0027.f880.f218 Outer DMAC: e41f.1343.97d2 Egress L2 Port: Po 11, Outer ctag: 10 BUM forwarder: yesPacket count: RX 18492 TX 1242Byte count : RX (NA) TX 356307

Verifying VNI MappingTo view verify VNI mapping, enter the following command:GW1# show vlan briefTotal Number of VLANs configured : 5 Total Number of VLANs provisioned : 5 Total Number of VLANs unprovisioned : 0 VLAN Name State Ports Classification (F)-FCoE (u)-Untagged, (t)-Tagged(R)-RSPAN (c)-Converged(T)-TRANSPARENT ================ =============== ========================== ============================= ===========================1 default ACTIVE Po 11(t) Te 239/0/44(t) 10 VLAN0010 ACTIVE Po 11(t) Te 239/0/22(u) 30 VLAN0030 ACTIVE Po 11(t) Tu 1(t) vni 100 Tu 2(t) vni 100 40 VLAN0040 INACTIVE(no member port) 1002(F) VLAN1002 INACTIVE(no member port)

Verification


Show MAC on GatewayFC1_ACC4# show macVlanId Mac-address Type State Ports30 5254.0021.914e Static Active Tu 1 <- Hypervisor MAC10 0005.3326.571e Dynamic Active Po 1110 0027.f880.f218 System Remote XX 239/X/X10 0050.5685.11a0 Dynamic Active Po 1110 0050.5685.5668 Dynamic Active Po 1110 0050.568d.16e1 Dynamic Active Po 1110 0050.568d.7b47 Dynamic Active Po 1110 02e0.5200.006e System Active XX 238/X/X10 e41f.1343.97d2 Dynamic Active Po 1110 e41f.1344.8a96 Dynamic Active Po 1130 001c.23d4.862e Dynamic Active Po 11

Viewing NSX StatisticsUse the following show command to view NSX tunnel statistics:show tunnel statisticsThe following is an example of the output on GW1:GW1# show tunnel statistics Tnl ID RX packets TX packets RX bytes TX bytes======== =============== =============== =============== ================1 18573 1242 (NA) 3563072 2 1 (NA) 114Use the following command to show VXLAN gateway statistics:show overlay-gateway [name <name>] statisticsThe following is an example of the output on GW1:GW1# show overlay-gateway name VXLAN statistics Gateway Name RX packets TX packets RX bytes TX bytes==================== =============== =============== =============== ========VXLAN 18000 1247 (NA) 356626Use the following command to show gateway statistics for a specific VLAN:show overlay-gateway <name> vlan statisticsThe following is an example of the output on GW1:GW1# show overlay-gateway name VXLAN vlan statistics VLAN ID RX packets TX packets========= ==================== ====================30 18065 19300

Clearing CountersThe clear counters all commands also clears VXLAN tunnel counters. The following clear commandsclear statistics on the VXLAN gateway.clear overlay-gateway <name>clear overlay-gateway <name> vlan <vid-range>These commands are directed to a specific VXLAN gateway instance, and clears the counters for alltunnels associated with that gateway. The operation can be applied on all the gateway RBridges in thecluster.

Verification
