brocade-ip-fabric-bvd-published

BROCADE VALIDATED DESIGN

Brocade IP Fabric and Network Virtualizationwith BGP EVPN

53-1004308-0312 August 2016

© 2016, Brocade Communications Systems, Inc. All Rights Reserved.

Brocade, Brocade Assurance, the B-wing symbol, ClearLink, DCX, Fabric OS, HyperEdge, ICX, MLX, MyBrocade, OpenScript, VCS, VDX, Vplane, andVyatta are registered trademarks, and Fabric Vision is a trademark of Brocade Communications Systems, Inc., in the United States and/or in othercountries. Other brands, products, or service names mentioned may be trademarks of others.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment,equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, withoutnotice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocadesales office for information on feature and product availability. Export of technical data contained in this document may require an export license from theUnited States government.

The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to the accuracy of thisdocument or any loss, cost, liability, or damages arising from the information contained herein or the computer programs that accompany it.

The product described by this document may contain open source software covered by the GNU General Public License or other open source licenseagreements. To find out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, andobtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.

Brocade IP Fabric and Network Virtualization with BGP EVPN2 53-1004308-03

http://www.brocade.com/support/oscd

ContentsList of Figures........................................................................................................................................................................................................................................................................... 5

Preface...........................................................................................................................................................................................................................................................................................7Brocade Validated Designs..................................................................................................................................................................................................................................... 7Purpose of This Document..................................................................................................................................................................................................................................... 7Target Audience..............................................................................................................................................................................................................................................................7About the Authors.........................................................................................................................................................................................................................................................7Document History........................................................................................................................................................................................................................................................8About Brocade............................................................................................................................................................................................................................................................... 8

Introduction ............................................................................................................................................................................................................................................................................... 9

Brocade IP Fabric Technology Overview.................................................................................................................................................................................................................. 11Benefits............................................................................................................................................................................................................................................................................... 11Terminology......................................................................................................................................................................................................................................................................11Functional Components of Brocade IP Fabric...........................................................................................................................................................................................12

Leaf-Spine Layer 3 Clos Topology (Two-Tier).................................................................................................................................................................................12Optimized 5-Stage Layer 3 Clos Topology (Three-Tier)...........................................................................................................................................................14Edge Services and Border Leafs............................................................................................................................................................................................................. 15Brocade IP Fabric Underlay Routing..................................................................................................................................................................................................... 15

Network Virtualization with BGP EVPN................................................................................................................................................................................................................... 19VXLAN Layer 2 Extension Using Flood and Learn..............................................................................................................................................................................20BGP EVPN for VXLAN..........................................................................................................................................................................................................................................22

VTEP...................................................................................................................................................................................................................................................................... 23Static Anycast Gateway................................................................................................................................................................................................................................23Overlay Gateway..............................................................................................................................................................................................................................................23BGP EVPN Control Plane..........................................................................................................................................................................................................................24ARP Suppression............................................................................................................................................................................................................................................ 25VLAN Scoping..................................................................................................................................................................................................................................................26Conversational Learning.............................................................................................................................................................................................................................. 27Integrated Routing and Bridging............................................................................................................................................................................................................ 28Multitenancy....................................................................................................................................................................................................................................................... 29Ingress Replication......................................................................................................................................................................................................................................... 30vLAG Pair............................................................................................................................................................................................................................................................ 30

IP Fabric Validated Designs...........................................................................................................................................................................................................................................33Pervasive eBGP......................................................................................................................................................................................................................................................... 33iBGP Within a PoD and eBGP Between PoDs........................................................................................................................................................................................34Hardware and Software Matrix...........................................................................................................................................................................................................................35Brocade IP Fabric Configuration...................................................................................................................................................................................................................... 35

Node ID Configuration................................................................................................................................................................................................................................. 35IP Fabric Infrastructure Links.....................................................................................................................................................................................................................37Loopback Interfaces.......................................................................................................................................................................................................................................37Server-Facing Links.......................................................................................................................................................................................................................................38Deployment Model-1: eBGP Underlay Configuration for Optimized 5-Stage Clos Fabric................................................................................. 40Deployment Model-1: eBGP Underlay Configuration for 3-Stage Clos Fabric.......................................................................................................... 46Deployment Model-2: iBGP Underlay Configuration for Optimized 5-Stage Clos Fabric................................................................................... 51

Network Virtualization with BGP EVPN........................................................................................................................................................................................................57

Brocade IP Fabric and Network Virtualization with BGP EVPN53-1004308-03 3

Overlay Gateway Configuration...............................................................................................................................................................................................................57Deployment Model-1: eBGP EVPN Configuration for Optimized 5-Stage Clos Fabric........................................................................................ 57Deployment Model-1: eBGP EVPN Configuration for 3-Stage Clos Fabric.................................................................................................................67Deployment Model-2: iBGP EVPN Configuration for Optimized 5-Stage Clos Fabric........................................................................................72Tenant Provisioning........................................................................................................................................................................................................................................83vLAG Pair Configuration..............................................................................................................................................................................................................................87

Illustration Examples................................................................................................................................................................................................................................................ 87Example-1: Tenant and L2 Extension Between Racks in a 3-Stage Clos Fabric........................................................................................................87Example-2: Tenant and L2 Extension Between PoDs in an Optimized 5-Stage Clos Fabric........................................................................... 101Example-3: Tenant Extension Outside the Fabric........................................................................................................................................................................ 116Example-4: VLAN Scoping at the ToR Level................................................................................................................................................................................126Example-5: VLAN Scoping at the Port Level Within a ToR..................................................................................................................................................135Example-6: Route Leaking for the Service VRF..........................................................................................................................................................................144

Design Considerations.................................................................................................................................................................................................................................................... 163

Appendix—Configuration of the Nodes................................................................................................................................................................................................................. 167vLAG Active/Active Pair Leaf............................................................................................................................................................................................................................ 167Individual Non-Redundant Leaf.......................................................................................................................................................................................................................173Spine Designated to Exchange Only Underlay Routes...................................................................................................................................................................... 177Spine Designated to Exchange Both Underlay and Overlay Routes......................................................................................................................................... 179Super-Spine Designated to Exchange Only Underlay Routes....................................................................................................................................................... 181Super-Spine Designated to Exchange Both Underlay and Overlay Routes......................................................................................................................... 183Edge Leaf..................................................................................................................................................................................................................................................................... 185

References............................................................................................................................................................................................................................................................................. 189


List of FiguresFigure 1 on page 14—Leaf-Spine L3 Clos Topology

Figure 2 on page 15—Optimized 5-Stage L3 Clos Topology

Figure 3 on page 17—eBGP for Underlay

Figure 4 on page 18—iBGP for Underlay

Figure 5 on page 20—VTEPs and L2 Extension with Flood and Learn

Figure 6 on page 22—Routing Between VXLANs in a Flood-and-Learn Topology

Figure 7 on page 23—VTEPs and L2 Extension with the BGP EVPN Control Plane

Figure 8 on page 26—ARP Suppression

Figure 9 on page 27—VLAN Scoping at the Leaf Level

Figure 10 on page 27—VLAN Scoping at the Port Level Within a ToR

Figure 11 on page 28—Asymmetric IRB

Figure 12 on page 29—Symmetric IRB

Figure 13 on page 30—Multitenancy

Figure 14 on page 31—Active-Active vLAG

Figure 15 on page 33—Pervasive eBGP in an Optimized 5-Stage IP Fabric

Figure 16 on page 34—Pervasive eBGP in a 3-Stage IP Fabric

Figure 17 on page 34—iBGP Within a PoD and eBGP Between PoDs in an Optimized 5-Stage IP Fabric

Figure 18 on page 88—Tenant and Layer 2 Extension Between Two Racks

Figure 19 on page 102—Tenant and Layer 2 Extension Between Two PoDs Connected by Super-Spines

Figure 20 on page 117—Tenant Extension Outside the Fabric Through Edge Leafs

Figure 21 on page 126—VLAN Scoping at the ToR Level

Figure 22 on page 136—VLAN Scoping at the Port Level Within a ToR

Figure 23 on page 145—Services Provisioning on the Border Leaf

Figure 24 on page 145—Service VRF with Route Leaking on the Border Leaf

Figure 25 on page 146—Topology of the Service VRF with Route Leaking from Tenants


List of Figures


Preface∙ Brocade Validated Designs.............................................................................................................................................................................................7∙ Purpose of This Document.............................................................................................................................................................................................7∙ Target Audience..................................................................................................................................................................................................................... 7∙ About the Authors.................................................................................................................................................................................................................7∙ Document History................................................................................................................................................................................................................8∙ About Brocade.......................................................................................................................................................................................................................8

Brocade Validated DesignsBrocade validated designs are reference architectures that are created and validated by Brocade engineers to address various customerdeployment scenarios and use cases. These validated designs provide a well-defined and standardized architecture for eachdeployment scenario, and they incorporate a broad set of technologies and feature sets across Brocade's product range that addresscustomer-unique requirements. These designs are comprehensively validated end-to-end so that the design solutions andconfigurations can be deployed more quickly, more reliably, and more predictably. Brocade validated designs are continuously validatedusing a test automation framework to ensure that once a design has been validated, it remains validated on new software releases andproducts.

Purpose of This DocumentThis Brocade validated design provides guidance for designing and implementing IP fabric in a data center network using Brocadehardware and software. It details the Brocade reference architecture for deploying IP fabric and EVPN-based VXLAN overlay.

It should be noted that not all features such as automation practices, zero-touch provisioning, and monitoring of the Brocade IP fabricare included in this document. Future versions of this document are planned to include these aspects of the Brocade IP fabric solution.

The design practices documented here follow the best-practice recommendations, but there are variations to the design that aresupported as well.

Target AudienceThis document is written for Brocade systems engineers, partners, and customers who design, implement, and support data centernetworks. This document is intended for experienced data center architects and engineers. This document assumes that the reader has agood understanding of data center switching and routing features and of Multi-Protocol BGP/MPLS VPN[5] for understandingmultitenancy in VXLAN EVPN networks.

About the AuthorsKrish Padmanabhan is a Principal Engineer on the IP SQA team at Brocade. Krish has extensive experience in the networking industryand in particular the data-center switching and routing, with roles ranging from product development, testing, systems and solutionvalidation, to customer-centric testing. At Brocade, he is focused on developing and validating solution architectures that customers canuse in deployments. He holds a CCIE certification in Routing and Switching.

Anuj Dewangan is the lead Technical Marketing Engineer (TME) for Brocade's data center switching products. He holds a CCIE inRouting and Switching and has several years of experience in the networking industry with roles in software development, solutionvalidation, and technical marketing. At Brocade, his focus is creating reference architectures, working with customers and account teams


to address their challenges with data center networks, and creating product and solution collateral. He speaks at industry events and hasauthored several white papers on data center networking.

Poorani Arthanari is a Staff Engineer on the IP SQA team at Brocade. Poorani has extensive experience testing data center fabric and IProuting technologies. She has been involved in validating solution architectures.

The authors would like to acknowledge the following Brocadians for their technical guidance in developing this validated design:

∙ Mangesh Shingane: Principal Engineer

∙ Syed Hasan Raza Naqvi: Technical Leader

∙ Venugopal Mundathaya: Senior Staff Engineer

Document HistoryDate Part Number Description

March 23, 2016 53-1004308-01 Initial release.

March 30, 2016 53-1004308-02 Minor formatting changes.

August 12, 2016 53-1004308-03 IP unnumbered interface support for 3-stagefabric.

Illustration examples for:

∙ VLAN scoping at the ToR level andwithin the ToR

∙ Route leaking with the service VRF onthe edge leaf

Additional design considerations.

About BrocadeBrocade® (NASDAQ: BRCD) networking solutions help the world's leading organizations transition smoothly to a world whereapplications and information reside anywhere. This vision is designed to deliver key business benefits such as unmatched simplicity,non-stop networking, application optimization, and investment protection.

Innovative Ethernet and storage networking solutions for data center, campus, and service provider networks help reduce complexity andcost while enabling virtualization and cloud computing to increase business agility.

To help ensure a complete solution, Brocade partners with world-class IT companies and provides comprehensive education, support,and professional services offerings (www.brocade.com).

Preface


http://www.brocade.com

IntroductionBased on the principles of the New IP, Brocade is building on the proven success of the VDX platform by expanding our cloud-optimized network and network virtualization architectures to meet customer demand for higher levels of scale, agility, and operationalefficiency.

This document describes cloud-optimized network designs using Brocade IP fabrics for building data-center sites. The configurationsand design practices documented here are fully validated and conform to the Brocade IP fabric reference architectures. The intention ofthis Brocade validated design document is to provide reference configurations and document best practices for building cloud-scaledata-center networks using Brocade VDX switches and Brocade IP fabric architectures.

This document describes the following architectures:

∙ Brocade IP fabric deployed in 3-stage and optimized 5-stage folded Clos topologies

∙ Brocade IP fabric with network virtualization using BGP EVPN deployed in 3-stage and optimized 5-stage folded Clostopologies

We highly recommend reviewing the data-center fabric architectures described in the Brocade Data Center Fabric Architectures[7] whitepaper for a detailed discussion on data-center architectures for building data-center sites.


Introduction


Brocade IP Fabric Technology Overview∙ Benefits.......................................................................................................................................................................................................................................11∙ Terminology............................................................................................................................................................................................................................. 11∙ Functional Components of Brocade IP Fabric.................................................................................................................................................. 12

Brocade IP fabric provides a Layer 3 Clos deployment architecture for data center sites. With Brocade IP fabric, all links in the Clostopology are Layer 3 links. The Brocade IP fabric includes the networking architecture; the protocols used to build the network; turnkeyautomation features used to provision, manage, and monitor the networking infrastructure; and the hardware differentiation with BrocadeVDX switches. The following sections describe the validated design for data center sites with Brocade IP fabrics. Because theinfrastructure is built on IP, advantages like the following are leveraged: loop-free communication using industry-standard routingprotocols, ECMP, very high solution scale, and standards-based interoperability.

BenefitsSome of the key benefits of deploying data center sites with Brocade IP fabrics:

Highly scalable infrastructure—Because the Clos topology is built with IP protocols, the scale of the infrastructure is very high. The portand rack scales are documented with descriptions of the Brocade IP fabric deployment topologies.

Standards-based and interoperable protocols—The Brocade IP fabric is built with industry-standard protocols like Border GatewayProtocol (BGP) and Open Shortest Path First (OSPF). These protocols are well understood and provide a solid foundation for a highlyscalable solution. In addition, industry-standard overlay control- and data-plane protocols like BGP-EVPN and Virtual Extensible LocalArea Network (VXLAN) are used to extend the Layer 2 domain and extend tenancy domains by enabling Layer 2 communications andVM mobility.

Active-active vLAG pairs—By supporting vLAG pairs on leaf switches, dual-homing of the networking endpoints is supported. Thisprovides higher redundancy. Also, because the links are active-active, vLAG pairs provide higher throughput to the endpoints. vLAGpairs are supported for all 10-GbE, 40-GbE, and 100-GbE interface speeds, and up to 32 links can participate in a vLAG.

Support for unnumbered interfaces—Using Brocade Network OS support for IP unnumbered interfaces, only one IP address per switchis required to configure the routing protocol peering. This support significantly reduces the planning and use of IP addresses, and itsimplifies operations.

Programmable automation—Brocade server-based automation provides support for common industry automation tools such as PythonAnsible, Puppet, and YANG model based REST and NETCONF APIs. The prepackaged PyNOS scripting library and editableautomation scripts execute predefined provisioning tasks, while allowing customization for addressing unique requirements to meettechnical or business objectives when the enterprise is ready.

Ecosystem integration—The Brocade IP fabric integrates with leading industry solutions and products like VMware vSphere, NSX, andvRealize. Cloud orchestration and control are provided through OpenStack and OpenDaylight based Brocade SDN Controller support.

TerminologyTerm Description

ARP Address Resolution Protocol

AS Autonomous System

ASN Autonomous System Number

BFD Bidirectional Forwarding Detection

BGP Border Gateway Protocol


Term Description

BUM Broadcast, Unknown unicast, and Multicast

DCI Data Center Interconnect

eBGP External Border Gateway Protocol

This refers to BGP peering between two nodes in two different autonomous systems.

ECMP Equal Cost Multi-Path

EVPN Ethernet Virtual Private Network

iBGP Internal Border Gateway Protocol

This refers to BGP peering between two nodes in the same autonomous system.

IP Internet Protocol

IRB Integrated Routing and Bridging

MAC Media Access Control

MP-BGP Multi-Protocol Border Gateway Protocol

MPLS Multi-Protocol Label Switching

ND Neighbor Discovery

NLRI Network Layer Reachability Information

PoD Point of Delivery

RD Route Distinguisher

RT Route Target

ToR Top of Rack switch

Also leaf or VTEP in an IP fabric context.

UDP User Datagram Protocol

vLAG Virtual Link Aggregation Group

VLAN Virtual Local Area Network

VM Virtual Machine

VNI VXLAN Network Identifier

VPN Virtual Private Network

VRF VPN Routing and Forwarding instance

An instance of the routing/forwarding table with a set of networks and hosts in a router. A router may have multiple suchinstances isolated from each other. Also referred to as a tenant. In IP fabric, this may be localized to one VTEP/leaf or may bespread across multiple VTEPs across the IP fabric and beyond the border leaf.

VTEP VXLAN Tunnel End Point

In IP fabric, leaf and VTEP are used interchangeably.

VXLAN Virtual Extensible Local Area Network

Functional Components of Brocade IP Fabric

Leaf-Spine Layer 3 Clos Topology (Two-Tier)The leaf-spine topology has become the de facto standard for networking topologies when building medium- to large-scale data centerinfrastructures. The leaf-spine topology is adapted from Clos telecommunications networks. The Brocade IP fabric within a PoDresembles a two-tier or 3-stage folded Clos fabric. The two-tier leaf-spine topology is shown in Figure 1. The bottom layer of the IPfabric has the leaf devices (top-of-rack switches), and the top layer has spines. The role of the leaf is to provide connectivity to the

Brocade IP Fabric Technology Overview


endpoints in the data center network. These endpoints include compute servers and storage devices as well as other networking deviceslike routers, switches, load balancers, firewalls, and any other physical or virtual networking endpoints. Because all endpoints connectonly to the leaf, policy enforcement, including security, traffic-path selection, QoS marking, traffic policing, and shaping, is implementedat the leaf.

More importantly, the leafs act as the anycast gateways for the server segments to facilitate mobility with the VXLAN overlay.

The role of the spine is to provide connectivity between leafs. The major role of the spine is to participate in the control-plane and data-plane operations for traffic forwarding between leafs. The spine devices serve two purposes: BGP control plane (route reflectors for leafor eBGP peering with leaf) and IP forwarding based on the outer IP header in the underlay network. Since there are no networkendpoints connected to the spine, tenant VRFs or VXLAN segments are not created on spines. Their routing table size requirements arealso very light to accommodate just the underlay reachability. Note that all spine devices need not act as BGP route reflectors; onlyselected spines in the spine layer can act as BGP route reflectors in the overlay design. More details are provided in the "BGP EVPNControl Plane" section of the "Network Virtualization with BGP EVPN" chapter.

As a design principle, the following requirements apply to the leaf-spine topology:

∙ Each leaf connects to all spines in the network through 40-GbE links.

∙ Spines are not interconnected with each other.

∙ Leafs are not interconnected with each other for data-plane purposes. (The leafs may be interconnected for control-planeoperations such as forming a server-facing vLAG.)

∙ The network endpoints do not connect to the spines.

This type of topology has the predictable latency and also provides the ECMP forwarding in the underlay network. The number of hopsbetween two leaf devices is always two within the fabric. This topology also enables easier scale out in the horizontal direction as the datacenter expands and is limited by the port density and bandwidth supported by the spine devices.

This validated design recommends the same hardware in the spine layer. Mixing different hardware is not recommended.

IP Fabric Infrastructure LinksAll fabric nodes—leafs, spines, and super-spines—are interconnected with Layer 3 interfaces. In the validated design,

∙ 40-GbE links are used between the fabric nodes.

∙ All these links are configured as Layer 3 interfaces with /31 IPv4 address. For a simple 3-stage fabric, IP unnumberedinterfaces can be used. We do not recommend a mix of unnumbered and numbered interfaces within a fabric. Also, for a5-stage IP fabric, numbered interfaces are highly recommended.

∙ The MTU for these links is set to jumbo MTU. This is a requirement to handle the VXLAN encapsulation of Ethernet frames.

Server-Facing LinksThe server-facing or access links are on the leaf nodes. In the validated design,

∙ 10-GbE links are used for server-facing VLANs.

∙ These links are configured as Layer 2 trunks with associated VLANs.

∙ The MTU for these links is set to the default: 1500 bytes.

∙ Spanning tree is disabled.1

1 Spanning tree must be enabled if there are Layer 2 switches/bridges between a leaf and servers.



FIGURE 1 Leaf-Spine L3 Clos Topology

Optimized 5-Stage Layer 3 Clos Topology (Three-Tier)Multiple PoDs based on leaf-spine topologies can be connected for higher scale in an optimized 5-stage folded Clos (three-tier)topology. This topology adds a new tier to the network, known as a super-spine. This architecture is recommended for interconnectingseveral EVPN VXLAN PoDs. Super-spines function similar to spines: BGP control-plane and IP forwarding based on the outer IPheader in the underlay network. No endpoints are connected to the super-spine. Figure 2 shows four super-spine switches connectingthe spine switches across multiple data center PoDs.

The connection between the spines and the super-spines follows the Clos principles:

∙ Each spine connects to all super-spines in the network.

∙ Neither spines nor super-spines are interconnected with each other.



FIGURE 2 Optimized 5-Stage L3 Clos Topology

Edge Services and Border LeafsFor two-tier and three-tier data center topologies, the role of the border leaf in the network is to provide external connectivity to the datacenter site. In addition, since all traffic enters and exits the data center through the border leaf switches, they present the ideal location inthe network to connect network services like firewalls, load balancers, and edge VPN routers. The border leaf switches connect to theWAN edge devices in the network to provide external connectivity to the data center site. As a design principle, two border leaf switchesare recommended for redundancy. The WAN edge devices provide the interfaces to the Internet and DCI solutions. For DCI, thesedevices function as the Provide Edge (PE) routers, enabling connections to other data center sites through WAN technologies likeMultiprotocol Label Switching (MPLS) VPN and Virtual Private LAN Services (VPLS). The Brocade validated design for DCI solutions isdiscussed in a separate validated design document.

There are several ways that the border leafs connect to the data center site. In three-tier (super-spine) architectures, the border leafs aretypically connected to the super-spines as depicted in Figure 2. In two-tier topologies, the border leafs are connected to the spines asdepicted in Figure 1. Certain topologies may use the spine as border leafs (known as a border spine), overloading two functions into one.This topology adds additional forwarding requirements to spines—they need to be aware of the tenants, VNIs, and VXLAN tunnelencapsulation and de-encapsulation functions.

Brocade IP Fabric Underlay RoutingIP fabric collectively refers to the following:

∙ IPv4 network address assignments to the links connecting the nodes in the fabric: spines, leafs, super-spines, and border leafs.

∙ Control-plane protocol used for reachability between the nodes. A smaller scale topology might benefit from a link-stateprotocol such as OSPF. Large scale topologies, however, typically use BGP. Brocade validated design recommends BGP asthe protocol for underlay network reachability.

∙ Resiliency feature such as BFD.



There are several underlay deployment options. In the validated design, we recommend two deployment models based on how the BGPprotocol is deployed in the IP fabric:

∙ eBGP for Underlay—eBGP peering between each tier of nodes: between the leaf and the spine; between the spine and thesuper-spine; and between the super-spine and the border leaf.

∙ iBGP for Underlay—iBGP peering between the leaf and the spine within the PoD and spines as BGP route reflectors. eBGPpeering between the PoDs through the super-spine layer for inter-PoD reachability.

eBGP for UnderlayThis deployment model refers to the usage of eBGP peering between the leaf and the spine in the fabric. In this model, each leaf node isassigned its own autonomous system (AS) number. The other nodes are grouped based on their role in the fabric, and each of thesegroups is assigned a separate AS number, as shown in Figure 3. Using eBGP in an IP fabric is simple and also provides the ability toapply BGP policies for traffic engineering on a per-leaf or per-rack basis since each leaf or rack in a PoD is assigned a unique ASnumber. Private AS numbers should be used in the fabric. One design consideration for the AS number assignment is that a 2-byte ASnumber provides a maximum of 1023 private AS numbers (ASN 64512 to ASN 65534); if the IP fabric is larger than 1023 devices, werecommend using 4-byte private AS numbers (ASN 4,200,000,000 to 4,294,967,294).

∙ Each leaf in a PoD is assigned its own AS number.

∙ All spines inside a PoD belong to one AS.

∙ All super-spines are configured in one AS.

∙ Edge or border leafs belong to a separate AS.

∙ Each leaf peers with all spines using eBGP.

∙ Each spine peers with all super-spines using eBGP.

∙ There is no eBGP peering between leafs.

∙ There is no eBGP peering between spines.

∙ There is no eBGP peering between super-spines.



FIGURE 3 eBGP for Underlay

iBGP for UnderlayIn this deployment model, each PoD and edge services PoD is configured with a unique AS number, as shown in Figure 4. The spinesand leafs in a PoD are configured with the same AS number. The iBGP design is different than the eBGP design because iBGP must befully meshed with all BGP-enabled devices in an IP fabric. In order to avoid the complexities of a full mesh, route reflectors must be usedin the fabric. iBGP peering is between the spine and the leaf in a PoD, and all spines in a PoD act as BGP route reflectors to the leafs forthe underlay.

eBGP is used to peer between spines and super-spines. The super-spine layer is configured with a unique AS number; all super-spinesuse the same AS number.

When an EVPN Address-Family is enabled for overlay,

∙ Two spines in each PoD are enabled with EVPN AFI, and they act as the RR to the leaf.

∙ Leafs exchange EVPN routes to the spine RRs.

∙ These spines also exchange EVPN routes with super-spines.

∙ Edge leafs exchange EVPN routes with super-spines.



FIGURE 4 iBGP for Underlay



Network Virtualization with BGP EVPN∙ VXLAN Layer 2 Extension Using Flood and Learn......................................................................................................................................20∙ BGP EVPN for VXLAN................................................................................................................................................................................................. 22

Network virtualization is the process of creating virtual, logical networks on physical infrastructures. With network virtualization, multiplephysical networks can be consolidated to form a logical network. Conversely, a physical network can be segregated to form multiplevirtual networks. Virtual networks are created through a combination of hardware and software elements spanning the networking,storage, and computing infrastructure. Network virtualization solutions leverage the benefits of software in terms of agility andprogrammability, along with the performance acceleration and scale of application-specific hardware.

Virtual Extensible LAN (VXLAN) is an overlay technology that provides Layer 2 connectivity for workloads residing across the datacenter network. VXLAN creates a logical network overlay on top of physical networks, extending Layer 2 domains across Layer 3boundaries. VXLAN provides decoupling of the virtual topology provided by the VXLAN tunnels from the physical topology of thenetwork. It leverages Layer 3 benefits in the underlay, such as load balancing on redundant links, which leads to higher networkutilization. In addition, VXLAN provides a large number of logical network segments, allowing for large-scale multitenancy in the network.VXLAN is based on the IETF RFC 7348 standard. VXLAN has a 24-bit Virtual Network ID (VNID) space, which allows for 16 millionlogical networks compared to a traditional VLAN, which supports a maximum of 4096 logical segments. VXLAN eliminates the need forSpanning Tree Protocol (STP) in the data center network, and it provides increased scalability and improved resiliency. VXLAN hasbecome the de facto standard for overlays that are terminated on physical switches or virtual network elements.

The traditional Layer 2 extension mechanisms using VXLAN rely on "Flood and Learn" mechanisms. These mechanisms are veryinefficient, delaying MAC address convergence and resulting in unnecessary flooding. Also, in a data center environment with VXLAN-based Layer 2 extension mechanisms, a Layer 2 domain and an associated subnet might exist across multiple racks and even across allracks in a data center site. With traditional underlay routing mechanisms, routed traffic destined to a VM or a host belonging to thesubnet follows an inefficient path in the network, because the network infrastructure is aware only of the existence of the distributedLayer 3 subnet, but it is not aware of the exact location of the hosts behind a leaf switch.

With Brocade BGP-EVPN network virtualization, network virtualization is achieved by creating a VXLAN-based overlay network.Brocade BGP-EVPN network virtualization leverages BGP-EVPN to provide a control plane for the virtual overlay network. BGP-EVPNenables control-plane learning for end hosts behind remote VXLAN tunnel end points (VTEPs). This learning includes reachability forLayer 2 MAC addresses and Layer 3 host routes.

Some key features and benefits of Brocade BGP-EVPN network virtualization are summarized as follows:

Active-active vLAG pairs—vLAG pairs for a multiswitch port channel for dual homing of network endpoints are supported at the leaf.Both switches in the vLAG pair participate in the BGP-EVPN operations and are capable of actively forwarding traffic.

Static anycast gateway—With static anycast gateway technology, each leaf is assigned the same default gateway IP and MAC addressesfor all connected subnets. This ensures that local traffic is terminated and routed at Layer 3 at the leaf. This also eliminates anysuboptimal inefficiencies found with centralized gateways. All leafs are simultaneously active forwarders for all default traffic for whichthey are enabled. Also, because the static anycast gateway does not rely on any control-plane protocol, it can scale to large deployments.

Efficient VXLAN routing—With the existence of active-active vLAG pairs and the static anycast gateway, all traffic is routed and switchedat the leaf. Routed traffic from the network endpoints is terminated in the leaf and is then encapsulated in the VXLAN header to be sentto the remote site. Similarly, traffic from the remote leaf node is VXLAN-encapsulated and must be decapsulated and routed to thedestination. This VXLAN routing operation in to and out of the tunnel on the leaf switches is enabled in the Brocade VDX 6740 and6940 platform ASICs. VXLAN routing performed in a single pass is more efficient than competitive ASICs.

Data-plane IP and MAC learning—With IP host routes and MAC addresses learned from the data plane and advertised with BGP-EVPN,the leaf switches are aware of the reachability of the hosts in the network. Any traffic destined to the hosts takes the most efficient routein the network.


Layer 2 and Layer 3 multitenancy—BGP-EVPN provides the control plane for VRF routing and for Layer 2 VXLAN extension. BGP-EVPN enables a multitenant infrastructure and extends it across the data center to enable traffic isolation between the Layer 2 and Layer3 domains, while providing efficient routing and switching between the tenant endpoints.

Dynamic tunnel discovery—With BGP-EVPN, the remote VTEPs are automatically discovered. The resulting VXLAN tunnels are alsoautomatically created. This significantly reduces operational expense (OpEx) and eliminates errors in configuration.

ARP/ND suppression—The BGP-EVPN EVI leafs discover remote IP and MAC addresses and use this information to populate theirlocal ARP tables. Using these entries, the leaf switches respond to any local ARP queries. This eliminates the need for flooding ARPrequests in the network infrastructure.

Conversational ARP/ND learning—Conversational ARP/ND reduces the number of cached ARP/ND entries by programming onlyactive flows into the forwarding plane. This helps to optimize utilization of hardware resources. In many scenarios, there are softwarerequirements for ARP and ND entries beyond the hardware capacity. Conversational ARP/ND limits storage-in-hardware to activeARP/ND entries; aged-out entries are deleted automatically.

VM mobility support—If a VM moves behind a leaf switch, with data-plane learning, the leaf switch discovers the VM and learns itsaddressing information. It advertises the reachability to its peers, and when the peers receive the updated information for the reachabilityof the VM, they update their forwarding tables accordingly. BGP-EVPN-assisted VM mobility leads to faster convergence in thenetwork.

Open standards and interoperability—BGP-EVPN is based on the open standard protocol and is interoperable with implementationsfrom other vendors. This allows the BGP-EVPN-based solution to fit seamlessly in a multivendor environment.

VXLAN Layer 2 Extension Using Flood and LearnLet's consider the simple topology shown in Figure 5, which represents VXLAN extension, to understand how VXLAN flood and learnworks before going into the details of control-based VXLAN using BGP EVPN and the various network functions that the EVPN controlplane enables.

FIGURE 5 VTEPs and L2 Extension with Flood and Learn

Network Virtualization with BGP EVPN


VXLAN tunnel end point (VTEP) may be implemented in hardware (leaf or ToR switch) or in virtualized environments. Each VTEP has aunique IP address and MAC address. Each VTEP can reach other VTEPs over the underlay IP network.

Each VTEP has its own end host/server segment connected to it. In this topology, all hosts belong to one Layer 2 broadcast domain or,in simple terms, one VLAN and one IP subnet. The local VLAN numbers may be different in each VTEP, but they are bound to one VNInumber, which is common on all VTEPs. So for all practical purposes, the LAN segment is now identified by a VXLAN VNI, and theVLAN numbers are only locally significant.

The logical dashed lines shown inside the IP network between the VTEPs represent the head-end or ingress replication paths. This isused to send what is known as the BUM traffic: Broadcast, Unknown Unicast, and Multicast frames on the Layer 2 segment. The VTEPunicasts these packets to all other VTEPS connected to a VXLAN segment. This may require additional configuration or provisioning oftunnels on each VTEP device to all other devices.

Let's consider that H1 wants to communicate with H2:

∙ H1 sends an ARP request.

∙ VTEP-A learns H1 as a local MAC and also maps this host to the VNI, and because the packet is a broadcast packet, it isencapsulated into the VXLAN packet and replicated; it is then unicast to each of the remote VTEPs participating in this VNIsegment. The outer-src-ip is set to 10.10.10.1, and the outer-dst-ip is the remote VTEP IP.

∙ This packet is sent to every VTEP.

∙ VTEP-B and VTEP-C decapsulate the packet and flood it into their local VXLAN network.

∙ They also learn three pieces of information: the source-ip of VTEP-A, the inner-src-mac of H1, and the VNI. This creates anL2-MAC-to-VTEP-IP binding: {mac H1, VTEP-ip 10.10.10.1, VNI 10}.

∙ When H2 responds to the ARP request, the packet is unicast to H1. This packet is encapsulated in a VXLAN packet by VTEP-Band sent as a unicast IP packet based on its routing table:

– outer-ip header - dst: 10.10.10.1, src 10.10.10.2

∙ VTEP-A decapsulates the packet and sends it to H1. It also creates an L2-MAC-to-VTEP-IP binding: {MAC H2, VTEP-ip10.10.10.2, VNI 10}

∙ Now the communication between H1 and H2 will be unicast. VTEP-A and VTEP-B now know sufficient information toencapsulate the packets between them. The multicast tree is not used.

When the hosts are in different subnets, we need a Layer 3 gateway in the network to connect to all VNI segments. As seen in Figure 6,VTEP-C is configured with all VNI numbers in the network and acts as the router or gateway between these VNI segments (see the blueand red dotted arrows routing between VLAN10 and VLAN20). When hosts send ARP messages for the gateway in their respectiveVLANs, VTEP-C will respond.



FIGURE 6 Routing Between VXLANs in a Flood-and-Learn Topology

For first-hop router redundancy, multiple VTEPs may be configured with all VNIs, and they may run an FHRP protocol between them.

BGP EVPN for VXLANAs we have seen in the VXLAN flood and learn case, the MAC learning is data frame-driven and flooding of broadcast or unknownunicast frames depends on ingress replication by VTEPs in the network.

With the BGP EVPN control plane, the MAC learning happens via BGP similar to IPv4/IPv6 route learning in a Layer 3 network. Thisreduces flooding in the underlay network except for remarkably silent hosts. This control-plane-based MAC learning enables severaladditional functions with BGP as the unified control plane for both Layer 2 and Layer 3 forwarding in the overlay network.

In Figure 7, each VTEP, being a BGP speaker, advertises the MAC and IP addresses of its local hosts to other VTEPs using the BGPEVPN control plane. A BGP route-reflector may be used for distribution of this information to the VTEPs. Both VTEP discovery andMAC/IP or MAC/IPv6 host learning happen through the control plane.

Since IPv4/IPv6 addresses are also exchanged in the control plane, each VTEP may act as a gateway for the VNI subnets configured onit. A centralized Layer 3 gateway is not required. This feature is also referred to as distributed gateway. Also, since each VTEP is aware ofMAC/IP or MAC/IPv6 host bindings, ARP requests need not be flooded between the VTEPS. The VTEP may respond to the ARPrequests on behalf of the target host, if the host address has already been learned. This is referred to as ARP/ND suppression in thefabric.



FIGURE 7 VTEPs and L2 Extension with the BGP EVPN Control Plane

BGP EVPN control-plane-based learning allows more flexibility to control the information flow between the VTEPs. It also enablesmultitenancy using VRFs similar to MPLS-VPN. Each VTEP may host several tenants and each tenant with a set of VXLAN segments.Depending on the interest, other VTEPs may import the tenant-specific information. This way both Layer 2 and Layer 3 extensions canbe provisioned on a tenant basis.

BUM traffic may be accommodated either with ingress replication or a multicast tree. Since VTEP discovery also happens through thecontrol plane, setting up ingress replication does not require additional provisioning or configuration about remote VTEPs. BrocadeEVPN implementation supports ingress replication.

VTEPIn IP fabric, the leaf and border leaf act as VTEPs. Note that only one VTEP is allowed per device. Every VTEP has an overlay interface,which identifies the VTEP IP address. The VTEP info is exchanged, and remote VTEPs are discovered over BGP EVPN.

Static Anycast GatewayEach leaf or VTEP has a set of server-facing VLANs that are mapped to VXLAN segments by a VNI number. These VLAN segmentshave an associated VE interface (a Layer 3 interface for the VLAN). Each tenant VLAN has anycast gateway IPv4/IPv6 addresses andassociated anycast gateway MAC addresses. These gateway IP/IPv6 addresses and gateway MAC address are consistent for the VLANsegments shared on all leafs in the fabric.

Overlay GatewayEach VTEP or leaf is configured with an overlay gateway. This defines the VTEP IP address, which is used as the source IP whenencapsulating packets and is used as the next-hop IP in the EVPN NLRIs. In this validated design, we are using an IPv4 underlay; hencethe overlay interface is associated with the IPv4 address of a loopback interface on the leaf.



BGP EVPN Control PlaneThe BGP EVPN control plane is used for VTEP discovery to learn MAC/IP routes from other VTEPs. The exchange of this informationtakes place using EVPN NLRIs. The NLRI uses the existing AFI of 25 (L2VPN). IANA has assigned BGP EVPNs a SAFI value of 70.The NLRI also carries a tunnel encapsulation attribute. For IP fabric using VXLAN encapsulation, the attribute is set to VXLAN.

In the leaf-spine topology (3-stage Clos or 5-stage Clos), all leafs and border leafs should be enabled with the BGP EVPN Address-Family to exchange EVPN routes (NLRI) and participate in VTEP discovery. Spine and super-spines do not participate in the VTEPfunctionality. However, selected spines in the spine layer should be enabled with the BGP EVPN Address-Family, and all leafs includingborder leafs must be peered with the spines who have the BGP EVPN Address-Family enabled.

In the deployment model where eBGP is used, a minimum of two spines in the PoD should be enabled with the EVPN Address-Family.Note that all spines participate in the eBGP underlay, but only a few designated spines participate in the EVPN.

In the deployment model where iBGP is used, two spines are selected as route-reflectors for the EVPN Address-Family, and eachVTEP leaf has two iBGP neighbors that are the two spine BGP route reflectors. Each spine BGP route reflector has all VTEP leaf nodesas route-reflector clients and reflects EVPN routes for the VTEP leaf nodes.

In the 5-stage Clos topology, a minimum of two super-spines should be enabled with the EVPN Address-Family, and only the spinesthat are enabled with EVPN are peered with these super-spines. More detailed design is discussed in the "Network Virtualization withBGP EVPN" section of the "IP Fabric Validated Designs" chapter.

EVPN Route TypesEVPN uses different route types to carry various network-layer reachability information. The following are the well-known route typesdefined in BGP EVPN:

∙ Route Type-1—Ethernet Auto Discovery. This route is used for remote VTEP discovery and association to the VLAN/VNI.

∙ Route Type-2: MAC/IP advertisement route:

– MAC-only route that carries {MAC address of the host, L2VNI of the VXLAN segment}. This route carries only the Layer 2information of a host. Whenever a VTEP learns a MAC from its server-facing subnets, it advertises this route into BGP.

– MAC/IP route that carries {MAC address of the host, IPv4/IPv6 address of the host, L2VNI of the VXLAN segment,L3VNI of the tenant VRF of the host}. This route carries both the Layer 2 and Layer 3 information of the hosts. This routeis advertised by the VTEP when it learns the IPv4/IPv6 host addresses via ARP or ND from the server-facing subnets.This information enables ARP/ND suppression on other VTEPs.

∙ Route Type-3—Inclusive Multicast Ethernet Tag route. This route is required for sending BUM traffic to all VTEPs interested fora given bridge domain or VXLAN segment.

∙ Route Type-4—Ethernet Segment route is used for multi-homing of server vlan segments to two ToRs. Only VLAG basedmulti-homing is supported.

∙ Route Type-5— IPv4/IPv6 prefix advertisement route {IPv4/IPv6 route, L3VNI, Router-MAC}. This route is advertised forevery Layer 3 server-facing subnet behind a VTEP or external routes.

Tunnel AttributeExtended community type 0x3, sub-type 0x0c, and tunnel encapsulation type 0x8 (VXLAN). This is included with all EVPN routes.

Layer 3 VNI or Tenant VRFEach tenant VRF is configured with a unique Layer 3 VNI. This is required for inter-subnet routing. This VNI must be the same for atenant VRF on all VTEPs including the border leaf. Both Type-2 and Type-5 routes carry this Layer 3 VNI.



Router-MAC Extended CommunityExtended community type EVPN (0x06) and sub-type 0x03.

The router-mac is the MAC address of the VTEP advertising a route. This is also required along with the Layer 3 VNI for inter-subnetrouting as explained in the "Integrated Routing and Bridging (IRB)" section of this chapter, and it is carried in both Type-2 MAC/IP routesand Type-5 prefix routes. In the data plane, this MAC address is used as the inner destination MAC address when a packet is routed.

MAC-Mobility AttributeExtended community type EVPN (0x06) and sub-type 0x00. Carries a 32-bit sequence number.

This enables MAC or station moves between the VTEPs. When a MAC moves, for example, from VTEP-1 to VTEP-2, VTEP-2advertises a MAC (or MAC/IP) route with a higher sequence number. This update triggers a best-path calculation on other VTEPs,thereby detecting the host move to VTEP-2.

ARP SuppressionControl-plane distribution of MAC/IP addresses enables ARP suppression in the fabric for Layer 2 extensions between racks. A portionof the fabric is shown in Figure 8 to illustrate the ARP suppression functionality in the fabric.

When the hosts come up, they typically ARP for the gateway IP that is hosted by leafs. Let's consider the case where H2 ARPs for thegateway address. Note that both leafs have the same anycast gateway address for the host VXLAN segment.

∙ Leaf2 learns the MAC/IP (or ARP) binding for H2.

∙ Leaf2 will advertise the MAC/IP route into the BGP EVPN Address-Family.

∙ Leaf1 will learn this route and populate it in its MAC/IP binding table.

∙ H1 sends an ARP request to H2. Leaf1 will respond on behalf of H2.

∙ Extending the same information flow for H1, when Leaf2 learns H1's MAC/IP route, it will respond to ARP requests on behalf ofH1.

Compared to the data-plane-based learning in Layer 2 extension technologies such as VPLS or VXLAN flood and learn, where ARPtraffic is also sent over an overlay network, VXLAN EVPN significantly reduces ARP/ND flooding in the fabric.



FIGURE 8 ARP Suppression

VLAN ScopingAs discussed earlier, in VXLAN networks, each VLAN is mapped to a VNI number of a VXLAN segment. This provides an interestingoption to break the 4K limit of the 802.1Q VLAN space. The VLAN tag (or c-tag) on the wire or the port VLAN membership may belocally scoped or locally significant at the leaf level or at the port level within a leaf.

VLAN Scoping at the Leaf LevelIn this case, the VLANs are scoped at the leaf or ToR level. Refer to Figure 9.

In this example, VLAN 10 is mapped to VNI 10 on Leaf1, and VLAN 20 is mapped to VNI 10 on Leaf2. By mapping to the same VNI, thetwo VLAN segments (VLAN 10 and VLAN 20) are on the same bridge domain. With this mapping, hosts on these VLANs have Layer 2extension between them, and they belong to one VXLAN segment identified by the VNI 10.



FIGURE 9 VLAN Scoping at the Leaf Level

VLAN Scoping at the Port Level Within a LeafVLAN scoping at the port level can be accomplished using the Virtual-Fabric feature on Brocade switches. The Virtual-Fabric featurebasically abstracts a VLAN or bridge domain and decouples the VLAN tag (or c-tag) on the wire.

Refer to Figure 10. In this example, Port1, VLAN tag 10, and Port2, VLAN tag 20, are mapped to a VLAN 5001, and VLAN 5001 ismapped to VNI 5001. With this mapping, the hosts H1 (VLAN 10), H2 (VLAN 20), and H3 (VLAN 501) are bound to one VXLANsegment identified by the VNI 5001.

FIGURE 10 VLAN Scoping at the Port Level Within a ToR

Conversational LearningConversational learning helps conserve the hardware forwarding table by programming only those ARP/ND or MAC entries for whichthere are active conversations or traffic flows. With this feature, the control plane may hold more host entries than what the hardware



table can support. When there is sufficient space in hardware, all host entries are programmed. When there is no space, conversationallearning kicks in and starts aging out the inactive entries. Note that the host subnets are inserted into the hardware (LPM table) regardlessof the activity. The host entries are inserted in the hardware (/32 IPv4 or /128 IPv6 host route table) based on the traffic.

Integrated Routing and BridgingWith the anycast gateway function, each VTEP or leaf acts as an Integrated Routing and Bridging (IRB) device providing Layer 2extension as well Layer 3 routing between the VXLAN segments in a tenant. Note that the tenant may span multiple leafs. There are twovariations of IRB implementation in the IP fabric: asymmetric IRB and symmetric IRB.

Asymmetric IRB

FIGURE 11 Asymmetric IRB

In Figure 11, a tenant, SALES, is provisioned in the fabric with two VNI segments, VNI 10 and VNI 20. Leaf1 has servers connected to iton VNI 10 only. Yet it is provisioned with both VXLAN segment VNI 10 and VNI 20. If H1 in VNI 10 needs to communicate with H3 inVNI 20, Leaf1 routes the packet first between the segments and then bridges the packet on VNI 20 and the packet is sent on theoverlay. Leaf2 will decapsulate the VXLAN headers and send the packet to H3.

Essentially, the ingress VTEP both routes and bridges the packet; this method is referred as asymmetric IRB. This also means that everyVTEP must be configured with all VXLAN segments in a given tenant regardless of any local servers connected to the VNI segment.

Symmetric IRBFigure 12 depicts symmetric IRB. Here, every tenant is assigned a Layer 3 VNI. This is analogous to a Layer 3 routing interface betweentwo switches. This VNI must be the same for a given tenant on all leafs where it is provisioned.

The MAC/IP host routes are advertised by the VTEP with the L2 VNI as well as an L3 VNI and the router-mac address of the VTEP.When a packet is routed over the L3 VNI, the dst-mac of the inner Ethernet payload is set to the router-mac of the remote VTEP. In



Figure 12, routing from H1 to H3 always occurs over this L3 VNI. That is, both leaf devices route the packet once: by the ingress leaffrom the server VLAN/VNI to the L3 VNI and by the egress leaf from the L3 VNI to the server VLAN/VNI.

A significant advantage of this method is that all VNIs of a given tenant need not be created on all leafs. They are created only whenthere is server connectivity to those VNIs. In Figure 12, Leaf1 is not configured with VNI 20. Also note that on Leaf2, even though VNI 10is present, a packet from H3 to H1 will be routed directly on to the L3 VNI of the tenant. This adds the additional requirement that thehost routes on all VXLAN segments in a given tenant need to be downloaded to the Leaf's forwarding table.

FIGURE 12 Symmetric IRB

Brocade IRB ImplementationBoth symmetric and asymmetric IRB methods are implemented on Brocade switches. If the target VNI segment is configured on aVTEP, asymmetric IRB is performed. Otherwise, the packet is routed over the L3 VNI or symmetric routing occurs. Every tenant VRF isassigned with an L3 VNI.

In the Brocade implementation, we get the best of both schemes:

∙ No need to create all server VNIs on all leafs for a tenant.

∙ If a target VNI segment is not local and is extended behind one or more remote VTEPs, download the host routes on that targetsegment into hardware based on traffic activity. Traffic to these hosts will be routed over the L3 VNI.

MultitenancyIn BGP EVPN, multiple tenants can co-exist and share a common IP transport network while having their own separate routing domainin the VXLAN overlay network. Every tenant in the EVPN network is identified by a VRF (VPN routing and forwarding instance), andVRFs can span multiple leafs in a data center. (Similar to Layer 3 MPLS VPNs with tenant VRFs on multiple PE devices). Each VRF canhave a set of server-facing VLANs and a Layer 3 VLAN interface with a unique VNI used for symmetric routing purposes. This VNIshould be the same if the same tenant VRF is provisioned on other leafs including a border leaf.



FIGURE 13 Multitenancy

Ingress ReplicationAlthough host reachability information is exchanged over the control plane to drastically reduce flooding in a VLAN, certain situationsrequire the flooding of frames, as in traditional Ethernet networks such as but not limited to:

∙ MAC aging

∙ Silent hosts

∙ L2 multicast or broadcast

Ingress replication is a technique used to accommodate flooding in such cases by the VTEPs in IP fabric. Each VTEP for a givenVXLAN segment (or server VLAN) computes the list of VTEPs having the same segment using the IMR (Inclusive Multicast Route)routes. Whenever the VTEP must flood a frame in a VXLAN segment, it replicates the frame in hardware and unicasts the frame to eachof the VTEPs in the IMR list for that segment.

vLAG PairvLAG is the solution recommended for leaf-level redundancy. Server multihoming is supported only through vLAG behind two VTEPs.Multihoming to two separate VTEPs is not supported. In the validated design, we have two pairs of VTEPs in each PoD operating invLAG mode, and servers are dual-homed to these VTEPs with a port channel.

When the two leafs are in vLAG mode, they act as one logical VTEP or end point. As shown in Figure 14, both leafs are configured withthe same VTEP IP address. From other VTEPs in the network, this pair appears as a single VTEP. This is very important because havingtwo physical switches in this mode on each rack does not result in an increased number of VTEPs or additional tunneling requirementson other VTEPs in the network.



FIGURE 14 Active-Active vLAG



IP Fabric Validated Designs∙ Pervasive eBGP.................................................................................................................................................................................................................33∙ iBGP Within a PoD and eBGP Between PoDs............................................................................................................................................... 34∙ Hardware and Software Matrix.................................................................................................................................................................................. 35∙ Brocade IP Fabric Configuration..............................................................................................................................................................................35∙ Network Virtualization with BGP EVPN................................................................................................................................................................57∙ Illustration Examples........................................................................................................................................................................................................87

This section provides the details of key deployment models with the validated configuration templates. Brocade validated designrecommends two models for the IP fabric deployment; these deployment models are categorized based on how the underlay isdesigned for interconnecting leaf, spine, super-spine, and border-leaf nodes. The first deployment model uses pervasive eBGP for theIPv4 underlay and EVPN peering. The second deployment model uses iBGP for the IPv4 underlay and EVPN peering within the PoDwith two spines as route-reflectors and eBGP for interconnecting the PoDs.

Pervasive eBGPThe design shown in Figure 15 uses eBGP as the control plane protocol between the layers of nodes, and each leaf is in its ownautonomous system. This design using eBGP as a routing protocol within the data center is based on the IETF draft: Use of BGP forrouting in large-scale data centers.[2] By adding the VXLAN EVPN control plane, this design is extended to support Layer 2 extensionand Layer 3 multitenancy in the fabric.

Figure 16 shows the design for a 3-stage IP fabric using eBGP as the control protocol. Note that the border leafs are connected to thespines in this design.

FIGURE 15 Pervasive eBGP in an Optimized 5-Stage IP Fabric


FIGURE 16 Pervasive eBGP in a 3-Stage IP Fabric

iBGP Within a PoD and eBGP Between PoDsThe design shown in Figure 17 uses iBGP as the control plane protocol within a PoD and eBGP between PoDs and super-spines.

FIGURE 17 iBGP Within a PoD and eBGP Between PoDs in an Optimized 5-Stage IP Fabric

IP Fabric Validated Designs


Hardware and Software MatrixTABLE 1 Platforms Used in This Validated Design

Places in the Network Brocade Platform Software Version

Leaf Nodes VDX 6740

VDX 6940-144S

Network OS 7.0.1

Spine Nodes VDX 6940-36Q Network OS 7.0.1

Super-Spine Nodes VDX 8770-4 Network OS 7.0.1

Edge or Border Leaf VDX 6940-36Q Network OS 7.0.1

WAN Edge Router MLXe-8 NetIron 5.9ba

TABLE 2 All Brocade Switch Platforms That Support IP Fabric

Places in the Network Brocade Platform Software Version

Leaf Nodes VDX 6740

VDX 6940-36Q

VDX 6940-144S

Network OS 7.0.1

Spine Nodes VDX 6940-36Q

VDX 8770-4

VDX 8770-8

Network OS 7.0.1

Super-Spine Nodes VDX 6940-36Q

VDX 8770-4

VDX 8770-8

Network OS 7.0.1

Edge or Border Leaf VDX 6940-36Q Network OS 7.0.1

WAN Edge Router MLXe-8 NetIron 5.9ba

Brocade IP Fabric ConfigurationThis section covers the aspects of provisioning and validation of the IP fabric network topology. The IPv4 fabric underlay alone issufficient for data centers where multitenancy or Layer 2 extension is not a requirement. In this case, the server VLANs or subnets maybe advertised directly into BGP to establish connectivity between the racks and PoDs in the data center and to external networks.

Node ID ConfigurationThe VDX platforms used as leaf, spine, and super-spine nodes are enabled with VCS ID 1 by default. Since these nodes will beindependent in IP fabric, we must ensure that they do not form a VCS fabric between them. This is achieved by configuring a uniqueVCS ID on each node.

In the validated design, each node—spine, leaf, super-spine, and edge leaf—is configured with a unique VCS ID. The RBridge ID may bere-used. We recommend using RBridge ID 1 for individual leafs and using RBridge IDs 1 and 2 for the vLAG pair.



Enable Virtual-Fabric on all leafs and edge leafs:

The vLAG pair is assigned its own unique VCS ID, and each node in the vLAG pair has a separate RBridge ID. For example, in thevalidated design, Leaf1 is a 2-node vLAG pair.

vLAG peer 1:

vLAG peer 2:

Verify the configuration:

From the primary node of the vLAG pair, enable virtual fabric. For instance, as shown above, RBridge 2 is the primary node in the Leaf1vLAG pair.



IP Fabric Infrastructure LinksAll nodes in the IP fabric—leafs, spines, and super-spines—are interconnected with Layer 3 interfaces. In the validated design,

∙ 40-G links are used between the nodes.

∙ All these links are configured as Layer 3 interfaces with /31 IPv4 address.2 For a simple 3-stage fabric, IP unnumberedinterfaces can be used. We do not recommend a mix of unnumbered and numbered interfaces within a fabric. Also for a 5-stage IP fabric, numbered interfaces are highly recommended.

∙ The MTU for these links is set to Jumbo MTU. This is a requirement to handle the VXLAN encapsulation of Ethernet frames.

∙ Disable the fabric ISL and trunk features.

Loopback InterfacesEach leaf and border leaf needs a loopback interface with a unique IPv4 address to use as the VTEP IP. This is not required on spinesand super-spines. This step may be skipped if VXLAN EVPN overlay is not used in the IP fabric.

Each device in the fabric needs one loopback interface with a unique IPv4 address for the purpose of router ID.

2 An IP unnumbered interface is another variation that can be used for the fabric links. This interface may be used in a 3-stage fabric. Refer to the"Deployment Model-1: eBGP Underlay Configuration for 3-Stage Clos Fabric" section.



Configure the IP router ID using the IP address of the loopback 2 interface.

Server-Facing Links

Individual Leaf/ToRThe server-facing or access links are on the leaf nodes. In the validated design:

∙ 10-G links are used for server-facing VLANs.

∙ These links are configured as Layer 2 trunks with VLANs associated.

∙ The MTU for these links is set to the default: 1500 bytes.

∙ Disable fabric ISL and trunk features.

∙ Spanning tree is disabled.3

vLAG Pair/ToRvLAG configuration involves three steps:

∙ Node ID configuration on the pair of devices.

∙ Inter-switch links or ISL configuration on both devices.

∙ Configuring the server-facing port channels and adding the required VLANs on them.

Node ID Configuration on the vLAG PairRefer to the "Node ID Configuration" section earlier in this chapter for assigning the node ID to the vLAG pair.

∙ Pod1-Leaf1-1, rbridge-id 1

3 If there are L2 switches or bridges between a leaf and servers, spanning tree must be enabled. If there is a possibility of enabling bridges inadvertentlyunder the leaf nodes, we recommend enabling spanning tree and configuring the server ports as edge ports.

POD1-Leaf3(conf-if-te-1/0/4)# spanning-tree autoedge



∙ Pod1-Leaf1-2, rbridge-id 2

ISL ConfigurationAs shown in the illustration below, the vLAG pair is interconnected by two 40-G Ethernet ports for ISL.

Server Port-Channel ConfigurationIn the configuration shown below, port channel 113 is configured as a vLAG.



Deployment Model-1: eBGP Underlay Configuration for Optimized 5-Stage Clos FabricKey points to consider as design principle for eBGP as IPv4 underlay. Refer to Figure 15 for the topology information.

∙ Each leaf is in a private AS.

∙ The vLAG pair (Dual-Leaf) is considered as one leaf; both devices in the pair are in the same private AS.

∙ All spines within a PoD are in one private AS.

∙ All super-spines are in one private AS.

∙ All border leafs are in one private AS.

∙ eBGP peering with MD5 authentication is used between the layers of nodes.

∙ BFD is enabled on each link with BGP as the client installing the BFD session between the neighbors. We recommend that youuse the default BFD timers.

∙ Two spines are designated to advertise the EVPN Address Family.

∙ Two super-spines are designated to advertise the EVPN Address Family.



Spine ConfigurationAll spines within a PoD have a similar configuration for IPv4 underlay. Peer groups are used to simplify the configurations and also forefficiency in BGP update processing.

∙ Configure the directly connected leafs' IP addresses in one peer group: leaf-group.

∙ Configure the directly connected super-spine IPs into another peer group: super-spine-group.

∙ Enable MD5 authentication and BFD to all peers.

Each spine should establish IPv4 Address Family peering with all leafs inside the PoD and super-spines. (Note that when verifying thepeerings, leafs in a vLAG pair share one common AS number between them, and super-spines belong to one AS number.)



Check the BFD adjacency with every connected device.

Leaf ConfigurationAll leafs within a PoD have a similar configuration for IPv4 underlay. Peer groups are used to simplify the configuration and also forefficiency in BGP update processing.

∙ Configure the directly connected IP addresses of the spines into a peer group: spine-group.

∙ Enable MD5 authentication to the peer group.

∙ Enable BFD to the peer group.

∙ Advertise the VTEP IP address if EVPN overlay needs to be provisioned.

∙ For IP fabric implementations without overlay EVPN, advertise server subnets as appropriate using either a network statementor a redistribute connected statement under the IPv4 Unicast Address Family.



Check the BGP neighbors. The leaf must be peering with all spines within the PoD for IPv4 Address Family route exchange.

BFD neighbors.



Check the route table to see the paths to other VTEP IPs in the fabric. For instance, in the table below taken from a leaf, it sees 4 paths(due to 4 spines) to every other VTEP IP in the fabric—both inside the PoD and the VTEPs in another PoD.

Super-Spine ConfigurationThis is applicable to all super-spines to exchange only IPv4 underlay routes. Peer groups are used to simplify the configuration.

∙ Create a peer group for each PoD:

– pod1_spine-group—Add the directly connected neighbor addresses of all spines in PoD1 to this group.– pod2_spine-group—Add the directly connected neighbor addresses of all spines in PoD2 to this group.

∙ Create a separate peer group for the edge leafs: edge-group. Add the directly connected neighbor addresses of edge leafs tothis group.

∙ Enable MD5 authentication to all peer groups.

∙ Enable BFD to all peer groups.



Each super-spine should be peering with four spines per PoD and two edge leafs for IPv4 Address Family route exchange.



BFD session with each BGP peer.

Border/Edge Leaf ConfigurationThe configuration of edge or border leafs is similar to that of leafs. They peer with the super-spines instead of spines.

∙ Configure a peer group superspine-group. Add the directly connected neighbor addresses of the super-spines to the group.These super-spines exchange only IPv4 routes.

∙ Enable MD5 authentication.

∙ Enable BFD.


∙ Optionally, advertise external networks directly into IPv4 underlay routing (for an IP fabric without EVPN overlays).

Deployment Model-1: eBGP Underlay Configuration for 3-Stage Clos FabricRefer to Figure 16 for the topology information. The underlay routing configuration for a 3-stage fabric is very similar to that of the5-stage fabric with the exception of peering to super-spines by spines and border leafs. Border leafs are directly connected to spines.



A 3-stage fabric may be built using either numbered or unnumbered fabric interfaces. This section explains building a 3-stage fabricwith unnumbered interfaces. (For numbered interfaces, refer to the "IP Fabric Infrastructure Links" section and the 5-stage deploymentmodel.)

Key points to note:

∙ Fabric links are configured as unnumbered interfaces.

∙ Each leaf is in a private AS.

∙ The vLAG pair is considered as one leaf; both devices in the pair are in the same private AS.

∙ All spines within the PoD are in one private AS.

∙ All border leafs are in one private AS.

∙ eBGP multihop peering is established over loopback interface IP addresses with MD5 authentication.

∙ BFD sessions are established on the links between the layers of nodes.

Fabric Infrastructure Links—UnnumberedThe IP unnumbered option for fabric interfaces4 significantly simplifies the fabric provisioning for a 3-stage fabric.

∙ No IP addressing scheme is needed for the links between the nodes. Each node is represented by just one IP address or routerID.

∙ The unnumbered interfaces are associated with a numbered loopback interface on the switch. This loopback interface’s IPaddress is used as the source address for BGP peering. This IP address is exchanged over LLDP between the nodes. Thiseliminates the need to run an IGP or static routing to reach the neighbor’s loopback address for BGP peering.

In the “Loopback Interfaces” section, we configured two loopback interfaces on each node to be used as the router ID. One of them isused as the router ID. The unnumbered interfaces are associated with this loopback interface, i.e. Loopback 2. For example, on Leaf1:

Verify the neighbor discovery over the link using LLDP. Also verify the reachability to the loopback interface of the neighbor connectedover this unnumbered link. For instance, a link between the nodes Leaf1-1 and Spine2:

4 Note that a 3-stage fabric can also be built using numbered fabric interfaces. We do not recommend having a mix of both numbered andunnumbered interfaces within a fabric. For a 5-stage IP fabric, we highly recommend using numbered interfaces.



Spine ConfigurationAll spines within a PoD have a similar configuration for IPv4 underlay. Peer groups are used to simplify the configurations and also forefficiency in BGP update processing.

∙ Configure the leafs' router IDs in one peer group: leaf-group.

∙ Configure the edge leafs' router IDs in one peer group: edge-group.

∙ Enable eBGP multihop, MD5 authentication, and BFD to both peer groups.

∙ Set the BGP peering source interface to the loopback interface (used as router ID).



Leaf ConfigurationAll leafs within a PoD have a similar configuration for IPv4 underlay. Peer groups are used to simplify the configuration and also forefficiency in BGP update processing.

∙ Configure the spines’ router ID loopback IP addresses into a peer group: spine-group.


∙ Set the BGP peering source interface to the loopback interface (used as the router ID).


∙ For IP fabric implementations without overlay EVPN, advertise server subnets as appropriate using either a network statementor a redistribute connected statement under IPv4 Unicast Address Family.



Border/Edge Leaf ConfigurationEdge or border leafs peer with the spines and exchange both IPv4 and EVPN routes.

∙ Configure the spines’ router ID loopback IP addresses into a peer group: spine-group.

∙ Enable eBGP multihop, MD5 authentication, and BFD to the peer group.

∙ Set the BGP peering source interface to the loopback interface (used as the router ID).

∙ Advertise the VTEP IP if EVPN overlay needs to be provisioned.

∙ Optionally, advertise external networks directly into IPv4 underlay routing (for an IP fabric without EVPN overlays).



Deployment Model-2: iBGP Underlay Configuration for Optimized 5-Stage Clos FabricKey points to consider as a design principle for iBGP as IPv4 underlay (refer to Figure 16 for topology information):

∙ Each PoD is in one private AS.

∙ iBGP is used as the underlay within a PoD.

∙ eBGP routes are exchanged between the PoDs and border leafs through super-spines.

∙ In each PoD, all four spines act as the IPv4 RR to leafs.

∙ In each PoD, only two spines act as the EVPN RR to leafs.

∙ Use peer groups to group neighbors into IPv4 only and IPv4+EVPN speakers.

Spine ConfigurationAll spines within a PoD have a similar configuration for IPv4 underlay. Peer groups are used to simplify configuration and also forefficiency in BGP update processing.

∙ Configure the directly connected leafs' IP addresses in one peer group: leaf-group.

∙ Configure the directly connected super-spine IPs into another peer group: super-spine-group.

∙ All spines should have one cluster ID since they are IPv4 route reflectors to leafs.




Each spine should establish IPv4 Address-Family peering with all leafs inside the PoD and all super-spines.



Leaf ConfigurationAll leafs within a PoD have a similar configuration for IPv4 underlay. Peer groups are used to simplify the configuration.

∙ Configure the directly connected IP addresses of the spines into a peer-group spine-group.

∙ Enable MD5 authentication to the peer group.

∙ Enable BFD to the peer group.

∙ Advertise the connected networks.

Each leaf should establish IPv4 Address-Family peering with four spines inside the PoD.



Super-Spine ConfigurationSuper-spines have a similar configuration for IPv4 underlay. Peer groups are used to simplify the configuration.

∙ Create two peer groups for each PoD, one group to exchange IPv4 routes and another group to exchange both IPv4 andEVPN routes:

– pod1-spine-ip-group—Two spines in each PoD support only IPv4 routes. Add the directly connected neighbor addressesof these two spines to this group.

– pod1-spine-evpn-group—Two spines in each PoD support both IPv4 and EVPN routes. Add the directly connectedneighbor addresses of these two spines to this group.

– Similar configuration for PoD2 and other PoDs.

∙ Create a separate peer group to the edge PoD. Add the directly connected neighbor addresses of edge leafs to this group.





Each super-spine should be peering with four spines per PoD and two edge leafs for the IPv4 Address Family.



Border/Edge Leaf ConfigurationEdge leafs peer with the super-spines and exchange both IPv4 and EVPN routes. So one peer group is sufficient.

∙ Configure a peer group, and add the directly connected neighbor addresses of the super-spines to the group.

– Enable MD5 authentication.– Enable BFD.

∙ Activate the peer group for the IPv4 Address Family.




Overlay Gateway ConfigurationFollowing are the steps involved in configuring the overlay gateway or VTEP on a leaf and border leaf.

∙ Create an overlay gateway, and assign it a name.

∙ Enable Layer 2 extension.

∙ Associate the loopback interface whose IPv4 address is used as the VTEP IP.

∙ Associate the rbridge-id of the leaf switch.

∙ Map the VLANs to the VNI number. In this validated design, we're using the auto mapping of VLAN to a VNI. For instance,VLAN 2001 is mapped to VNI 2001. (This simplified mapping option should work for most implementations unless there is aspecific requirement to map the server VLAN range to a specific VNI range in the VXLAN domain.)

Deployment Model-1: eBGP EVPN Configuration for Optimized 5-Stage Clos FabricThis configuration is applicable to the model shown in Figure 15, where eBGP is used as the control protocol for underlay.



BGP Underlay ConfigurationWhen enabling network virtualization with EVPN overlay, the underlay configuration needs a few changes to accommodate the BGPpeers that exchange only IPv4 routes and the BGP peers that exchange both IPv4 and EVPN routes. This is accomplished by usingBGP peer-groups. In the 5-stage fabric:

∙ Two spines in each PoD exchange only IPv4 Address-Family routes.

∙ Two spines in each PoD exchange both IPv4 and EVPN Address-Family routes—referred to as EVPN spines.

∙ Two super-spines exchange only IPv4 Address-Family routes.

∙ Two super-spines exchange both IPv4 and EVPN Address-Family routes—referred to as EVPN super-spines.

Leaf ConfigurationThis is applicable to all Leafs. With EVPN control-plane, the configuration needs to accommodate the exchange of EVPN routes onlywith two designated spines. Peer-groups are used to simplify the configuration and also for efficiency in BGP update processing.

∙ Configure the directly connected IP addresses of the spines into two peer-groups—spine-evpn-group and spine-ip-group. Thisis required because only 2 spines exchange EVPN routes, but all 4 spines exchange ipv4 routes. (Refer to the "NetworkVirtualization with BGP EVPN" for EVPN Address-Family configuration.) For simple IP fabric implementation, this may beignored and all spines can be added to one peer group.

∙ Enable MD5 authentication to both peer groups.

∙ Enable BFD to both peer groups.

∙ Enable the IPv4 Address-Family, and advertise the VTEP IP address.

Spine ConfigurationThis is applicable to the two spines designated to exchange only IPv4 routes with leafs and super-spines.



∙ Configure the directly connected leafs IP addresses in one peer group leaf-group.

∙ Configure the directly connected super-spine IPs into another peer group super-spine-group.


∙ Enable the IPv4 Address-Family.

EVPN Spine ConfigurationThis is applicable only on the two spines designated to exchange IPv4 and EVPN routes.

∙ Configure the directly connected leafs IP addresses in one peer-group leaf-group.

∙ Configure the directly connected super-spine IPs into two peer-groups superspine-ip-group and superspine-evpn-group. Thesecond group will contain only the two super-spines designated to exchange IPv4 and EVPN routes.

∙ Enable MD5 authentication to all peers.

∙ Enable BFD to all peers with default timer values.




Super-Spine ConfigurationThis is applicable to two super-spines designated to exchange only IPv4 underlay routes. Peer-groups are used to simplify theconfiguration.

∙ Create a peer-group for each PoD:

– pod1_spine-group—Add the directly connected neighbor addresses of all spines in PoD1 to this group.– pod2_spine-group—Add the directly connected neighbor addresses of all spines in PoD2 to this group.

∙ Create a separate peer-group for the Edge leafs—edge-group. Add the directly connected neighbor addresses of edge leafs tothis group.






EVPN Super-Spine ConfigurationThis is applicable only on the super-spines designated to exchange both IPv4 and EVPN routes. This can be skipped for the IP fabricimplementation without the EVPN control-plane.

∙ Create two peer-groups for each PoD, one group to exchange only IPv4 routes and the other group to exchange both IPv4 andEVPN routes. For simple IP fabric implementation, this may be ignored and all spines in a PoD can be added to one peer-group.

– pod1_spine-ip-group—Two spines in each PoD support only IPv4 routes. Add the directly connected neighbor addressesof these two spines to this group.

– pod1_spine-evpn-group—Two spines designated in each PoD support both IPv4 and EVPN routes. Add the directlyconnected neighbor addresses of these two spines to this group.


∙ Create a separate peer-group for the Edge leafs—edge-group. Add the directly connected neighbor addresses of Edge leafs tothis group.

∙ Enable MD5 authentication and BFD to all peer-groups.




Border/Edge Leaf ConfigurationThe configuration of edge or border leafs is similar to that of leafs. They peer with the super-spines. They exchange IPv4 routes with allsuper-spines and EVPN routes with two designated super-spines.

∙ Configure a peer group superspine-ip-group. Add the two directly connected neighbor addresses of the two super-spines tothe group. These super-spines exchange only IPv4 routes.

∙ Configure another peer group superspine-evpn-group. Add the two designated super-spine addresses to this group. Thesesuper-spines exchange both IPv4 and EVPN routes. For simple IP fabric implementation, this step may be skipped and allsuper-spine neighbors may be added to just one peer group.



∙ Enable MD5 authentication and BFD to all peer groups.

∙ Enable the IPv4 Address-Family, and advertise the VTEP IP address.

BGP Overlay Configuration

Leaf ConfigurationThis configuration is applicable to all leafs in each of the PoDs. They exchange EVPN routes with two designated spines in theirrespective PoDs.

∙ Enable the EVPN Address-Family.

∙ Activate the designated EVPN spines under EVPN Address-Family. (Use the peer-group already configured in the underlayconfiguration.)

∙ Enable the "allowas-in 1" feature on vLAG leafs to facilitate learning of the routes between the vLAG peers. This is a requirementbecause the vLAG pair is in the same AS number. This is the case in the pervasive eBGP model of underlay.

∙ When EVPN routes are advertised into eBGP by a node, the next hop is set to its peering address. This follows the standardBGP behavior. The next hop should always point to the IP address of the VTEP that originated these routes. Enable the "next-hop unchanged" configuration to the peers.



All leafs should see two EVPN neighbors. (Two spines participate in EVPN route exchange.)

EVPN Spine ConfigurationThis is applicable only to the two spines in each PoD designated to exchange the EVPN routes with leafs and super-spines.


∙ Activate the leaf group already created in the underlay configuration into the EVPN Address-Family.

∙ Activate the superspine-evpn-group into the EVPN Address-Family.




Each EVPN spine will establish EVPN Address-Family adjacency with all leafs inside the PoD and two designated super-spines.

EVPN Super-Spine ConfigurationThis is applicable to the super-spines designated for the EVPN route exchange in the fabric with spines and edge leafs.


∙ Activate the spine-evpn-group peer groups of each PoD into the EVPN Address-Family.

∙ Activate the edge leafs peer group into the EVPN Address-Family.




Each super-spine has two spines in each of the PoDs and two border leafs as EVPN Address-Family neighbors.

Border/Edge Leaf ConfigurationThis is applicable to all border leafs in the fabric.


∙ Activate the superspine-evpn-group peer groups into the EVPN Address-Family.

∙ When EVPN routes are advertised into eBGP by a node, the next hop is set to its peering address. This follows standard BGPbehavior. The next hop should always point to the IP address of the VTEP that originated these routes. Enable the "next-hopunchanged" configuration to the peers.



Each border leaf establishes EVPN peering with two super-spines.

Deployment Model-1: eBGP EVPN Configuration for 3-Stage Clos FabricThis configuration is application to the deployment model shown in Figure 16, where eBGP is used as the underlay routing protocol in a3-stage Clos fabric.

BGP Underlay ConfigurationWhen enabling network virtualization with EVPN overlay, the underlay configuration needs a few changes to accommodate the BGPpeers that exchange only IPv4 routes and the BGP peers that exchange both IPv4 and EVPN routes. This is accomplished by usingBGP peer groups.

∙ Two spines exchange only IPv4 Address-Family routes.

∙ Two spines exchange both IPv4 and EVPN Address-Family routes.

Leaf ConfigurationThis is applicable to all leafs. With the EVPN control plane, the configuration needs to accommodate the exchange of EVPN routes onlywith two designated spines. Peer groups are used to simplify the configuration and also for efficiency in BGP update processing.

∙ Configure the router ID loopback IP addresses of the spines into two peer groups: spine-evpn-group and spine-ip-group. Thisis required because only two spines exchange EVPN routes, but all four spines exchange IPv4 routes.


∙ BGP peering source interface set to loopback interface (used as the router ID).



∙ Enable the IPv4 Address Family, and advertise the VTEP IP address.

Spine ConfigurationThis is applicable to all spines inside a PoD.

∙ Configure the router ID loopback IP addresses of the leafs in one peer-group leaf-group.

∙ Configure the router ID loopback IP addresses of the edge leafs' IPs into a peer-group edge-group.



∙ Enable the IPv4 Address Family.



Border/Edge Leaf ConfigurationThe configuration of edge or border leafs is similar to that of leafs. They peer with the spines. They exchange IPv4 routes with all spinesand EVPN routes with two designated spines.

∙ Configure a peer group spine-ip-group. This group consists of the router IDs of spines that exchange only IPv4 routes.

∙ Configure another peer group spine-evpn-group. This group consists of router IDs of spines that exchange both IPv4 andEVPN routes.






Leaf ConfigurationThis is applicable to all leafs.

∙ Activate the designated EVPN spines under the EVPN Address Family (use the peer group already configured in the underlayconfiguration).

∙ Enable the "allowas-in 1" feature on vLAG leafs to facilitate learning of the routes between the vLAG peers. This is a requirementbecause the vLAG pair is in the same AS number. This is the case in the pervasive eBGP model of underlay.

∙ When EVPN routes are advertised into eBGP by a node, the next hop is set to its peering address. This follows standard BGPbehavior. The next hop should always point to the IP address of the VTEP that originated these routes. Enable the next-hopunchanged configuration to the peers.



As shown below (with the show ip bgp summary command), there are four neighbors for IPv4 AFI. Of these four neighbors, two arelisted as neighbors for EVPN AFI (show bgp evpn summary). In other words, all four spines exchange IPv4 routes, and only twoexchange EVPN routes.

EVPN Spine ConfigurationThis is applicable only to the two spines designated to exchange EVPN routes with leafs and edge leafs.

∙ Enable the EVPN Address Family.

∙ Activate the leaf-group peer group into the EVPN Address Family.

∙ Activate the edge-leaf's peer group into the EVPN Address Family.



Border/Edge Leaf ConfigurationThis is applicable to all edge leafs. Activate the EVPN route exchange with the designated spines for EVPN.

Deployment Model-2: iBGP EVPN Configuration for Optimized 5-Stage Clos FabricThis configuration is applicable to the deployment model shown in Figure 16, where iBGP is used as the underlay routing protocol withina PoD.

BGP Underlay ConfigurationWhen enabling network virtualization with EVPN overlay, the underlay configuration needs a few changes to accommodate the BGPpeers that exchange only IPv4 routes and the BGP peers that exchange both IPv4 and EVPN routes. This is accomplished by usingBGP peer groups. In the 5-stage fabric using iBGP inside a PoD:

∙ All spines exchange IPv4 routes with leafs and super-spines.

∙ All spines act as the route reflector to all leafs inside their PoD for IPv4 Address-Family routes.

∙ Two spines are designated to exchange EVPN routes with leafs and super-spines. These are referred to as EVPN spines.

∙ EVPN spines act as the route reflector to all leafs inside their PoD for EVPN Address-Family routes.

∙ Two super-spines are designated to exchange EVPN routes with spines in each PoD and border leafs. These are referred to asEVPN super-spines.



Spine ConfigurationThis configuration is applicable to the spines in each POD that exchange only IPv4 routes with leafs and super-spines. Peer groups areused to simplify configuration and also for efficiency in BGP update processing.

∙ Configure the directly connected leaf IP addresses in one peer group leaf-group.

∙ Configure the directly connected super-spine IPs into another peer group super-spine-group.


∙ All spines to have one cluster ID.

∙ Enable IPv4 Address-Family, redistribute connected-routes.

∙ Enable IPv4 Address-Family route reflection to all leafs in leaf-group.

Each spine should establish IPv4 Address-Family peering with all leafs inside its PoD and all super-spines.



EVPN Spine ConfigurationThis is applicable only on the two spines designated to exchange IPv4 and EVPN routes with leafs and super-spines.

∙ Configure all leafs in a peer group leaf-group.

∙ Configure the directly connected super-spine IPs into two peer groups superspine-ip-group and superspine-evpn-group. Thesecond group will contain only those two super-spines designated to exchange IPv4 and EVPN routes.


∙ All spines are to have one cluster ID.

∙ Enable IPv4 Address-Family; redistribute connected routes.

∙ Enable IPv4 Address-Family route reflection to all leafs in leaf-group.



Leaf ConfigurationThis is applicable to all leafs in a PoD. Peer groups are used to simplify the configuration.

∙ Configure the directly connected IP addresses of the spines into two peer groups: spine-evpn-group and spine-ip-group. Thisis required because only two spines exchange EVPN routes, but all four spines exchange IPv4 routes.

∙ Enable MD5 authentication to both peer groups.

∙ Enable BFD to both peer groups.


∙ Advertise the connected networks.



Each leaf should establish IPv4 Address-Family peering with all inside the PoD.

Super-Spine Configuration

This is applicable to super-spines that exchange only IPv4 routes with spines in each PoD.

∙ Create one peer group for each PoD.

– pod1_spine-group—All spines in PoD1 and exchange only IPv4 routes. Add the directly connected neighbor addresses ofthese two spines to this group.

– pod2_spine-group—All spines in PoD2 and exchange only IPv4 routes. Add the directly connected neighbor addresses ofthese two spines to this group.

∙ Create a separate peer group to the Edge PoD—edge-group. Add the directly connected neighbor addresses of the edge leafsto this group.





Each super-spine should be peering with four spines per PoD and two edge leafs for the IPv4 Address-Family.



EVPN Super-Spine ConfigurationThis is applicable to the super-spines designated to exchange both IPv4 and EVPN routes with spines in each PoD and edge leafs.

∙ Create two peer groups for each PoD: one group to exchange IPv4 routes and another group to exchange both IPv4 andEVPN routes:

– pod1-spine-ip-group—Two spines in each PoD support only IPv4 routes. Add the directly connected neighbor addressesof these two spines to this group.

– pod1-spine-evpn-group—Two spines in each PoD support both IPv4 and EVPN routes. Add the directly connectedneighbor addresses of these two spines to this group.


∙ Create a separate peer group to the edge PoD—edge-group. Add the directly connected neighbor addresses of edge leafs tothis group.





Each super-spine should peer with four spines per PoD and two edge leafs for the IPv4 Address-Family.



Border/Edge-Leaf ConfigurationThe configuration of border or edge leafs is similar to that of leafs. But they peer with the super-spines. They exchange IPv4 routes withall super-spines and EVPN routes with two designated super-spines.

∙ Configure a peer group superspine-ip-group. Add two directly connected neighbor address of two super-spines to the group.These super-spines exchange only IPv4 routes.

∙ Configure another peer-group superspine-evpn-group. Add the two designated super-spine addresses to this group. Thesesuper-spines exchange both IPv4 and EVPN routes. For simple IP fabric implementation, this step may be skipped and allsuper-spine neighbors may be added to just one peer group.


∙ Enable the IPv4 Address-Family and advertise the VTEP IP address.



The border leaf should establish IPv4 peering with all super-spines.


Leaf ConfigurationThis is applicable to all leafs in each PoD.


∙ Activate the designated EVPN spines under the EVPN Address-Family. (Use the peer group already configured in the underlayconfiguration.)



∙ Enable the "allowas-in 1" feature on vLAG leafs to facilitate learning of the routes between the vLAG peers. This is a requirementbecause the vLAG pair is in the same AS. This is the case in the pervasive eBGP model of underlay.

EVPN Spine ConfigurationThis is applicable only to the two spines designated to exchange the EVPN routes with leafs and super-spines.


∙ Activate EVPN super-spines under the EVPN Address-Family.

∙ Activate all leafs under the EVPN Address-Family.

∙ Act as the route reflector of the EVPN Address-Family to the leafs peer group.

EVPN Super-spine ConfigurationThis configuration is applicable to the super-spines designated to exchange both IPv4 and EVPN routes with spines in each PoD.


∙ Activate EVPN spines in each PoD under the EVPN Address-Family.



∙ Activate all edge leafs under the EVPN Address-Family.

Border/Edge Leaf ConfigurationThis is applicable to all border leafs. They exchange EVPN routes with two designated super-spines.


∙ Activate EVPN super-spines under the EVPN Address-Family.

Tenant ProvisioningTenant provisioning refers to the configuration on leafs to enable server VLANs and network connectivity to tenant VRF contexts andmapping these VLANs and VRFs to the overlay control and forwarding planes to establish Layer 2 extension and multitenancy. This isapplicable to both 3-stage and 5-stage Clos fabrics.

Enable Conversational Learning of MAC EntriesThis is applicable to all leafs in the fabric for conservation of L2 forwarding table space.



Anycast Gateway MAC ConfigurationAnycast gateway MAC configuration is applied to all leafs (except edge leafs) in the data center. This is used as the gateway MAC orrouter MAC for all server-facing subnets. This enables seamless workload move within and across the PoDs in the data center. Werecommend setting the U/L bit to 1 in the MAC address to indicate that it is a locally administered MAC address and not to conflict withany real MAC addresses.

The MAC addresses must be different for IPv4 and IPv6, but the OUI portion (first three bytes) must be same.

Enable Conversational Learning of ARP/ND Host EntriesThis is required on all leafs and edge leafs.

VRFs, Server VLANs, and Subnets ConfigurationFollowing are the steps involved in tenant VRF configuration.

1. Assign a unique RD. Every tenant must have a unique RD value per leaf/ToR where it is provisioned. In the validated design, weare using the following format: IPv4_Address:nn where

∙ IPv4_Address is the router ID of the VTEP.

∙ nn is a unique number for the tenant VRF. This value is re-used on other leafs as well where the same tenant isprovisioned.

For example, vrf201 has the following RD values on leafs where it is provisioned.

– On leaf1: 10.121.1.11:201– On leaf5: 10.121.1.51:201– On border-leaf1: 10.123.4.1:201

2. Assign a unique L3 VNI number.

3. Assign import and export route targets for IPv4 and IPv6 tenant routes.

In the configuration templates below, the following tenant profile is enabled on a leaf:

Configure Tenant VRF Profile:

∙ Name: vrf101



∙ L3 VNI: 7101

∙ IPv4 and IPv6: enabled

∙ Route-target 101:101

∙ Server-facing VLAN 2001

Assign Layer 3 Interface for the L3 VNI of the Tenant VRF:

This is the routing interface for the Integrated Routing and Bridging (IRB) operation on the leaf.

Assign Server-Facing VLAN:

Assign VE (L3) Interface for the Server-Facing VLAN:



Advertise Tenant Layer 3 Routes from the LeafIPv4

IPv6

Enable the EVPN Instance for the Tenant VLAN SegmentsOnce the server-facing VLANs are created and mapped to VNI segments on the leaf, those VNI segments must be enabled into thecontrol plane. As was done for the tenant VRF, the VNI segments also require an RD (route distinguisher) and an RT (route target). This isalso defined as the MAC-VRF and enables learning remote MAC addresses when the same VLAN segment is extended to other leafs orVTEPs in the fabric.

The RD and RT configuration is set to auto in this design for simplicity and may be followed for most of the deployments. Advancedusers may define a different scheme of RD and RT. A user-defined RD/RT is not covered in this document.



vLAG Pair ConfigurationA vLAG pair or redundant ToR requires a few additional configuration steps:

∙ Same VTEP IP

∙ Separate or unique router IDs

The configuration of two leafs in a dual-ToR vLAG pair is shown side-by-side for comparison. (Please note that the configuration forboth switches in the vLAG pair can be done from the primary node.)

∙ The Loopback1 interface has the same IP address on both nodes; this is used as the VTEP IP under overlay gateway.

∙ The Loopback2 interface has a unique IP address on each node; this is used as the IP router ID for the node.

∙ Attach both RBridge IDs under the overlay gateway.

Illustration ExamplesIn this section we illustrate the use cases by using sections of the validated design network topology as appropriate. This will help thereader to further understand the deployment scenarios.

Example-1: Tenant and L2 Extension Between Racks in a 3-Stage Clos FabricFigure 18 shows a section of the topology to illustrate the following with configuration and verification. Two racks are shown in thediagram.



∙ Rack1 has a redundant vLAG ToR, leaf1-1 and leaf1-2, referred to as leaf1 collectively.

∙ Rack5 has an individual ToR, leaf5.

∙ A tenant VRF vrf201 is provisioned on both racks.

∙ The tenant has two server VLANs mapped to VNIs 3001 and 3801.

∙ Server VLAN 3001 is extended between these two racks. VLAN/VNI 3001 is provisioned on both racks, and there are hosts onthese racks.

∙ Server VLAN 3801 is a VLAN provisioned on Rack1 only, but it belongs to the same tenant. Routing between VNI 3001 and3801 is required within this tenant both in the same rack and across the racks.

∙ This example also illustrates the symmetric and asymmetric routing operation.

The configuration on leafs is identical on each of the leafs except for the VTEP IP, router ID, and RD configurations. The vLAG pair isrepresented with one VTEP IP address. The use of anycast gateway addresses for the server-facing VLAN interfaces simplifies theconfiguration drastically. Please note that the configuration for the vLAG pair is done from the primary node.

FIGURE 18 Tenant and Layer 2 Extension Between Two Racks

Configuration

Check the Node ID on Each ToRThe RBridge ID is required for the Layer 3 and EVPN configuration on each node.

For the vLAG pair, Leaf1-2 is the primary node. The configuration for both devices in the pair is done from Leaf1-2. The RBridge IDs are45 and 46 for Leaf1-1 and Leaf1-2, respectively. These IDs are used for the ports and for the Layer 3 configuration.



Leaf5 is an individual ToR with an RBridge ID of 51.

Configuration on the Leaf1 vLAG PairThe configuration is shown in three parts for clarity. Common configuration such as port channel and VLANs are shown in one block.The tenant, Layer 3 interfaces, and BGP-EVPN configuration is shown in the second block under each RBridge ID. The commonoverlay-gateway configuration is shown in the third block. Please note that the entire configuration is applied from the primary node inthis two-node vLAG pair.

The configuration is pretty much the same except for the router ID and RD of the tenant VRF. This makes it easier to automate theprovisioning on various nodes.



Configuration on Leaf5



Verification

Verify VLAN Extension Between the RacksCheck the L2 extended VLAN on each node. This should show the local L2 trunk ports and also the tunnels to all remote VTEPs wherethe same VLAN segment is extended.

In the following output from the Leaf1 vLAG pair, there are five tunnels for VLAN 3001, which indicates that the same VLAN/VNIsegment is provisioned on five other VTEPs or ToRs. Note that one of the tunnels, Tu 61442, is destined to Leaf5. Also note that thereare four underlay next hops to reach this tunnel destination in the fabric.



In the following output shown from Leaf5, Tunnel 61441 is destined to the vLAG Leaf1 pair's VTEP IP: 10.121.1.1.



VLAN Layer 3 Interface State on the vLAG Pair



VLAN Layer 3 Interface State on the Leaf5 ToR

Local Host Entries on Each Leaf

Depending on the port-channel hashing on server-facing links, the ARP entries may be learned on any of the nodes in the vLAG pair.Make sure that all host entries are learned collectively in the vLAG pair.



Remote Host Entries in the Extended VLAN

BGP and ARP Table on Leaf5

The following table shows the BGP and ARP entries of the remote host behind the Leaf1 pair. Note that the next hop is set to 10.121.1.1,which is a common VTEP IP of the vLAG pair. This causes the redundant leaf to appear as one VTEP in the underlay network, and loadbalancing is accomplished.

In the ARP table, both the local and remote entries are indicated with different types. BGP EVPN for remote entries signify that they werelearned over BGP EVPN. The local entries are shown as "Dynamic" entries.

Verify Tenant Extension Between the RacksTenant extension ensures routing between the VXLAN segments within the same tenant.



As shown in Figure 18, VNI segment 3802 is provisioned only on the vLAG ToR but is part of the tenant on both ToRs. Let's go over alist of verification steps required to ensure that communication between the hosts in VNI 3001 on Leaf5 and hosts in VNI 3802 onvLAG Leaf1.

RMAC of Each Node

There is one RMAC assigned to every VTEP. This information can be obtained by looking at any of the L3 interfaces or the L3 VNI'sassociated VLAN interface. For the vLAG pair, even though they have same VTEP IP, they are assigned a unique router MAC.

L3 VNI State on the Nodes

L3 VNI 7201 is assigned to the tenant VRF. Make sure that the vLAG ToR and Leaf5 have tunnels established to each other and that thisVNI is activated on it.

As seen in the following table for the output from Leaf1, the tunnel source is the VTEP IP of the vLAG, 10.121.1.1, and the destination IP isthe VTEP IP of Leaf5, 10.121.1.5. (Notice additional tunnels in the list; these are destined to other VTEPs where the same tenant isprovisioned.)



L3 VNI state from Leaf5:



Verify the Route to the Remote Subnet of the Same Tenant

The following table shows the BGP entry on Leaf5 for the remote subnet of VNI 3802. (Note that the host entries are also advertisedover BGP, but will be ignored by Leaf5 since this VNI is not locally provisioned and only routing is desired.)

There are four entries in the BGP table: two originators in the vLAG pair, and those two entries are learned from two spines exchangingEVPN routes. Again, the next hop is the same due to the common VTEP IP used by the vLAG pair.



Example-2: Tenant and L2 Extension Between PoDs in an Optimized 5-Stage ClosFabricIn this example, we illustrate the extension of a tenant and a Layer 2 segment between racks in two different PoDs. As shown inFigure 19, tenant VRF vrf101 is extended between these two racks: POD1-leaf1 and POD2-leaf1 dual or vLAG pair. VXLAN segment 2001is extended across the PoD. VLAN 3901 is provisioned only on the Leaf1 pair in POD1.



FIGURE 19 Tenant and Layer 2 Extension Between Two PoDs Connected by Super-Spines

Configuration

Check the Node ID on Each ToRThe RBridge ID is required for the Layer 3 and EVPN configuration on each node.

For the POD1 vLAG pair, Leaf1-2 is the primary node. The configuration for both devices in the pair is done from Leaf1-2. The RBridgeIDs are 45 and 46 for Leaf1-1 and Leaf1-2, respectively. These IDs are used for the ports and for the Layer 3 configuration.

For the POD2 vLAG pair, Leaf1-2 is the primary node. The configuration for both devices in the pair is done from Leaf1-2. The RBridgeIDs are 45 and 46 for Leaf1-1 and Leaf1-2, respectively.



Configuration on the PoD1-leaf1 vLAG PairThe configuration is shown in three parts for clarity. Common configuration such as port channel and VLANs is shown in one block. Thetenant, Layer 3 interfaces, and BGP-EVPN configuration is shown in the second block under each RBridge ID. The common overlay-gateway configuration is shown in the third block. Please note that the entire configuration is applied from the primary node in this two-node vLAG pair.



Configuration on the POD2-leaf1 vLAG Pair



Verification

Verify VLAN Extension Between the NodesCheck the L2 extended VLAN on each node. This should show the local L2 trunk ports and also the tunnels to all remote VTEPs wherethe same VLAN segment is extended.

In the output below from the POD1-Leaf1 vLAG ToR, there are six tunnels for VLAN 2001, which indicates that the same VLAN/VNIsegment is provisioned on six other VTEPS or ToRs. Note that one of the tunnels, Tu 61448, is destined to the POD2-Leaf1 vLAG ToR.Also note that there are four underlay next hops to reach this tunnel destination in the fabric as there are four spines.



The output below from the POD2-Leaf1 vLAG shows the state of VLAN 2001.

VLAN Layer 3 Interface State on the POD1-Leaf1 vLAG



VLAN Layer 3 Interface State on the POD2-Leaf1 vLAG



Local Host Entries on Each Leaf/ToR


Remote Host Entries in the Extended VLAN

BGP and ARP Table on POD1-Leaf1

The following table shows a BGP entry and ARP entries of the remote hosts behind the POD2-leaf1 pair. Note that the next hop is set to10.122.2.1, which is the common VTEP IP of the vLAG pair. This causes the redundant leaf to appear as one VTEP in the underlaynetwork, and load balancing is accomplished.

In the ARP table, both local and remote entries are indicated with different types: "Dynamic" for local entries; and BGP-EVPN for remoteentries, signifying that they were learned over BGP EVPN.

10.107.1.20 and 10.107.1.21 are the local hosts. (Even though 10.107.1.21 is shown as remote, the MAC entry lookup makes it a local host inthe vLAG pair).

10.107.1.30 and 10.107.1.31 are the hosts attached to the POD2-Leaf1 pair.



BGP and ARP Table on POD2-Leaf1



Verify Tenant Extension Between the RacksTenant extension ensures routing between the VXLAN segments within the same tenant.

As shown in Figure 19, VNI segment 3901 is provisioned only on the POD1-Leaf1 vLAG pair, but it is part of the tenant on both leafs.Let's go over a list of verification steps required to ensure that communication between the hosts in VNI 2001 on POD2-Leaf1 and hostson VNI 3901 on POD1-Leaf1.

RMAC of Each Node

There is one RMAC assigned to every VTEP. This information can be obtained by looking at any L3 interface or the Layer 3 VNIsassociated the VLAN interface. For the vLAG pair, even though they have same VTEP IP, they are assigned a unique router MACaddress.



L3 VNI State on the Nodes

L3 VNI 7101 is assigned to the tenant VRF. Make sure that the vLAG pair and leaf5 have tunnels established to each other and that thisVNI is activated on it.

As seen in the following table for the output taken from POD1-Leaf1, the tunnel source is the VTEP IP of the vLAG (10.121.1.1), and thedestination IP is the vLAG VTEP IP of POD2-Leaf1 (10.122.2.1). (Notice additional tunnels in the list; these are destined to other VTEPswhere the same tenant is provisioned.)



The L3 VNI state from POD2-Leaf1 is shown below.



Verify the Route to the Remote Subnet of the Same Tenant

The following table shows the BGP entry on POD2-Leaf1 for the remote subnet of VNI 3901. (Note that the host entries are alsoadvertised over BGP, but they will be ignored by this leaf as this VNI is not locally provisioned and only routing is desired).



There are four entries in BGP table: two originators in the vLAG pair, and those two entries are learned from two spines exchangingEVPN routes. The next hop is the same due to the common VTEP IP used by the vLAG pair.

Example-3: Tenant Extension Outside the FabricIn "Example-2: Tenant and L2 Extension Between PoDs in an Optimized 5-Stage Clos Fabric," we illustrated extending a tenant VRFacross racks in two PoDs. In this section, let's see the steps involved in extending the same tenant outside the fabric through the borderor edge leafs.

Figure 20 shows a section of the validated design. Here, we're extending tenant vrf vrf101 outside the fabric through the edge leaf. Theedge leaf is connected to a WAN edge router, and the tenant VRF is extended to the WAN edge.



FIGURE 20 Tenant Extension Outside the Fabric Through Edge Leafs

ConfigurationWe will skip through the configurations of the POD1-Leaf1 and POD2-Leaf1 vLAG pairs since they have already been covered earlierand will focus on the configurations of the edge leafs.

Edge-Leaf1 ConfigurationOn the edge leaf, we do not recommend any server VLAN segments.

For the fabric side, we need only a VNI segment for the purpose of the L3 routing VNI for the tenant VRF. This VNI must be consistentwith other leafs for a given tenant. In this example, we're using VNI 7101 as the L3 VNI for the tenant vrf101.

For the external-facing side, we need another VLAN for peering with external routers.



Edge-Leaf2 ConfigurationOn the edge leaf, we do not recommend any server VLAN segments.

For the fabric side, we need only a VNI segment for the purpose of the L3 routing VNI for the tenant VRF. This VNI must be consistentwith other leafs for a given tenant. In this example, we're using VNI 7101 as the L3 VNI for tenant vrf101.

For the external facing side, we need another VLAN for peering with external routers.



Verification

RMAC of Each NodeThere is one RMAC assigned to every VTEP. This information can be obtained by looking at any L3 interface or the VLAN interfaceassociated with the Layer 3 VNI. For the vLAG pair, even though the nodes have the same VTEP IP, they are assigned a unique routerMAC.

POD1-Leaf1 Pair

POD2-Leaf1 Pair

Verify the L3 VNI State on the NodesHere we need to make sure that the Layer 3 VNI is associated with tunnels to every other node that has been provisioned with the sametenant.

For instance, the output from POD1-Leaf1-1 shows three tunnels. Looking at the destination IPs, we can confirm that POD2-Leaf1, Edge-Leaf1, and Edge-Leaf2 have been associated with the Layer 3 VNI of 7101 of tenant vrf101. (The source IP is the VTEP IP of the POD1-Leaf1 vLAG pair.)



The following shows the VNI state from Edge-Leaf1. It is associated with tunnels destined to POD1-Leaf1 (10.121.1.1) and POD2-Leaf1(10.122.2.1).

On Edge-Leaf2 also, let's ensure that the tunnels to POD1-Leaf1 (10.121.1.1) and POD2-Leaf1 (10.122.2.1) are associated with Layer 3 VNI7101.



Verify the Route to a Fabric Segment on the Edge LeafLet's look at the route entry to the subnet of VLAN/VNI 2001 (10.107.1.0/24). It is advertised by the vLAG pairs in two PoDs. Effectively,we should see two equal paths. Since the RMACs are different between vLAG peers within the vLAG pair, we see four paths, as shownbelow. Also, note that the route is advertised by the edge leaf to its external BGP peer.

(The "show ip bgp routes <prefix> vrf <vrf-name>" command lists the routes sent to the route-table manager after the best-pathcomputations are complete. If this output is not correct, check the "show bgp evpn routes type ipv4-prefix <> tag 0" command.)

Similarly, for the route to the VNI 3901 subnet learned from the POD1-Leaf1 vLAG pair whose VTEP IP is 10.121.1.1:



Verify the Route to an External Network on the Internal LeafsAs shown in Figure 20, external network 172.23.150.0/26 must be reachable from the tenant VRF of the internal leafs. Let us look at theroute verification, step by step, starting from the edge leaf.

First, verify the route on Edge-Leaf1. As shown, the route is installed in the correct VRF and is pointing to the external next hop of theWAN edge router.

The next step is to verify that this route gets advertised by the edge leafs into the fabric in EVPN Address-Family. The important fields tolook at in this output are L3 VNI, Router MAC, RD, RT, and Next Hop, as highlighted below.



Now let's look at the BGP entry on one of the internal leafs, say POD1-Leaf1. It should see two paths to the external network as bothedge leafs are advertising that network into the fabric. As you see in the output below, there are four entries—due to the fact that they'relearned from two spines. Essentially, there are two unique entries.

Verify that the routes are sent to the route table by BGP.



Example-4: VLAN Scoping at the ToR LevelVLAN scoping is briefly discussed in the “VLAN Scoping” section under the technology overview “Network Virtualization with BGPEVPN" chapter.

Refer to the Figure 21 for the topology used to illustrate the VLAN scoping at the leaf or ToR level. For the purpose of illustration, we’vechosen a vLAG pair and an individual leaf. Both ToRs may be vLAG pairs or individual leafs.

As seen in the figure, each leaf has a server VLAN that requires a Layer 2 extension to the other rack. Also note that the VLAN numbersare different. By mapping these VLANs to the same VNI number—8000 in this case—we achieve bridging or L2 extension betweenthem. The servers now have L2 adjacency between them. In other words, they are in the same bridge domain or broadcast domain. Inessence, the VLAN tag on the wire between the servers and the leaf is decoupled from the bridge domain. This VLAN tag need not beidentical on both sides to have Layer 2 adjacency or extension. In other words, the VLAN number is relevant only at the ToR level.

FIGURE 21 VLAN Scoping at the ToR Level

ConfigurationThe configuration steps are similar to the L2 extension illustrated in “Example-1: Tenant and L2 Extension between Racks in a 3-StageClos Fabric.” The difference is in the VLAN-to-VNI mapping under the overlay gateway configuration. A sample configuration is shownbelow for a quick reference; as highlighted, a server VLAN is manually mapped to a VNI number.



The table below summarizes the provisioning of L2 extension on the leafs.

Leaf 1 Leaf 5

∙ Server traffic is tagged with VLAN 100.

– Create VLAN 100.– Create the VE 100 Layer 3 interface for first-hop routing.– Assign the anycast GW 10.100.1.254 address to VE 100.

∙ Map VLAN 100 to VNI 8000 under the overlay gateway.

∙ Server traffic is tagged with VLAN 20.

– Create VLAN 20.– Create the VE 20 Layer 3 interface for first-hop routing.– Assign the anycast GW 10.100.1.254 address to VE 20.

∙ Map VLAN 20 to VNI 8000 under the overlay gateway.

Complete configurations and verification steps on the leafs in the Figure 21 topology are given in the sections that follow.

Configuration on the Leaf1 vLAG PairThe configuration is shown in three parts for clarity:

∙ Common configurations, such as port channel and VLANs, are shown in one block.

∙ The tenant, Layer 3 interfaces, and BGP EVPN configurations are shown in the second block under each RBridge ID.

∙ The common overlay-gateway configuration is shown in the third block.

Please note that the entire configuration is applied from the primary node in this two-node vLAG pair.



Verification

Verify VLAN Extension Between the RacksCheck the L2-extended VLAN on each node. This should show the local L2 trunk ports and also the tunnels to all remote VTEPs wherethe same VLAN segment is extended.

In the output below from the Leaf1 vLAG pair, there is one tunnel for VLAN 100, which indicates that the same VLAN/VNI segment isprovisioned on one other VTEP or ToR. Note that one of the tunnels, Tu 61445, is destined to Leaf5. Also note that there are fourunderlay next hops to reach this tunnel destination in the fabric.



In the output below from Leaf5, Tunnel 61442 is destined to the vLAG Leaf1 pair's VTEP IP 10.121.1.1.



VLAN Layer 3 Interfaces State on the vLAG Pair

VLAN Layer 3 Interfaces State on the Leaf5 ToR





Remote Host Entries in the Extended VLAN BGP and ARP Table on Leaf5

The table below from Leaf5 shows the BGP and ARP entries of a remote host behind the Leaf1 pair. Note that the next hop is set to10.121.1.1, which is a common VTEP IP of the vLAG pair. There are two entries in BGP since there are two spines exchanging the EVPNroutes.

In the hardware ARP table, both the local and remote entries are indicated with different types. The local host entries are of typeDynamic, and the remote host entries are of type BGP-EVPN. Note that the remote host entries are shown under the virtual interface oflocal VLAN 20 on Leaf5 (not VLAN 100 as in the remote ToR).



Example-5: VLAN Scoping at the Port Level Within a ToRVLAN scoping is briefly discussed in the “VLAN Scoping” section under the “Network Virtualization with BGP EVPN” chapter.

Port VLAN scoping enables complete abstraction of a bridge domain where the VLAN tags on the server-side data frame on two portscan be different and still be bridged between the ports. The VLAN tag is localized at the port level rather than at the ToR level.

Refer to the topology shown in Figure 22.

On the vLAG leaf, there are two port channels or LAG bundles: po111 and po112. Each has server traffic tagged with an 802.1q VLAN tagof 10 and 30, respectively. From the port VLAN scoping perspective, these tags are referred to as c-tags. The {port,vlan} is added as amember of a virtual-fabric VLAN. In this case, there is a fabric VLAN ID 6000. (Note that this number is above the 802.1q VLAN rangeof 4096.)

In summary, VLAN 6000 comprises two members (port, vlan). (Unlike the ports in traditional VLAN cases.)

∙ (po111, vlan tag 10)

∙ (po112, vlan tag 30)

On Leaf5, VLAN 40 is mapped to VNI 8001. On the Leaf1 pair, VLAN 6000 is mapped to VNI 8001. Thus we're providing Layer 2extension within and between the leafs for server-side traffic with different dot1q VLAN tags.



FIGURE 22 VLAN Scoping at the Port Level Within a ToR

ConfigurationThe configuration steps are similar to the L2 extension illustrated in “Example-4: VLAN Scoping at the ToR Level.” The difference is inthe virtual-fabric port-VLAN scoping on the vLAG pair.

A sample configuration is given below as a quick reference for port-VLAN scoping. In this example, {po111, c-tag 10} and {Te 1/0/3, c-tag20} are mapped to VLAN 6000.5 With this configuration, it is possible to bridge traffic on these ports with the specified dot1q tags.

Configuration on the Leaf1 vLAG PairThe configuration is shown in three parts for clarity:

∙ Common configurations, such as port channel and VLANs, are shown in one block.

∙ The tenant, Layer 3 interfaces, and BGP EVPN configurations are shown in the second block under each RBridge ID.

∙ The common overlay-gateway configuration is shown in the third block.

Please note that the entire configuration is applied from the primary node in this two-node vLAG pair.

5 Multiple c-tags on the same L2 port cannot be mapped to a VLAN.



Verification

Verify VLAN Extension Between the RacksCheck the L2 extended VLAN on each node. This should show the local L2 trunk ports and also the tunnels to all remote VTEPs wherethe same VLAN segment is extended.

In the output below from the Leaf1 vLAG pair, there is one tunnel for VLAN 6000, which indicates that the same VLAN/VNI segment isprovisioned on one other VTEP or ToR. Note that one of the tunnels, Tu 61445, is destined to Leaf5. Also note that there are fourunderlay next hops to reach this tunnel destination in the fabric.



In the output below from Leaf5, Tunnel 61442 is destined to the vLAG Leaf1 pair's VTEP IP 10.121.1.1





Remote Host Entries in the Extended VLAN BGP and ARP Table on Leaf5



The table below taken on Leaf5 shows the BGP and ARP entries of the remote hosts behind the Leaf1 pair. Note that the next hop is setto 10.121.1.1, which is a common VTEP IP of the vLAG pair. There are two entries in BGP since there are two spines exchanging theEVPN routes.

In the ARP table, both the local and remote entries are indicated with different types: BGP-EVPN for remote entries, signifying that theywere learned over BGP-EVPN; Dynamic for local entries. Note that the remote host entries are imported into the virtual interface of localVLAN 40 on Leaf5.

Example-6: Route Leaking for the Service VRFWith network virtualization for multitenant environments, typically the tenant VRFs are extended to the border leaf and they areconnected to the service VRF through a firewall/NAT/LB appliance to a service VRF. This poses a challenge of VRF and interfacescalability on the border leaf. In these cases, we recommend provisioning multiple border leafs and distributing the tenants across them.



FIGURE 23 Services Provisioning on the Border Leaf

A service VRF with route leaking addresses the scalability requirements on the border leaf for certain controlled deployments. The routesto the services are leaked to the tenants in the fabric and vice-versa without the need to extend these tenant VRFs to the border leaf. Asshown in Figure 24, the edge leaf does not have the tenant VRFs provisioned on it. The routes from the tenants are imported into theservice VRF, and the service VRF typically advertises a default route toward the tenants in the fabric. There are other possible variationswith this approach. One may connect the storage directly to the service VRF itself. It is also possible to connect to the Internet directlyfrom the service VRF if the tenants have globally scoped addresses or if address translation occurs elsewhere.

FIGURE 24 Service VRF with Route Leaking on the Border Leaf

Since the routes between the tenants and the service VRF are leaked between each other, consider the following points:

∙ Unique IP addressing is needed for the tenants.

∙ Provisioning a per-tenant stateful firewall would be a challenge. One device must be able to handle all the transactions. Socarefully consider the scale requirements of the firewall.

∙ Intertenant traffic is possible through the service VRF because all routes are imported there. To prevent this, we recommendhaving the necessary safeguards inside the tenants.



FIGURE 25 Topology of the Service VRF with Route Leaking from Tenants

Figure 25 shows a part of the validated topology to illustrate route leaking between tenant VRFs and the service VRF. As shown, thereare two tenant VRFs in the fabric: VRF202 and VRF203. Also note that VRF202 is also extended to Leaf5 (in other words, the tenant isprovisioned on two racks). These tenants are expected to have access to a common service attached to the border leaf. The border leafshave been configured with a service VRF. Each VRF has its own L3 VNI for symmetric routing.

The routes from tenants are leaked into the service VRF, and routes from Service are leaked into all tenant VRFs using export/importroute targets, as shown in the table below.

Leaf1 VLAG Pair Leaf5 Edge-Leaf1 Edge-Leaf2

Tenant vrf202 vrf202, L3VNI 7202

Export RT 202:202

Import RT 202:202

Import RT 8190:8190

vrf202, L3VNI 7202

Export RT 202:202

Import RT 202:202

Service, L3VNI 8190

Import RT 202:202

Export RT 8190:8190

Import RT 203:203

Service, L3VNI 8190

Import RT 202:202

Export RT 8190:8190

Import RT 203:203

Tenant vrf203 vrf203, L3VNI 7203

Export RT 203:203

Import RT 203:203

Import RT 8190:8190

Not provisioned

As explained in the earlier sections on routing and in tenant extension illustrations, when the routes are exported or advertised from theVRF, the L3VNI associated with the VRF is also included with the route. This creates an asymmetry in the L3VNI numbers in this case.For example, see the table below:

Leaf1 Pair - VRF vrf202 Edge-Leaf1 - VRF Service Edge-Leaf2 - VRF Service

Advertise EVPN type-5 prefix route10.111.9.0/24 and type-2 host routes10.111.9.20/32 and 10.111.9.21/32.

∙ Export RT 202:202

The received route 10.111.9.0/24 matches importRT 202:202.

But the L3VNI is 7202 and not 8190 (of VRFservice).

The received route 10.111.9.0/24 matches importRT 202:202.

But the L3VNI is 7202 and not 8190 (of VRFservice).




∙ Next hop 10.121.1.1

∙ L3VNI 7202

∙ Create a VE interface and associatewith VNI 7202.


The received route matches import RT8190:8190.

But the L3VNI is 8190 and not 7202 (ofvrf202).


Advertise EVPN prefix route 0/0 and172.161.108.0/24.

∙ RT 8190:8190

∙ Next hop 10.123.3.1

∙ L3VNI 8190


∙ RT 8190:8190

∙ Next hop 10.123.3.1

∙ L3VNI 8190

Similarly for the tenant VRF vrf203:


Advertise EVPN type-5 prefix route10.111.17.0/24 and type-2 host routes10.111.17.20/32 and 10.111.17.21/32.

∙ RT 203:203

∙ Next hop 10.121.1.1

∙ L3VNI 7203

Received route 10.111.17.0/24 matches import RT203:203.

But L3VNI is 7203 and not 8190 (of VRFservice).


Received route 10.111.17.0/24 matches import RT203:203.

But L3VNI is 7203 and not 8190 (of VRFservice).


The received route matches the import RT8190:8190.

But the L3VNI is 8190 and not 7203 (ofvrf203).



∙ Export RT 8190:8190

∙ Next hop 10.123.3.1

∙ L3VNI 8190

Advertise EVPN prefix routes 0/0 and172.16.108.0/24.

∙ Export RT 8190:8190

∙ Next hop 10.123.3.1

∙ L3VNI 8190

In summary:

∙ On the leafs, we must create one additional VE interface in the default VRF and associate it with a VNI number equal to theL3VNI of the service VRF.

∙ On the border leaf, for every tenant that is leaked into the service VRF, create a VE interface in the default VRF and associate itwith the VNI number equal to the L3VNI of the tenant.

These additional VNIs must be activated in the EVPN instance by the leafs and border leafs.

Leaf1 Pair Leaf5 Border Leafs

VNI 8190, VLAN/VE 8190 in the default VRF VNI 8190, VLAN/VE 8190 in the default VRF VNI 7202, VLAN/VE 7202 in the default VRF

VNI 7203, VLAN/VE 7203 in the default VRF

ConfigurationThe following sections provide the incremental configuration relevant to the route leaking between the services and the tenant VRFs. Adefault route and a subnet route are injected from the service VRF of the edge leaf into the fabric, and the tenants import it. The tenants'VLAN subnets and host routes are similarly imported by the service VRF.

Configuration on the Leaf1 vLAG PairThe Leaf1 vLAG pair has both vrf202 and vrf203 tenant VRFs.



Configuration on Leaf5Leaf5 has been provisioned with just the vrf202 tenant VRF.



Configuration on the Edge LeafThe edge leaf is provisioned with only the service VRF. In this illustration, the edge leaf advertises two routes: a default route (say to aservice appliance) and a subnet route (say of a VLAN connecting storage network).



Verification

Route Learning from the Service VRF into TenantsIn the topology used in this illustration, the Service VRF is advertising a default route and a subnet route toward the tenants in the fabricas an EVPN type-5 prefix route. The tenants (VRFs) on the leafs import these routes.

Route Origination from the Service VRF of the Edge Leaf :

Service VRF Routing Table

Service VRF BGP Entries

Advertising the Routes into EVPN



Routes Received by the Leaf1 vLAG Pair from the Service VRF:

EVPN Routes Received from Edge Leafs

There are two entries for the default route from each edge leaf, as there are two EVPN spines in the fabric. Also note that the Leaf1 vLAGpair has both vrf202 and vrf203 tenants. The routes received from edge leafs are imported into both VRFs. The following output istaken from one of the nodes in the vLAG pair. Verification steps are the same for the second node also.



VE Interface States



Tenant VRF vrf202

Tenant VRF vrf203



Routes Received by Leaf5 from the Service VRF

Leaf5 receives the routes advertised by the two edge leafs from two EVPN spine neighbors. The CLI output shows the BGP entry forthe default route.



Leaf5 Tenant VRF vrf202

Leaf5 imports the routes received from the service into tenant VRF vrf202.



Route Learning into the Service VRF from TenantsThe service VRF on the edge leaf learns hosts and subnet routes from the tenants in EVPN type-2 and type-5 routes respectively.

Leaf1 advertises subnet and hosts routes from tenants vrf202 and vrf203.

Leaf5 advertises subnet and hosts routes from tenant vrf202.

Tenant vrf202 has the same subnet extended (L2 extension) between Leaf1 and Leaf5. So verification should include the host entries aswell to ensure that they point to the correct VTEP IP of the ToR to which they're connected.

Leaf1 Leaf5 Edge-Leaf

Tenant vrf202

Subnet:

10.111.9.0/24

Hosts:

10.111.9.20

10.111.9.21

Tenant vrf202

Subnet:

10.111.9.0/24

Hosts:

10.111.9.50

10.111.9.51

VRF service

Subnets as trap routes:

10.111.9.0/24

10.111.17.0/24

Hosts routes behind VTEP next hops:

10.111.9.20 --> Leaf1 VTEP IP 10.121.1.1, VE 7202. VNI 7202






Tenant vrf203

Subnet:

10.111.17.0/24

Hosts:

10.111.17.20

10.111.17.21

Tenant vrf202 notprovisioned

Edge-Leaf1

Note that the subnet routes in the route table point to the VTEP next hops, but in hardware they're programmed as trap entries tofacilitate conversational host route download into the hardware.

The EVPN entry for one of the subnets, 10.111.9.0/24, is shown below. This route is advertised by both the Leaf1 vLAG pair (two nodes)and Leaf5 (individual ToR). In the vLAG pair, both the nodes advertise the routes into BGP EVPN. So we see three BGP entries receivedfrom two EVPN spines; hence a total of six entries.



VE Interface States



Routes Received from Tenant vrf202

Routes Received from Tenant vrf203



Design ConsiderationsScale

The following table gives various scale parameters and platforms used in this validated test topology. Note that this is not a measure ofthe maximum scale that can be supported with Brocade switches in IP fabric.

Parameter PoDY1 PoD2 Border Leaf

Platform used as leaf VDX 6940-144S VDX 6740 VDX 6940-36Q

Platform used as spine VDX 6940-36Q VDX 6940-36Q N/A

Number of server racks/leafs 8 8 N/A

Number of spines 4 4 N/A

Number of tenant VRFs per rack 106 20 70

Number of tenants local to the leaf (not extended toother racks)

4 4 N/A

Number of tenants extended within the PoD to all racks 100 16 N/A

Number of server VLAN segments per rack 507 505 N/A

Number of VLANs used for L3 VNI of tenant VRFs perrack

106 20 70

Number of L2 VNIs per rack 507 505 N/A

Number of L2 VNIs (server VLAN segments) extendedwithin the PoD to all leafs/racks

400 400 N/A

ARP-suppressed VLANs per leaf/rack 64 64 N/A

ND-suppressed VLANs per leaf/rack 12 12 N/A

Platform used as super-spine VDX 8770-4

Number of super-spines 4

Number of tenants extended between the PoDs 16

ARP/ND Suppression Guidelines

∙ This feature is enabled on a per-VLAN basis.

∙ Enabling this feature involves the hardware ACL table, and this resource is shared with other ACL features as well.

∙ ARP/ND suppression is needed only on server-facing VLANs.

∙ Enable ARP/ND suppression on both nodes of vLAG pairs.

∙ On individual non-redundant leafs, suppression is required only if the VLAN is L2-extended to other leafs.

∙ Use the DAI TCAM profile. With this profile, the validated scale is 64 and 12 VLANs for IPv4 and IPv6 respectively per leaf/rack.


∙ In the case of a vLAG pair, the profile configuration must be set for each RBridge in the pair.

Recommendations for ISL Ports in a vLAG Pair Leaf

∙ We recommend picking ISL ports from the same port group on the switch. Port-group information about the leaf platforms isgiven in the Brocade VDX hardware installation guides.

∙ For redundancy, we recommend having a minimum of two ISL ports between the switches in the vLAG pair

∙ The bandwidth requirement for ISL links depends on the number of fabric links and the traffic pattern. The ISL links areprimarily used for routed traffic received over the L3 VNI depending on the router MAC used in the data packet. A good rule ofthumb is to provision links with half the bandwidth of the fabric links. For example, if there are four 40G fabric links on eachswitch, provision two 40G links as ISL between the switches.

Fabric Link Tracking on a vLAG Pair

With BGP/EVPN network virtualization, two spines are designated to exchange EVPN AFI routes. Loss of both links connecting theseEVPN spines would result in a traffic black-hole for the tenants. In a vLAG ToR, we can prevent this by tracking the links to EVPN spinesand isolating the node from the fabric if it loses those links by shutting down the remaining fabric links and server port-channel memberports.

∙ On each node of the vLAG pair, identify the links connected to the spines that exchange EVPN routes.

∙ Track these links under other fabric links and the server-facing port-channel member ports.

The steps are shown in the following captures from one of the nodes in a vLAG leaf. Repeat the steps on the other node as well.

Design Considerations


Track these two links under the remaining fabric ports.

Track under the server-facing port-channel member ports.



L2 Loop Detection and Prevention

Brocade leaf platforms provide two options for L2 loop detection and prevention.

∙ Detect MAC move and shut the L2 port.

∙ BGP EVPN dampening mechanism for L2 routes or MAC routes.

We recommend the following configuration to make the L2 port-shut take precedence. With this configuration, the L2 port will be shutdown if a MAC moves 5 times within an interval of 10s.

BGP TTL Security

This is applicable for eBGP peering only. This configuration can be applied to a specific neighbor or a peer group.



Appendix—Configuration of the NodesThis appendix includes the relevant configurations of a few nodes in the fabric.

vLAG Active/Active Pair Leaf! 2-node vLAG pair! Node 1, Rbridge-id 45! Node 2, Rbridge-id 46vcs virtual-fabric enableinterface Vlan 701 description VLAN 701, VNI 701, Tenant vrf71;!interface Vlan 2001 description VLAN 2001, VNI 2001, Tenant vrf101; extended to POD2!interface Vlan 3001 description VLAN 3001, VNI 3001, Tenant vrf101; extended within POD1!interface Vlan 3802 description VLAN 3802, VNI 3802, Tenant vrf201;!interface Vlan 7071 description VLAN 7071, VNI 7071, Tenant vrf71; Layer 3 VNI!interface Vlan 7101 description VLAN 7101, VNI 7101, Tenant vrf101; Layer 3 VNI!interface Vlan 7201 description VLAN 7201, VNI 7201, Tenant vrf201; Layer 3 VNI!! Node 1 in the vLAG pair! L3, tenant VRFs, BGP, and EVPN-instance configurationrbridge-id 45 ip anycast-gateway-mac 0201.0101.0101 ip router-id 10.121.1.11 vrf vrf101 rd 10.121.1.11:101 vni 7101 address-family ipv4 unicast route-target export 101:101 evpn route-target import 101:101 evpn ! address-family ipv6 unicast route-target export 101:101 evpn route-target import 101:101 evpn ! ! vrf vrf201 rd 10.121.1.11:201 vni 7201 address-family ipv4 unicast route-target export 201:201 evpn route-target import 201:201 evpn ! address-family ipv6 unicast route-target export 201:201 evpn route-target import 201:201 evpn ! ! vrf vrf71 rd 10.121.1.11:71 vni 7071 address-family ipv4 unicast route-target export 71:71 evpn


route-target import 71:71 evpn ! address-family ipv6 unicast route-target export 71:71 evpn route-target import 71:71 evpn ! ! host-table aging-mode conversational evpn-instance pod1-leaf1 route-target both auto ignore-as rd auto duplicate-mac-timer 5 max-count 3 vni add 4-6,701,2001,3001,3802 ! router bgp local-as 4200000001 capability as4-enable neighbor spine-evpn-group peer-group neighbor spine-evpn-group remote-as 4200000000 neighbor spine-evpn-group password 2 $PVNHITJVPWQ= neighbor spine-evpn-group bfd neighbor spine-ip-group peer-group neighbor spine-ip-group remote-as 4200000000 neighbor spine-ip-group password 2 $PVNHITJVPWQ= neighbor spine-ip-group bfd neighbor 10.11.1.0 peer-group spine-ip-group neighbor 10.12.1.0 peer-group spine-evpn-group neighbor 10.13.1.0 peer-group spine-evpn-group neighbor 10.14.1.0 peer-group spine-ip-group address-family ipv4 unicast network 10.121.1.1/32 maximum-paths 8 graceful-restart ! address-family ipv4 unicast vrf vrf101 redistribute connected maximum-paths 8 ! address-family ipv4 unicast vrf vrf201 redistribute connected maximum-paths 8 ! address-family ipv4 unicast vrf vrf71 redistribute connected maximum-paths 8 ! address-family ipv6 unicast vrf vrf101 redistribute connected maximum-paths 8 ! address-family ipv6 unicast vrf vrf201 redistribute connected maximum-paths 8 ! address-family ipv6 unicast vrf vrf71 redistribute connected maximum-paths 8 ! address-family l2vpn evpn graceful-restart neighbor spine-evpn-group activate neighbor spine-evpn-group allowas-in 1 neighbor spine-evpn-group next-hop-unchanged ! ! ipv6 anycast-gateway-mac 0201.0102.0202 interface Loopback 1 no shutdown ip address 10.121.1.1/32 ! interface Loopback 2 no shutdown

Appendix—Configuration of the Nodes


ip address 10.121.1.11/32 ! interface Ve 701 vrf forwarding vrf71 ipv6 anycast-address fd2d:d47f:115:2bd::254/64 ipv6 nd cache expire 270 ip anycast-address 10.115.1.254/24 ip arp-aging-timeout 4 no shutdown ! interface Ve 2001 vrf forwarding vrf101 ipv6 anycast-address fd2d:d47f:107:1::254/64 ipv6 nd cache expire 270 ip anycast-address 10.107.1.254/24 ip arp-aging-timeout 4 no shutdown ! interface Ve 3001 vrf forwarding vrf201 ipv6 anycast-address fd2d:d47f:111:bb9::254/64 ipv6 nd cache expire 270 ip anycast-address 10.111.1.254/24 ip arp-aging-timeout 4 no shutdown ! interface Ve 7071 vrf forwarding vrf71 ipv6 address use-link-local-only no shutdown ! interface Ve 7101 vrf forwarding vrf101 ipv6 address use-link-local-only no shutdown ! interface Ve 7201 vrf forwarding vrf201 ipv6 address use-link-local-only no shutdown !! ! Node 2 in the vLAG pair! L3, tenant VRFs, BGP, and EVPN-instance configurationrbridge-id 46 ip anycast-gateway-mac 0201.0101.0101 ip router-id 10.121.1.12 vrf vrf101 rd 10.121.1.12:101 vni 7101 address-family ipv4 unicast route-target export 101:101 evpn route-target import 101:101 evpn ! address-family ipv6 unicast route-target export 101:101 evpn route-target import 101:101 evpn ! ! vrf vrf201 rd 10.121.1.12:201 vni 7201 address-family ipv4 unicast route-target export 201:201 evpn route-target import 201:201 evpn ! address-family ipv6 unicast route-target export 201:201 evpn route-target import 201:201 evpn ! ! vrf vrf71



rd 8052:71 vni 7071 address-family ipv4 unicast route-target export 71:71 evpn route-target import 71:71 evpn ! address-family ipv6 unicast route-target export 71:71 evpn route-target import 71:71 evpn ! ! host-table aging-mode conversational evpn-instance pod1-leaf1 route-target both auto ignore-as rd auto duplicate-mac-timer 5 max-count 3 vni add 4-6,701,2001,3001,3802 ! router bgp local-as 4200000001 capability as4-enable neighbor spine-evpn-group peer-group neighbor spine-evpn-group remote-as 4200000000 neighbor spine-evpn-group password 2 $PVNHITJVPWQ= neighbor spine-evpn-group bfd neighbor spine-ip-group peer-group neighbor spine-ip-group remote-as 4200000000 neighbor spine-ip-group password 2 $PVNHITJVPWQ= neighbor spine-ip-group bfd neighbor 10.11.2.0 peer-group spine-ip-group neighbor 10.12.2.0 peer-group spine-evpn-group neighbor 10.13.2.0 peer-group spine-evpn-group neighbor 10.14.2.0 peer-group spine-ip-group address-family ipv4 unicast network 10.121.1.1/32 maximum-paths 8 graceful-restart ! address-family ipv4 unicast vrf vrf101 redistribute connected ! address-family ipv4 unicast vrf vrf201 redistribute connected maximum-paths 8 ! address-family ipv4 unicast vrf vrf71 redistribute connected maximum-paths 8 ! address-family ipv6 unicast vrf vrf101 redistribute connected ! address-family ipv6 unicast vrf vrf201 redistribute connected maximum-paths 8 ! address-family ipv6 unicast vrf vrf71 redistribute connected maximum-paths 8 ! address-family l2vpn evpn graceful-restart neighbor spine-evpn-group activate neighbor spine-evpn-group allowas-in 1 neighbor spine-evpn-group next-hop-unchanged ! !ipv6 anycast-gateway-mac 0201.0102.0202 interface Loopback 1 no shutdown ip address 10.121.1.1/32 !



interface Loopback 2 no shutdown ip address 10.121.1.12/32 ! interface Ve 701 vrf forwarding vrf71 ipv6 anycast-address fd2d:d47f:115:2bd::254/64 ipv6 nd cache expire 270 ip anycast-address 10.115.1.254/24 ip arp-aging-timeout 4 no shutdown ! interface Ve 2001 vrf forwarding vrf101 ipv6 anycast-address fd2d:d47f:107:1::254/64 ipv6 nd cache expire 270 ip anycast-address 10.107.1.254/24 ip arp-aging-timeout 4 no shutdown ! interface Ve 3001 vrf forwarding vrf201 ipv6 anycast-address fd2d:d47f:111:bb9::254/64 ipv6 nd cache expire 270 ip anycast-address 10.111.1.254/24 ip arp-aging-timeout 4 no shutdown ! interface Ve 7071 vrf forwarding vrf71 ipv6 address use-link-local-only no shutdown ! interface Ve 7101 vrf forwarding vrf101 ipv6 address use-link-local-only no shutdown ! interface Ve 7201 vrf forwarding vrf201 ipv6 address use-link-local-only no shutdown !!! Fabric infrastructure L3 links, server-facing links, and vLAGsinterface TenGigabitEthernet 45/0/5 channel-group 111 mode active type standard fabric isl enable fabric trunk enable lacp timeout long no shutdown!interface TenGigabitEthernet 45/0/6 channel-group 112 mode active type standard fabric isl enable fabric trunk enable lacp timeout long no shutdown!interface TenGigabitEthernet 45/0/7 channel-group 113 mode active type standard fabric isl enable fabric trunk enable lacp timeout long no shutdown!interface TenGigabitEthernet 46/0/5 channel-group 111 mode active type standard fabric isl enable fabric trunk enable lacp timeout long no shutdown



!interface TenGigabitEthernet 46/0/6 channel-group 112 mode active type standard fabric isl enable fabric trunk enable lacp timeout long no shutdown!interface TenGigabitEthernet 46/0/7 channel-group 113 mode active type standard fabric isl enable fabric trunk enable lacp timeout long no shutdown!interface FortyGigabitEthernet 45/0/97 mtu 9216 description Link to spine1 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.11.1.1/31 no shutdown!interface FortyGigabitEthernet 45/0/98 mtu 9216 description Link to spine2 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.12.1.1/31 no shutdown!interface FortyGigabitEthernet 45/0/103 mtu 9216 description Link to spine3 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.13.1.1/31 no shutdown!interface FortyGigabitEthernet 45/0/104 mtu 9216 description Link to spine4 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.14.1.1/31 no shutdown!interface FortyGigabitEthernet 46/0/97 mtu 9216 description Link to spine1 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.11.2.1/31 no shutdown!interface FortyGigabitEthernet 46/0/98 mtu 9216 description Link to spine2 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp



ip address 10.12.2.1/31 no shutdown!interface FortyGigabitEthernet 46/0/103 mtu 9216 description Link to spine3 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.13.2.1/31 no shutdown!interface FortyGigabitEthernet 46/0/104 mtu 9216 description Link to spine4 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.14.2.1/31 no shutdown!interface Port-channel 111 vlag ignore-split switchport switchport mode trunk-no-default-native switchport trunk allowed vlan add 701,3001 spanning-tree shutdown no shutdown!interface Port-channel 112 vlag ignore-split switchport switchport mode trunk-no-default-native switchport trunk allowed vlan add 3802 spanning-tree shutdown no shutdown!interface Port-channel 113 vlag ignore-split switchport switchport mode trunk-no-default-native switchport trunk allowed vlan add 2001 spanning-tree shutdown no shutdown!mac-address-table learning-mode conversationaloverlay-gateway leaf1 type layer2-extension ip interface Loopback 1 attach rbridge-id add 45-46 map vlan vni auto activate!

Individual Non-Redundant Leaf!Rbridge-id 51vcs virtual-fabric enableinterface Vlan 1101 description VLAN 1101, VNI 1101, Tenant VRF vrf111;!interface Vlan 2401 description VLAN 2401, VNI 2401, Tenant VRF vrf109;!interface Vlan 3001 description VLAN 3001, VNI 3001, Tenant VRF vrf201;!



interface Vlan 7109 description VLAN 7109, VNI 7109, Tenant vrf109; Layer 3 VNI!interface Vlan 7111 description VLAN 7111, VNI 7111, Tenant vrf111; Layer 3 VNI!interface Vlan 7201 description VLAN 7201, VNI 7201, Tenant vrf201; Layer 3 VNI!rbridge-id 51 ip anycast-gateway-mac 0201.0101.0101 ip router-id 10.121.1.51 vrf vrf109 rd 10.121.1.51:109 vni 7109 address-family ipv4 unicast route-target export 109:109 evpn route-target import 109:109 evpn ! address-family ipv6 unicast route-target export 109:109 evpn route-target import 109:109 evpn ! ! vrf vrf111 rd 10.121.1.51:111 vni 7111 address-family ipv4 unicast route-target export 111:111 evpn route-target import 111:111 evpn ! address-family ipv6 unicast route-target export 111:111 evpn route-target import 111:111 evpn ! ! vrf vrf201 rd 10.121.1.51:201 vni 7201 address-family ipv4 unicast route-target export 201:201 evpn route-target import 201:201 evpn ! address-family ipv6 unicast route-target export 201:201 evpn route-target import 201:201 evpn ! ! host-table aging-mode conversational evpn-instance pod1-leaf5 route-target both auto ignore-as rd auto duplicate-mac-timer 5 max-count 3 vni add 1101,2401,3001 ! router bgp local-as 4200000005 capability as4-enable neighbor spine-evpn-group peer-group neighbor spine-evpn-group remote-as 4200000000 neighbor spine-evpn-group password 2 $PVNHITJVPWQ= neighbor spine-evpn-group bfd neighbor spine-ip-group peer-group neighbor spine-ip-group remote-as 4200000000 neighbor spine-ip-group password 2 $PVNHITJVPWQ= neighbor spine-ip-group bfd neighbor 10.11.7.0 peer-group spine-ip-group neighbor 10.12.7.0 peer-group spine-evpn-group neighbor 10.13.7.0 peer-group spine-evpn-group neighbor 10.14.7.0 peer-group spine-ip-group address-family ipv4 unicast network 10.121.1.5/32



maximum-paths 8 graceful-restart ! address-family ipv4 unicast vrf vrf109 maximum-paths 8 redistribute connected ! address-family ipv4 unicast vrf vrf111 redistribute connected maximum-paths 8 ! address-family ipv4 unicast vrf vrf201 redistribute connected maximum-paths 8 ! address-family ipv6 unicast vrf vrf109 redistribute connected maximum-paths 8 ! address-family ipv6 unicast vrf vrf111 redistribute connected maximum-paths 8 ! address-family ipv6 unicast vrf vrf201 redistribute connected maximum-paths 8 ! address-family l2vpn evpn graceful-restart neighbor spine-evpn-group activate neighbor spine-evpn-group next-hop-unchanged ! ! ipv6 anycast-gateway-mac 0201.0102.0202 interface Loopback 1 no shutdown ip address 10.121.1.5/32 ! interface Loopback 2 no shutdown ip address 10.121.1.51/32 ! interface Ve 1101 vrf forwarding vrf111 ipv6 anycast-address fd2d:d47f:119:44d::254/64 ipv6 nd cache expire 270 ip anycast-address 10.119.1.254/24 ip arp-aging-timeout 4 no shutdown ! interface Ve 2401 vrf forwarding vrf109 ipv6 anycast-address fd2d:d47f:108:81::254/64 ipv6 nd cache expire 270 ip anycast-address 10.108.147.254/24 ip arp-aging-timeout 4 no shutdown ! interface Ve 3001 vrf forwarding vrf201 ipv6 anycast-address fd2d:d47f:111:bb9::254/64 ipv6 nd cache expire 270 ip anycast-address 10.111.1.254/24 ip arp-aging-timeout 4 no shutdown ! interface Ve 7109 vrf forwarding vrf109 ipv6 address use-link-local-only no shutdown ! interface Ve 7111



vrf forwarding vrf111 ipv6 address use-link-local-only no shutdown ! interface Ve 7201 vrf forwarding vrf201 ipv6 address use-link-local-only no shutdown !interface TenGigabitEthernet 51/0/4 switchport switchport mode trunk switchport trunk allowed vlan add 1101,2401,3001 switchport trunk tag native-vlan spanning-tree shutdown no fabric isl enable no fabric trunk enable no shutdown!interface FortyGigabitEthernet 51/0/97 mtu 9216 description Link to spine1 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.11.7.1/31 no shutdown!interface FortyGigabitEthernet 51/0/98 mtu 9216 description Link to spine2 fabric isl enable fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.12.7.1/31 no shutdown!interface FortyGigabitEthernet 51/0/103 mtu 9216 description Link to spine3 fabric isl enable fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.13.7.1/31 no shutdown!interface FortyGigabitEthernet 51/0/104 mtu 9216 description Link to spine4 fabric isl enable fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.14.7.1/31 no shutdown!mac-address-table learning-mode conversationaloverlay-gateway leaf5 type layer2-extension ip interface Loopback 1 attach rbridge-id add 51 map vlan vni auto activate!



Spine Designated to Exchange Only Underlay Routesrbridge-id 41 ip router-id 10.124.11.1 router bgp local-as 4200000000 capability as4-enable fast-external-fallover neighbor leaf-group peer-group neighbor leaf-group password 2 $PVNHITJVPWQ= neighbor leaf-group bfd neighbor 10.11.1.1 remote-as 4200000001 neighbor 10.11.1.1 peer-group leaf-group neighbor 10.11.2.1 remote-as 4200000001 neighbor 10.11.2.1 peer-group leaf-group neighbor 10.11.3.1 remote-as 4200000002 neighbor 10.11.3.1 peer-group leaf-group neighbor 10.11.4.1 remote-as 4200000002 neighbor 10.11.4.1 peer-group leaf-group neighbor 10.11.5.1 remote-as 4200000003 neighbor 10.11.5.1 peer-group leaf-group neighbor 10.11.6.1 remote-as 4200000004 neighbor 10.11.6.1 peer-group leaf-group neighbor 10.11.7.1 remote-as 4200000005 neighbor 10.11.7.1 peer-group leaf-group neighbor 10.11.8.1 remote-as 4200000006 neighbor 10.11.8.1 peer-group leaf-group neighbor 10.41.1.0 peer-group superspine-group neighbor 10.42.1.0 peer-group superspine-group neighbor 10.43.1.0 peer-group superspine-group neighbor 10.44.1.0 peer-group superspine-group address-family ipv4 unicast maximum-paths 8 graceful-restart ! interface Loopback 2 no shutdown ip address 10.124.11.1/32 !!interface FortyGigabitEthernet 41/0/1 mtu 9216 description Link to leaf1-1 vLAG pair no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.11.1.0/31 no shutdown!interface FortyGigabitEthernet 41/0/3 mtu 9216 description Link to leaf1-2 vLAG pair no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.11.2.0/31 no shutdown!interface FortyGigabitEthernet 41/0/4 mtu 9216 description Link to superspine-4 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.44.1.1/31 no shutdown!interface FortyGigabitEthernet 41/0/5



mtu 9216 description Link to superspine-3 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.43.1.1/31 no shutdown!interface FortyGigabitEthernet 41/0/6 mtu 9216 description Link to superspine-2 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.42.1.1/31 no shutdown!interface FortyGigabitEthernet 41/0/7 mtu 9216 description Link to superspine-1 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.41.1.1/31 no shutdown!interface FortyGigabitEthernet 41/0/10 mtu 9216 description Link to leaf2-1 vLAG pair no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.11.3.0/31 no shutdown!interface FortyGigabitEthernet 41/0/12 mtu 9216 description Link to leaf2-1 vLAG pair no fabric isl enable no fabric trunk enable ip mtu 9018 ip address 10.11.4.0/31 no shutdown!interface FortyGigabitEthernet 41/0/20 mtu 9216 description Link to leaf3 no fabric isl enable no fabric trunk enable ip mtu 9018 ip address 10.11.5.0/3 no shutdown!interface FortyGigabitEthernet 41/0/22 mtu 9216 description Link to leaf4 no fabric isl enable no fabric trunk enable ip mtu 9018 ip address 10.11.6.0/31 no shutdown!interface FortyGigabitEthernet 41/0/28 mtu 9216 description Link to leaf5 no fabric isl enable no fabric trunk enable ip mtu 9018



ip address 10.11.7.0/31 no shutdown!interface FortyGigabitEthernet 41/0/30 mtu 9216 description Link to leaf6 no fabric isl enable no fabric trunk enable ip mtu 9018 ip address 10.11.8.0/31 no shutdown!

Spine Designated to Exchange Both Underlay and Overlay Routesrbridge-id 42 ip router-id 10.124.12.1 router bgp local-as 4200000000 capability as4-enable fast-external-fallover neighbor leaf-group peer-group neighbor leaf-group password 2 $PVNHITJVPWQ= neighbor leaf-group bfd neighbor superspine-evpn-group peer-group neighbor superspine-evpn-group remote-as 4200000020 neighbor superspine-evpn-group password 2 $PVNHITJVPWQ= neighbor superspine-evpn-group bfd neighbor superspine-ip-group peer-group neighbor superspine-ip-group remote-as 4200000020 neighbor superspine-ip-group password 2 $PVNHITJVPWQ= neighbor superspine-ip-group bfd neighbor 10.12.1.1 remote-as 4200000001 neighbor 10.12.1.1 peer-group leaf-group neighbor 10.12.2.1 remote-as 4200000001 neighbor 10.12.2.1 peer-group leaf-group neighbor 10.12.3.1 remote-as 4200000002 neighbor 10.12.3.1 peer-group leaf-group neighbor 10.12.4.1 remote-as 4200000002 neighbor 10.12.4.1 peer-group leaf-group neighbor 10.12.5.1 remote-as 4200000003 neighbor 10.12.5.1 peer-group leaf-group neighbor 10.12.6.1 remote-as 4200000004 neighbor 10.12.6.1 peer-group leaf-group neighbor 10.12.7.1 remote-as 4200000005 neighbor 10.12.7.1 peer-group leaf-group neighbor 10.12.8.1 remote-as 4200000006 neighbor 10.12.8.1 peer-group leaf-group neighbor 10.41.2.0 peer-group superspine-ip-group neighbor 10.42.2.0 peer-group superspine-evpn-group neighbor 10.43.2.0 peer-group superspine-evpn-group neighbor 10.44.2.0 peer-group superspine-ip-group address-family ipv4 unicast maximum-paths 8 graceful-restart ! address-family l2vpn evpn graceful-restart retain route-target all neighbor superspine-evpn-group activate neighbor superspine-evpn-group next-hop-unchanged neighbor leaf-group activate neighbor leaf-group next-hop-unchanged ! ! interface Loopback 2 no shutdown ip address 10.124.12.1/32 !



!interface FortyGigabitEthernet 42/0/1 mtu 9216 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.12.1.0/31 no shutdown!interface FortyGigabitEthernet 42/0/3 mtu 9216 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.12.2.0/31 no shutdown!interface FortyGigabitEthernet 42/0/5 mtu 9216 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.41.2.1/31 no shutdown!interface FortyGigabitEthernet 42/0/6 mtu 9216 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.42.2.1/31 no shutdown!interface FortyGigabitEthernet 42/0/7 mtu 9216 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.43.2.1/31 no shutdown!interface FortyGigabitEthernet 42/0/8 mtu 9216 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.44.2.1/31 no shutdown!interface FortyGigabitEthernet 42/0/10 mtu 9216 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.12.3.0/31 no shutdown!interface FortyGigabitEthernet 42/0/12 mtu 9216 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.12.4.0/31 no shutdown



!interface FortyGigabitEthernet 42/0/20 mtu 9216 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.12.5.0/31 no shutdown!interface FortyGigabitEthernet 42/0/22 mtu 9216 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.12.6.0/31 no shutdown!interface FortyGigabitEthernet 42/0/28 mtu 9216 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.12.7.0/31 no shutdown!interface FortyGigabitEthernet 42/0/30 mtu 9216 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.12.8.0/31 no shutdown!

Super-Spine Designated to Exchange Only Underlay Routesrbridge-id 67 ip router-id 10.125.5.1 router bgp local-as 4200000020 capability as4-enable fast-external-fallover neighbor edge-group peer-group neighbor edge-group remote-as 4200000021 neighbor edge-group password 2 $PVNHITJVPWQ= neighbor edge-group bfd neighbor pod1_spine-group peer-group neighbor pod1_spine-group remote-as 4200000000 neighbor pod1_spine-group password 2 $PVNHITJVPWQ= neighbor pod1_spine-group bfd neighbor pod2_spine-group peer-group neighbor pod2_spine-group remote-as 4200000010 neighbor pod2_spine-group password 2 $PVNHITJVPWQ= neighbor pod2_spine-group bfd neighbor 10.31.1.1 peer-group edge-group neighbor 10.31.2.1 peer-group edge-group neighbor 10.41.1.1 peer-group pod1_spine-group neighbor 10.41.2.1 peer-group pod1_spine-group neighbor 10.41.3.1 peer-group pod1_spine-group neighbor 10.41.4.1 peer-group pod1_spine-group neighbor 10.41.5.1 peer-group pod2_spine-group neighbor 10.41.6.1 peer-group pod2_spine-group neighbor 10.41.7.1 peer-group pod2_spine-group neighbor 10.41.8.1 peer-group pod2_spine-group address-family ipv4 unicast



maximum-paths 8 graceful-restart ! ! interface Loopback 2 no shutdown ip address 10.125.5.1/32 !!interface FortyGigabitEthernet 67/1/1 mtu 9216 description Link to pod1-spine1 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.41.1.0/31 no shutdown!interface FortyGigabitEthernet 67/1/2 mtu 9216 description Link to pod1-spine2 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.41.2.0/31 no shutdown!interface FortyGigabitEthernet 67/1/3 mtu 9216 description Link to pod1-spine3 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.41.3.0/31 no shutdown!interface FortyGigabitEthernet 67/1/4 mtu 9216 description Link to pod1-spine4 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.41.4.0/31 no shutdown!interface FortyGigabitEthernet 67/1/5 mtu 9216 description Link to pod2-spine1 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.41.5.0/31 no shutdown!interface FortyGigabitEthernet 67/1/6 mtu 9216 description Link to pod2-spine2 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.41.6.0/31 no shutdown!interface FortyGigabitEthernet 67/1/7 mtu 9216 description Link to pod2-spine3



no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.41.7.0/31 no shutdown!interface FortyGigabitEthernet 67/1/8 mtu 9216 description Link to pod2-spine4 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.41.8.0/31 no shutdown!interface FortyGigabitEthernet 67/1/9 mtu 9216 description Link to edge-leaf1 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.31.1.0/31 no shutdown!interface FortyGigabitEthernet 67/1/10 mtu 9216 description Link to edge-leaf2 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.31.2.0/31 no shutdown!

Super-Spine Designated to Exchange Both Underlay and Overlay Routesrbridge-id 68 ip router-id 10.125.5.2 router bgp local-as 4200000020 capability as4-enable fast-external-fallover neighbor edge-group peer-group neighbor edge-group remote-as 4200000021 neighbor edge-group password 2 $PVNHITJVPWQ= neighbor edge-group bfd neighbor pod1_spine-evpn-group peer-group neighbor pod1_spine-evpn-group remote-as 4200000000 neighbor pod1_spine-evpn-group password 2 $PVNHITJVPWQ= neighbor pod1_spine-evpn-group bfd neighbor pod1_spine-ip-group peer-group neighbor pod1_spine-ip-group remote-as 4200000000 neighbor pod1_spine-ip-group password 2 $PVNHITJVPWQ= neighbor pod1_spine-ip-group bfd neighbor pod2_spine-evpn-group peer-group neighbor pod2_spine-evpn-group remote-as 4200000010 neighbor pod2_spine-evpn-group password 2 $PVNHITJVPWQ= neighbor pod2_spine-evpn-group bfd neighbor pod2_spine-ip-group peer-group neighbor pod2_spine-ip-group remote-as 4200000010 neighbor pod2_spine-ip-group password 2 $PVNHITJVPWQ= neighbor pod2_spine-ip-group bfd neighbor 10.32.1.1 peer-group edge-group neighbor 10.32.2.1 peer-group edge-group neighbor 10.42.1.1 peer-group pod1_spine-ip-group



neighbor 10.42.2.1 peer-group pod1_spine-evpn-group neighbor 10.42.3.1 peer-group pod1_spine-evpn-group neighbor 10.42.4.1 peer-group pod1_spine-ip-group neighbor 10.42.5.1 peer-group pod2_spine-ip-group neighbor 10.42.6.1 peer-group pod2_spine-evpn-group neighbor 10.42.7.1 peer-group pod2_spine-evpn-group neighbor 10.42.8.1 peer-group pod2_spine-ip-group address-family ipv4 unicast maximum-paths 8 graceful-restart ! address-family l2vpn evpn graceful-restart retain route-target all neighbor pod2_spine-evpn-group activate neighbor pod2_spine-evpn-group next-hop-unchanged neighbor pod1_spine-evpn-group activate neighbor pod1_spine-evpn-group next-hop-unchanged neighbor edge-group activate neighbor edge-group next-hop-unchanged ! ! interface Loopback 2 no shutdown ip address 10.125.5.2/32 !!interface FortyGigabitEthernet 68/1/1 mtu 9216 description Link to pod1-spine1 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.42.1.0/31 no shutdown!interface FortyGigabitEthernet 68/1/2 mtu 9216 description Link to pod1-spine2 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.42.2.0/31 no shutdown!interface FortyGigabitEthernet 68/1/3 mtu 9216 description Link to pod1-spine3 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.42.3.0/31 no shutdown!interface FortyGigabitEthernet 68/1/4 mtu 9216 description Link to pod1-spine4 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.42.4.0/31 no shutdown!interface FortyGigabitEthernet 68/1/5 mtu 9216 description Link to pod2-spine1 no fabric isl enable no fabric trunk enable



ip mtu 9018 ip proxy-arp ip address 10.42.5.0/31 no shutdown!interface FortyGigabitEthernet 68/1/6 mtu 9216 description Link to pod2-spine2 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.42.6.0/31 no shutdown!interface FortyGigabitEthernet 68/1/7 mtu 9216 description Link to pod2-spine3 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.42.7.0/31 no shutdown!interface FortyGigabitEthernet 68/1/8 mtu 9216 description Link to pod2-spine4 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.42.8.0/31 no shutdown!interface FortyGigabitEthernet 68/1/9 mtu 9216 description Link to edge-leaf1 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.32.1.0/31 no shutdown!interface FortyGigabitEthernet 68/1/10 mtu 9216 description Link to edge-leaf2 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.32.2.0/31 no shutdown!

Edge Leaf!Rbridge-id 71vcs virtual-fabric enableinterface Vlan 3945 description Connectivity to the external router for vrf71!interface Vlan 3957 description Connectivity to the external router for vrf101!interface Vlan 7071 description VLAN 7071, VNI 7071, L3 VNI for VRF71!



interface Vlan 7101 description VLAN 7101, VNI 7101, L3 VNI for VRF101!rbridge-id 71 ip router-id 10.123.4.1 vrf vrf101 rd 10.123.4.1:101 vni 7101 address-family ipv4 unicast route-target export 101:101 evpn route-target import 101:101 evpn ! address-family ipv6 unicast route-target export 101:101 evpn route-target import 101:101 evpn ! ! vrf vrf71 rd 10.123.4.1:71:71 vni 7071 address-family ipv4 unicast route-target export 71:71 evpn route-target import 71:71 evpn ! address-family ipv6 unicast route-target export 71:71 evpn route-target import 71:71 evpn ! ! evpn-instance edge-leaf route-target both auto ignore-as rd auto duplicate-mac-timer 5 max-count 3 ! router bgp local-as 4200000021 capability as4-enable neighbor superspine-evpn-group peer-group neighbor superspine-evpn-group remote-as 4200000000 neighbor superspine-evpn-group password 2 $PVNHITJVPWQ= neighbor superspine-evpn-group bfd neighbor superspine-ip-group peer-group neighbor superspine-ip-group remote-as 4200000000 neighbor superspine-ip-group password 2 $PVNHITJVPWQ= neighbor superspine-ip-group bfd neighbor 10.31.1.0 peer-group superspine-ip-group neighbor 10.32.1.0 peer-group superspine-evpn-group neighbor 10.33.1.0 peer-group superspine-evpn-group neighbor 10.34.1.0 peer-group superspine-ip-group address-family ipv4 unicast redistribute connected network 10.123.3.1/32 maximum-paths 8 graceful-restart ! address-family ipv4 unicast vrf vrf101 redistribute connected neighbor 172.16.101.2 remote-as 101 neighbor 172.16.101.2 password 2 $PVNHITJVPWRNNl5D neighbor 172.16.101.2 update-source ve-interface 3957 maximum-paths 8 ! address-family ipv4 unicast vrf vrf71 redistribute connected neighbor 172.16.71.2 remote-as 101 neighbor 172.16.71.2 password 2 $PVNHITJVPWRNNl5D neighbor 172.16.71.2 update-source ve-interface 3945 maximum-paths 8 ! address-family ipv6 unicast vrf vrf101 redistribute connected neighbor fd2d:d47a:101:1::2 remote-as 101



neighbor fd2d:d47a:101:1::2 activate neighbor fd2d:d47a:101:1::2 password 2 $PVNHITJVPWRNNl5D neighbor fd2d:d47a:101:1::2 update-source ve-interface 3957 maximum-paths 8 ! address-family ipv6 unicast vrf vrf71 neighbor fd2d:d47a:71:1::2 remote-as 101 neighbor fd2d:d47a:71:1::2 activate neighbor fd2d:d47a:71:1::2 password 2 $PVNHITJVPWRNNl5D neighbor fd2d:d47a:71:1::2 update-source ve-interface 3945 maximum-paths 8 ! address-family l2vpn evpn graceful-restart neighbor superspine-evpn-group activate neighbor superspine-evpn-group next-hop-unchanged ! ! interface Loopback 1 no shutdown ip address 10.123.3.1/32 ! interface Loopback 2 no shutdown ip address 10.123.4.1/32 ! interface Ve 3945 vrf forwarding vrf71 ipv6 address fd2d:d47a:71:1::1/64 ip proxy-arp ip address 172.16.71.1/24 no shutdown ! interface Ve 3957 vrf forwarding vrf101 ipv6 address fd2d:d47a:101:1::1/64 ip proxy-arp ip address 172.16.101.1/24 no shutdown ! interface Ve 7071 vrf forwarding vrf71 ipv6 address use-link-local-only no shutdown !interface Ve 7101 vrf forwarding vrf101 ipv6 address use-link-local-only no shutdown !interface TenGigabitEthernet 71/0/36:1 switchport switchport mode trunk switchport trunk allowed vlan add 3921-3969 switchport trunk tag native-vlan spanning-tree shutdown fabric isl enable fabric trunk enable no shutdown!interface FortyGigabitEthernet 71/0/9 mtu 9216 description Link to superspine-1 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.31.1.1/31 no shutdown!interface FortyGigabitEthernet 71/0/10 mtu 9216



description Link to superspine2 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.32.1.1/31 no shutdown!interface FortyGigabitEthernet 71/0/11 mtu 9216 description Link to superspine3 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.33.1.1/31 no shutdown!interface FortyGigabitEthernet 71/0/12 mtu 9216 description Link to superspine4 no fabric isl enable no fabric trunk enable ip mtu 9018 ip proxy-arp ip address 10.34.1.1/31 no shutdown!overlay-gateway edge-leaf type layer2-extension ip interface Loopback 1 attach rbridge-id add 71 map vlan vni auto activate!



References1. BGP MPLS-Based Ethernet VPN

https://tools.ietf.org/html/rfc7432

2. Use of BGP for routing in large-scale data centers

https://datatracker.ietf.org/doc/draft-ietf-rtgwg-bgp-routing-large-dc/

3. Integrated Routing and Bridging in EVPN

https://datatracker.ietf.org/doc/draft-ietf-bess-evpn-inter-subnet-forwarding/

4. RFC 4760: Multiprotocol Extensions for BGP-4

https://datatracker.ietf.org/doc/rfc4760/

5. RFC 4364: BGP/MPLS IP Virtual Private Networks (VPNs)


6. A Network Virtualization Overlay Solution using EVPN

https://datatracker.ietf.org/doc/draft-ietf-bess-evpn-overlay/

7. Brocade Data Center Fabric Architectures white paper

http://www.brocade.com/content/dam/common/documents/content-types/whitepaper/brocade-data-center-fabric-architectures-wp.pdf

8. Brocade VDX hardware installation guides

http://www.brocade.com/content/html/en/hardware-installation-guide/vdx6740-installguide/index.html



https://tools.ietf.org/html/rfc7432

https://datatracker.ietf.org/doc/draft-ietf-rtgwg-bgp-routing-large-dc/

https://datatracker.ietf.org/doc/draft-ietf-bess-evpn-inter-subnet-forwarding/



https://datatracker.ietf.org/doc/draft-ietf-bess-evpn-overlay/





brocade-ip-fabric-bvd-published

Documents