cisco fabric path-690079

7/27/2019 Cisco Fabric Path-690079

1/53

2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 5

Cisco FabricPath Design Guide:

Using FabricPath with anAggregation and Access Topology

December, 2011

For further information, questions and comments please contact [email protected]

Design Guide


2/53


Introduction.............................................................................................................................................................. 3

Hardware and Software Support ............................................................................................................................ 3Next-Generation Data Center Architecture............................................................................................................ 4FabricPath Technology Overview .......................................................................................................................... 7

Spine and Edge ................................................................................................................................................... 8Switch-ids ............................................................................................................................................................ 9Conversational Learning.................................................................................................................................. 11VLAN Trunking.................................................................................................................................................. 11Metrics ............................................................................................................................................................... 11Multipath Load Balancing ................................................................................................................................ 12Multidestination Trees...................................................................................................................................... 12Multicast Forwarding........................................................................................................................................ 14vPC+................................................................................................................................................................... 14

Design Considerations for the Spine/Aggregation Layer ..................................................................................14Topology Considerations................................................................................................................................. 16

Declaring VLANs as FabricPath VLANs ..................................................................................................... 16Declaring Ports as FabricPath Ports .......................................................................................................... 17Topology 0 .................................................................................................................................................... 17Unicast Layer 2 Multipathing ...................................................................................................................... 17

Building a Routed Spine .................................................................................................................................. 17F1 and M1 Card Cooperation ...................................................................................................................... 17M1 Card Routing Traffic............................................................................................................................... 18F1 Card Architecture and MAC Optimization............................................................................................. 18MAC Scalability Considerations ................................................................................................................. 20

Gateway Routing............................................................................................................................................... 22Active/Active Gateway Routing................................................................................................................... 22Benefits of vPC+ at the Spine ..................................................................................................................... 24Dual-Active Exclude (Peer-Link Failure).................................................................................................. 24

Connecting FabricPath Edge or Leaf Layer to the Spine Layer....................................................................25Avoiding Flooding by Tuning ARP and Layer 2 Table ................................................................................... 25Multicast Routing Considerations................................................................................................................... 25

Multicast Forwarding in FabricPath ........................................................................................................... 26Routed Multicast Configuration.................................................................................................................. 26Routed Multicast with vPC+ ........................................................................................................................ 27

Summary Recommendations...........................................................................................................................28Design Considerations for the Edge and Access Layer .................................................................................... 28

FabricPath VLANs............................................................................................................................................. 29MAC Learning at the Access Layer ................................................................................................................. 29Learning the HSRP MAC Address ................................................................................................................... 29Direct Connectivity between Edge Switches.................................................................................................. 30vPC+ at the Edge .............................................................................................................................................. 30Integration with FEX .........................................................................................................................................32Summary............................................................................................................................................................ 33

FabricPath Scalability Considerations ................................................................................................................ 33FabricPath Convergence Times ...........................................................................................................................37Sample Configurations .........................................................................................................................................38

Cisco Nexus 7000 Series Spine 1 (DC1-Agg1) ...............................................................................................38Cisco Nexus 7000 Series Spine 2 (DC1-Agg2) ...............................................................................................42Edge Cisco Nexus 5500 Switch (DC1-5500-1) without vPC+ ........................................................................ 46


3/53


Edge Cisco Nexus 5500 Switch (DC1-5500-1) with vPC+ and FEX A/A ....................................................... 47Edge Cisco Nexus 5500 Switch (DC1-5500-2) with vPC+ and FEX A/A ....................................................... 50


4/53


Introduction

The introduction of Cisco

FabricPath technology in Cisco NX-OS Software Release 5.1(3) brings the benefit of

routing protocols to Layer 2 network Ethernet environments. Cisco FabricPath technology is described in several

documents available on the cisco.com page:

http://www.cisco.com/en/US/prod/switches/ps9441/fabric_path_promo.html

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/at_a_glance_c45-605626.pdf

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-605488.html

http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-

os/fabricpath/configuration/guide/fp_cli_Book.html

The advantages of using FabricPath include:

MAC address scalability with conversational learning

Spanning Tree Protocol independence: No reliance on Spanning Tree anymore. Each switch has a

complete view of the Layer 2 topology, and it calculates the Layer 2 forwarding table based a shortest-path-

first calculation.

Traffic distribution for unicast: Unicast Layer 2 traffic can take multiple equal-cost Layer 2 paths.

Traffic distribution for multicast on multiple distribution trees: Multicast traffic can be distributed along

two multidestination trees.

More direct communication paths: Any topology is possible, so cabling two access or edge switches

directly to each other creates a direct communication path, unlike what happens with Spanning Tree.

Simplicity of configuration: The configuration of FabricPath is very simple. Tuning may still be advised

but by default the switches negotiate switch-ids allocation with a protocol called Dynamic Resource

Allocation Protocol (DRAP).

Loop mitigation with TTL in the frame field: Layer 2 loops, as they are known of today in Spanning-

Tree-Protocol-based Layer 2 networks, are mitigated by dropping frames that have been propagated

across too many hops. The Layer 2 FabricPath frames include a Time to Live (TTL) field that is

decremented at each hop. The value of the TTL field is 32.

This document describes one of many possible designs that are enabled by Cisco FabricPath, and it includes

tuning recommendations and sample configurations.

Hardware and Software Support

The solution described in this document requires the following hardware and software:

Cisco Nexus

5500 products as the edge (or access) device in a FabricPath topology with NX-OS Software

Release 5.1(3)N1(1). Note that in the context of FabricPath domain, the edge device is also known as the

leaf device.

Cisco Nexus 7000 Series products as the spine (or aggregation) device in a FabricPath topology with NX-

OS 5.1(1) Release or later. At the time of this writing, the Cisco Nexus 7000 Series configuration that

enables FabricPath is based on the F1 series cards, which consist of 32 ports 1/10 Gbps. The Layer 3

functionality is provided by the use of M1 series cards.
http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/fabricpath/configuration/guide/fp_cli_Book.htmlhttp://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/fabricpath/configuration/guide/fp_cli_Book.htmlhttp://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-605488.htmlhttp://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/at_a_glance_c45-605626.pdfhttp://www.cisco.com/en/US/prod/switches/ps9441/fabric_path_promo.html


5/53


Cisco Nexus Fabric Extenders (FEX) platforms are supported with the FEX ports acting as FabricPath edge

ports. FEX can be used to augment the number of edge ports or Classic Ethernet (CE) ports without adding

configuration points.

Proper licensing: You need the ENHANCED_LAYER2_PKG license installed. For details, visit:

http://www.cisco.com/en/US/docs/switches/datacenter/sw/nx-os/licensing/guide/Cisco_NX-

OS_Licensing_Guide_chapter1.html.

Note: The F1 cards are complemented by M1 card for routing purposes. When using M1 cards in the same

virtual device context (VDC) as the F1 card, routing is offloaded to the M1 cards, and more routing capacity is

added to the F1 card by putting more M1 ports into the same VDC as the F1 card.

This design guide refers to the following FabricPath features:

FabricPath Shortest-Path-First-based forwarding

FabricPath Layer 2 multipathing (Unicast traffic) FabricPath multidestination trees load balancing (unknown unicast, broadcast, and multicast traffic)

Active/Active gateway routing using vPC+ configuration

vPC+ at the edge switch for the purpose of connecting dual-homed servers or dual-homed Cisco Fabric

Extenders

Next-Generation Data Center Architecture

The introduction of FabricPath enables a variety of Layer 2 topologies that were not possible before. Figure 1

illustrates some of them.

Figure 1. FabricPath enables new topologies
http://www.cisco.com/en/US/docs/switches/datacenter/sw/nx-os/licensing/guide/Cisco_NX-OS_Licensing_Guide_chapter1.htmlhttp://www.cisco.com/en/US/docs/switches/datacenter/sw/nx-os/licensing/guide/Cisco_NX-OS_Licensing_Guide_chapter1.html


6/53


On the top-left of Figure 1 (topology a), you see a design where two spines connect to a set of edge devices,

which looks more like an aggregation/access design. However, in contrast to designs based on Spanning Tree

Protocol, any two endpoints can take all available paths via either spine.

The second design, topology b at the top-right, illustrates a topology where servers connect to edge devices in

vPC mode, which in the case of FabricPath is called vPC+.

The left-bottom design, topology c, illustrates the ability for FabricPath to use multiple spines instead of just two.

This gives a very granular control on the available bandwidth: the more spines you add, the lower oversubscription

between the edge and the rest of the network.

The bottom-right design, topology d, illustrates the ability for FabricPath to provide direct connectivity between

edge devices in case direct communication between any edge devices is suitable. Most of these topologies would

not make sense with Spanning Tree, but they are almost plug-and-play topologies in a FabricPath environment.

Note: Direct connectivity between the spine devices is not strictly necessary. It depends on where the routing is

performed and on whether you consider it necessary to establish a direct communication path between spines,

whether spines are just used for fabric switching or are used for end node attachment too, and so on.

Figure 2 illustrates the traffic paths that any two given servers can take across a Cisco FabricPath topology:

Figure 2. Traffic paths in FabricPath designs


7/53


The Layer 3 demarcation point can be placed in several ways, but a very simple deployment model is to place the

Layer 3 boundary at the aggregation point where all edge or access switches are aggregated. Figure 3 illustrates

adding the routing function at the spine layer for the FabricPath cloud to connect to the rest of the network

infrastructure:

Figure 3. Placing the Router at the Spine Layer

When placing the routing at the spine, the spine keeps switching Layer 2 traffic based on the switch-id, but it also

becomes an edge for routed traffic. Yet another option to place the routing function is to connect the routers as an

edge device to the spine. This approach ensures that the spine switches traffic exclusively based on switch-id, and

it makes changes to the spine configuration very rare. Figure 14 illustrates this design. Analyzing this last design

option is outside of the scope of this document.

In this design guide, the focus is on a topology that uses two spines and multiple edges. The reason for this is that

its an easy transition for customers who are deploying Spanning Tree today and want to benefit from FabricPath

without having to rewire their data center. In addition, it is easy to add additional spines in case you want to lowerthe oversubscription ratio, without any substantial change to the design considerations that are the subject of this

document.


8/53


Figure 4 describes the topology used in this design guide:

Figure 4. Topology used in this design guide

If you want, you can add ports to this design by adding Cisco Fabric Extenders (FEX), as shown in Figure 5:

Figure 5. Adding FEX to a FabricPath topology

FabricPath Technology Overview

This section provides an overview of FabricPath but it is not meant to be a tutorial on the technology. Several

other white papers already exist on the Cisco public webpage that cover this topic.

The reason that FabricPath is almost a plug-and-play protocol, in contrast to several other technologies available

in the market, is that it enables each device to build an overall view of the topology in a way that is similar to Open

Shortest Path First (OSPF) protocol and other link state protocols.


9/53


Each device in the topology is identified by a switch-id and all Layer 2 forwarding tables are built based on

reachability to each switch-id (as opposed to reachability to a MAC address). The switch-id is dynamically

assigned via the Dynamic Resource Allocation Protocol (DRAP), so no manual allocation of switch-ids is required.

However, it is possible to manually assign a SW-ID while a dynamic check for misconfiguration is performed (as

discussed in the Switch-ids section).

FabricPath multipath forwarding relies on building a view of the network using a Link State Routing Protocol similar

to the one used in Layer 3 networks. This is achieved with a Layer 2 protocol based on Intermediate-System-to-

Intermediate-System (IS-IS) routing, which doesnt require any Layer 3 capability on the FabricPath switching

devices.

Spine and Edge

In FabricPath topologies, there are two types of functions (which can be performed by all FabricPath hardware):

Edge (or leaf) devices: These devices have ports connected to Classic Ethernet devices (servers, firewalls,

router ports, and so on) and ports connected to the FabricPath cloud (or FabricPath ports). Edge devices

are able to map a MAC address to the destination switch-id.

Spine devices: These devices exclusively interconnect edge devices. Spine devices switch exclusively

based on the destination switch-id.

Figure 6. Classic Ethernet ports in a FabricPath topology

Traffic from edge ports is looked up in the Layer 2 forwarding table, and encapsulated into the MAC-in-MAC frame

whose destination switch-id is the switch that the destination host is attached to. This allows sending traffic along

multiple paths with the same end-to-end total cost and provides the ability to construct practically any topology and

be sure that traffic will take the most direct path (lowest cost) between any two devices.


10/53


Switch-ids

In order to understand these concepts, lets consider the topology in Figure 7:

Figure 7. Example switch-id allocation

The following CLI capture from an edge device shows the output of the show fabric switch-id command. The

output shows all the switch-ids present in the topology, and it highlights with a * the switch-id of the device that theuser is on:

DC1-5500-1# show fabricpath switch-id

FABRICPATH SWITCH-ID TABLE

Legend: * - this system

=========================================================================

SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED

----------+----------------+------------+-----------+--------------------

*111 0005.73d4.9141 Primary Confirmed Yes No

112 0005.73fc.2001 Primary Confirmed Yes No

113 0005.73fc.207c Primary Confirmed Yes No

121 0022.5579.d3c1 Primary Confirmed Yes No

122 108c.cf18.0941 Primary Confirmed Yes No

The edge switch maintains a MAC address table with the mapping of MAC address to switch-ids as well as the

local source MACs from its CE ports, as illustrated here:

DC1-5500-1# show mac address-table vlan 101

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports

---------+-----------------+--------+---------+------+----+------------------

* 101 0000.0700.0000 dynamic 0 F F Eth1/5

* 101 0000.0700.0100 dynamic 10 F F 112

* 101 0022.5579.d3c1 dynamic 10 F F 121

* 101 108c.cf18.0941 dynamic 0 F F 122

As you can see from the output, the locally attached MAC addresses are associated with the local port. The

remote MAC addresses are associated with the switch-id of the destination.

On the spine devices, there is no MAC address table learning, so the forwarding table is purely switch-id-based. In

the following example, you can see each switch-id and which port can be used to reach the destination:

FabricPath Unicast Route Table for Topology-Default


11/53


0/121/0, number of next-hops: 0

via ----, [60/0], 3 day/s 08:11:51, local


via Eth8/3, [115/40], 0 day/s 06:15:55, isis_fabricpath-default1/112/0, number of next-hops: 1

via Eth8/5, [115/40], 3 day/s 08:09:17, isis_fabricpath-default

[]

Figure 8 illustrates how traffic is forwarded from MAC address A to MAC address B in a Cisco FabricPath

topology:

Figure 8. FabricPath forwarding

Notice how the switch-id values are assigned by default to each FabricPath device via the DRAP protocol.

However, it may be good practice to manually assign them in order to provide a more meaningful numbering

scheme. In this latter scenario, a specific check is performed every time a new switch-id value is configured on a

given FabricPath node, in order to prevent you from configuring a duplicate switch-id within the same domain.

However, if two FabricPath switches from two different and isolated FabricPath domains converge into a single

FabricPath domain, a mechanism intervenes to resolve conflicts:

DC1-5500-1# 2011 Aug 17 15:32:57 DC1-5500-1 %$ VDC-1 %$ fabricpath %FABRICPATH-2-FABRICPATH_LINK_BRINGUP_STALLED_STATIC: Link bringup stalled due to conflicts

DC1-5500-1# show fabricpath conflict all

====================================================

Fabricpath Port State Table

Port State

---------------+-------------------------+----------

port-channel2 Suspended due to conflicts

==============================================

Fabricpath Conflicts

SYSTEM-ID SWITCH-ID STATIC

---------------+--------------+---------------

0005.73d4.9141 3 Yes

108c.cf18.0941 3 Yes


12/53


Conversational Learning

Edge switches learn MAC addresses only if they have a node connected on the edge port or if the switch is a

remote MAC and is in active conversation with a known MAC address connected to an edge port. In other words,a simple flood of traffic (either unknown unicast or broadcast) from a remote switch is not going to populate the

MAC address table of an edge switch.

VLAN Trunking

In spanning tree networks, the user has to specify which VLANs belong to which port by using the switchport

trunk allowed vlan command. With FabricPath, the user doesnt need to explicitly specify which VLANs are

carried on a FabricPath-enabled link (VLANs obviously must be defined on the switches).

The configuration of a FabricPath core port is performed with the command switchport mode fabricpath. The

command switchport mode trunk puts a port into Classic Ethernet mode instead of FabricPath mode.

MetricsThe preferred path to any switch-id is calculated based on the metric to any given destination. The metric is as

follows:

1-Gbps Ethernet links have a cost of 400

10-Gigabit Ethernet links have a cost of 40

20-Gbps have a cost of 20

Figure 9 illustrates the use of the linkcost in a real topology:

Figure 9. Metrics calculation in FabricPath

The following code shows the cost of fabricpath links from the switch 112 from Figure 9.

DC1-5500-2# show fabricpath isis interface br

Fabricpath IS-IS domain: default

Interface Type Idx State Circuit MTU Metric Priority Adjs/AdjsUp

--------------------------------------------------------------------------------

Ethernet1/1 P2P 1 Up/Ready 0x01/L1 1500 40 64 1/1



Multipath Load Balancing

Figure 10 illustrates unicast load balancing in a FabricPath network:


13/53


Figure 10. Unicast load balancing in FabricPath

In this example, each edge device has two equal-cost-path routes to each others spine:

DC1-5500-2: show fabricpath route:




The load balancing is based on fields such as the IP address, Layer 4 port, and so on, and it is configured with the

following command:

DC1-5500-2(config)# fabricpath load-balance unicast < destination, include-vlan,layer-3, layer4, mixed, source, source-destination>

Multidestination Trees

In FabricPath multicast traffic, broadcast and flooded traffic is forwarded along a multidestination tree. FabricPath

allows for multiple multidestination trees in order to achieve traffic load balancing for multidestination frames. This

is achieved by using a special Tag present in FabricPath frames, which is called Forwarding Tag or FTag.

The FTag, together with the type of the frames destination address (unknown unicast, broadcast, or multicast),

selects a forwarding graph, which is the set of possible switches and interfaces along which the frame is

forwarded. At each switch, the FTag carried in the frame header is used to forward the frame along the interfaces

that are in the forwarding graph.

FabricPath uses a link state protocol to determine the forwarding trees in the network. FabricPath technology

allows the definition of multiple topologies.

At the time of this writing, only one topology is supported, in which there are two multidestination trees. As Figure

11 illustrates, two FTags are used: FTag1 and FTag2. In this example, each one of these multidestination trees is

rooted at one of the spines.


14/53


Figure 11. FTag trees assignment in FabricPath

FTag1 is used for the first multidestination tree and FTag2 is used for the second multidestination tree. FTag trees

are used as follows:

FTag1 is used for unknown unicast, broadcast, and multicast. The highest priority switch in the FabricPath

topology is chosen as the root for FTag1. As discussed later in the guide, it is possible to modify the default

FabricPath root priority value in order to place the root for FTag1 or FTag2 on a given device in a

deterministic fashion. If the default priority is maintained on all the switches as part of the FabricPath

domain, the switch system-id value is used as a tiebreak to determine the switch with the highest root

priority.

FTag2 is used only for multicast traffic. The second highest priority switch in the FabricPath topology is

chosen as the root for FTag2.

Here are the factors, in order, that are used in order to elect the root for FTag1 and subsequently for FTag2:

Highest root priority

Highest system-id Highest switch-id

Figure 12 illustrates the forwarding of flooded frames along the FTag1 tree:

Figure 12. Forwarding of unknown unicast and broadcast frames


15/53


Multicast Forwarding

Multicast traffic forwarding uses FTag trees based both on load balancing of multicast frames and on tree pruning

that is governed by Internet Group Management Protocol (IGMP) snooping (platform-specific load-balancing

methods are discussed in the design sections of this guide). The edge device performs IGMP snooping and the

member reports are conveyed in FabricPath via the Group-Membership Link State Protocol (GM-LSP). IGMP

snooping and IS-IS work in conjunction to build per-VLAN multicast group-based trees in the FabricPath network.

vPC+

A virtual PortChannel (vPC) allows links that are physically connected to two different Cisco Nexus switches to

appear as a single PortChannel to a third device. This provides a loop free topology eliminating the spanning-tree-

blocked ports and maximizing the bandwidth usage. In a FabricPath network, a host or legacy (not FabricPath-

enabled) Ethernet switch can be connected through a port channel to two FabricPath edge switches by using a

configuration construct called emulated switch. The emulated switch implementations in FabricPath, where two

FabricPath edge switches provide a vPC to a third-party device, is called vPC+.

Emulated switch is a construct in which two FabricPath switches emulate a single switch to the rest of the

FabricPath network. The packets originated by the two emulated switches are sourced with the emulated switch-

id. The other FabricPath switches are not aware of this and simply see the emulated switch (identified by a

dedicated switch-id value) as reachable through both switches. This means that the two emulated switches have

to be directly connected via peer link, and there should be a peer-keepalive path between the two switches to form

the vPC+.

Other than the fact that a peer-link is used to synchronize MAC addresses between vPC+ peers, the peer-link is a

regular FabricPath link, and as such it can be used by orphan ports for direct communication.

The following list provides the reference terminology for vPC+ components:

vPC+: The functionality enabled on two independent upstream devices to make them appear as a single

logical device to the downstream switches, allowing the establishment of PortChannel connections. All the

MAC addresses learned from devices locally attached to downstream devices are advertised to the

FabricPath domain as connected to the defined emulated switch.

vPC+ peer device: One of a pair of devices that are connected with the special port channel known as the

vPC+ peer link.

vPC+ peer link: The link used to synchronize states between the vPC+ peer devices. Both ends must be

on 10 Gigabit Ethernet interfaces.

vPC+ domain: This domain is formed by the two vPC+ peer link devices. It is also a configuration mode for

configuring some of the vPC+ peer link parameters.

vPC+ peer-keepalive link: The peer-keepalive link, a Layer 3 link between the vPC+ peer devices, is used

to ensure that both devices are up. The peer-keepalive link sends periodic keepalive messages between

the vPC peers on an out-of-band path.

vPC+ member port: Interface that belongs to the vPC+.

Design Considerations for the Spine/Aggregation Layer

The specific topology discussed and validated in the context of this design guide consists of a pair of spine

devices connected to multiple edge devices. Cisco Nexus 7000 Series platforms are used as spine devices,

whereas Cisco Nexus 5500 Switches are connected at the edge.


16/53


The spine in this design is also used to perform the routing function between the FabricPath cloud and the rest of

the network. The default gateway for the servers is located in the spine layer, which, being also the router in the

topology, performs the function of an edge in FabricPath. At the time of this writing, the Cisco Nexus 7000 Series

hardware combination that provides this capability consists of F1 cards (spine function) together with the M1 cards

(routing function).

In the topology shown in Figure 13, the spine layer, which is also an aggregation layer since it aggregates multiple

edge switches, operates simultaneously in two modes:

It forwards Layer 2 traffic between the edge devices exclusively based on the destination switch-id, without

the need to learn MAC addresses (east-to-west traffic).

It also learns MAC addresses exclusively for the purpose of encapsulating routed traffic into FabricPath

frames (north-to-south traffic).

Figure 13. Layer 3 Aggregation/Spine

These two behaviors are completely independent, in that even if, as a theoretical example, the 128-K adjacency

table of the M1 card fully filled up, the E-W Layer 2 traffic would still be forwarded based on the switch-id and not

flooded.

Compared with a spanning-tree design or a vPC design, this design includes the following advantages:

Ease of configuration

Multipath forwarding for unicast and multicast Layer 2 and Layer 3 traffic

Faster convergence times

Loop mitigation thanks to the use of a Time to Live (TTL) field in FabricPath frames

The focus of this section is on the design of this hybrid Layer 2/Layer 3 spine/aggregation layer. This is not the

only possible design option. For instance, you can also add Layer 3 to an edge switch and use the spine as a pure

Layer 2 spine, or connect a pair of Layer 2 edge switches to dedicated routing devices, as highlighted in Figure

14. It is outside of the scope for this document to discuss these specific design options further.


17/53


Figure 14. Fabricpath design with separate L3 edge

Topology Considerations

FabricPath technology allows building multiple logical topologies on the same physical topology. At the time of this

writing, this functionality has not yet been enabled by the software, so you should not attempt to create additional

topologies besides the default topology 0.

Declaring VLANs as FabricPath VLANs

All VLANs that are meant to be forwarded according to Cisco FabricPath rules must be created as FabricPath

VLANs.

A VLAN needs to be explicitly configured for Classic Ethernet (CE) mode or for FabricPath mode. By default, all

VLANs are in CE mode. These CE VLANs are not brought up on FabricPath core links. They can be forwarded on

the FabricPath edge ports locally or to other Classic Ethernet switches.

The command required to configure a VLAN in FabricPath mode is:

DC1-Agg1(config)# vlan 101

DC1-Agg1(config-vlan)# mode fabricpath


18/53


The FabricPath VLANs are forwarded on FabricPath core links using MAC-in-MAC header encapsulation and on

FabricPath edge links without the MAC-in-MAC header. FabricPath VLANs can be grouped into topologies for

traffic engineering. By default, all FabricPath VLANs belong to the base topology 0.

CE VLANs and FabricPath VLANs cannot share a link unless the link is a FabricPath edge port.

Declaring Ports as FabricPath Ports

All ports that are carrying FabricPath traffic are defined as type fabricpath. With FabricPath, you dont configure

trunking and pruning of VLANs on FabricPath ports. The required configuration to declare a port as a FabricPath

port is very simple:

DC1-Agg1(config)# interface ethernet 8/3

DC1-Agg1(config-if)# description FP Link to DC1-5500-1

DC1-Agg1(config-if)# switchport mode fabricpath

DC1-Agg1(config-if)# no shutdown

Topology 0

Topology 0 is the default topology and cannot be removed; it exists by default:

DC1-Agg1(config)# fabricpath topology ?

Fabricpath Topology ID 1-63

DC1-Agg1# show fabricpath isis topology ?

Specific topology information

summary Display summary topology information

The only configuration that you may want to modify in order to optimize traffic distribution is the root choice for the

multidestination trees.

Unicast Layer 2 Multipathing

Unicast Layer 2 multipathing is on by default, but you can select the load balancing mechanism (by default it uses

Layer 2, Layer 3, or Layer 4 source and destination IP and VLAN) for Layer 2 traffic with the following command:

fabricpath load-balance unicast < destination, include-vlan, layer-3, layer4,mixed, source, source-destination>

Building a Routed Spine

In this design recommendation, the spine also performs routing functions, and as a result the default gateway

must be configured. When the spine is built with the Cisco Nexus 7000 Series product family, you need to

understand the integration of F1 and M1 cards. This integration provides the combined benefits of a FabricPath

switch for Layer 2 traffic and decapsulation and encapsulation of Layer 3 traffic into the FabricPath cloud, without

a need for any external wiring or additional equipment.

F1 and M1 Card Cooperation

When an F1 card is used in a Cisco Nexus 7000 Series that provides the spine function (pure spine deployment),

no remote MAC address learning occurs on the spine, as previously described. The F1 card doesnt do routing

either.

M1 cards instead provide the ability to perform routing between VLANs and toward the routed domain of the data

center network.


19/53


If an M1 card is present in the same virtual device context (VDC) as the F1 card, the F1 card will perform learning,

but this doesnt mean that the F1 card forwards FabricPath traffic based on MAC addresses. The F1 card forwards

the east-to-west traffic using the remote switch-id.

In this configuration, MAC addresses are learned on the F1 card exclusively with the purpose of being able to

encapsulate north-to-south routed traffic into FabricPath.

In addition to this, the learning is optimized in the sense that each component of the F1 card learns only the locally

connected MAC addresses.

This optimization requires the use of FabricPath on the links connecting to the edge. The reason is that the edge

encapsulates the frames with the destination switch-id so even if learning is performed, it is only for the purpose of

routed traffic, not for the east-to-west traffic.

In order to fully understand, this concept you need to consider the architecture of the F1 card, which is highly

optimized for MAC scalability.

M1 Card Routing Traffic

Figure 15 illustrates the traffic flow for FabricPath traffic that is routed via the M1 card. Traffic routed from one

FabricPath VLAN to another FabricPath VLAN enters via F1 cards, exists via F1 cards, and is routed by the M1

card.

If traffic needs to be routed from the FabricPath cloud to a Layer 3 infrastructure, it enters from an F1 card and it

exits via an M1 card port.

Figure 15. Traffic forwarding with M1/F1 linecards in a Nexus 7000 chassis

F1 Card Architecture and MAC Optimization

East-to-west FabricPath traffic (or in other words pure Layer 2 traffic) doesnt engage the M1 card at all, nor does

it require it. Pure Layer 2 traffic is handled by F1 cards and switched exclusively based on the switch-id

information, as illustrated in Figure 16:


20/53


Figure 16. Forwarding with L2 traffic in a mixed M1/F1 chassis

Because the M1 card is present and routed traffic requires being encapsulated into a FabricPath header, the F1

card also stores MAC address mappings to destination switch-ids; hence some learning occurs, which is not used

for Layer 2 switched traffic. In order to understand the scalability of this solution, you have to consider the

architecture of the F1 card, which is based on the Switch-on-a-Chip concept (SoC).

Each SoC performs switching as it if was an independent switch, and forwarding tables are not synchronized.

Therefore, by distributing edge switches on multiple SoCs, you can take advantage of the aggregated capacity of

all forwarding tables on the SoCs (Figure 17).

Figure 17. High level view of ASICs arrangement in the F1 linecard


21/53


MAC Scalability Considerations

In an M1+F1 solution and assuming that the edge switches are connected via FabricPath to the F1 card, each

SoC learns the MAC addresses that are specific to it.

Layer 2 traffic that doesnt leverage the default gateway doesnt require space in the Layer 2 table as it is switched

exclusively based on the switch-id information. The MAC learning is required exclusively for routing purposes and

Layer 2 switched traffic source MACs arent learned on the F1 card unless a switch virtual interface (SVI) is

present on the VLAN.

Since each SoC provides 16K MAC address capacity for the purpose of encapsulating routed traffic, the hardware

could provide 256K MAC addresses capacity on the F1 card and 128K MAC addresses on the M1 card so the

solution could accommodate up to 128K end devices routed by this combination of cards. A feature called

mac-proxy (currently planned for CY12) could unleash this scalability. At the time of this writing, in certain failure

scenarios, the traffic routed to the fabricpath cloud may use up to a maximum of 16k MAC addresses, after which

traffic routed to any additional MAC would be flooded on one of the FTAG trees.

Notice that this behavior has no impact for the Layer 2 east-to-west traffic because this one is switch exclusively

based on the remote switch- ids.

Figure 18 illustrates how the Nexus 7k builds the adjacency table for traffic routed to the Fabricpath cloud: 16k

source MAC addresses sourced by Cisco Nexus 5500-1 are learned only by the SoC associated with port 8/3.

The 8k source MAC addresses sourced by the Cisco Nexus 5500-2 are learned only by the SoC associated with

port 8/5.

Figure 18. MAC address learning with M1/F1 linecards

The following CLI capture shows the SoC ASIC utilization with the traffic pattern described in Figure 18.

L2 Forwarding Resources

-----------------------

L2 entries: Module inst total used mcast ucast lines lines_full

8 1 16384 4 0 4 1024 0

8 2 16384 16002 0 16002 1024 642

8 3 16384 8001 0 8001 1024 0

8 4 16384 1 0 1 1024 0

8 5 16384 0 0 0 1024 0

8 6 16384 0 0 0 1024 0


22/53


8 7 16384 0 0 0 1024 0

8 8 16384 0 0 0 1024 0

8 9 16384 2001 0 2001 1024 0

So what is the scalability in terms of MAC addresses on a FabricPath design? For traffic that is purely Layer 2 -

that is, traffic whose VLAN doesnt include an SVI - the scalability is exclusively dependent on the MAC table sizes

of the edge devices in the topology; thus, the more edge devices, the more MAC entries you can switch. As an

example of such VLANs, consider VLANs that are used to interconnect virtual machines (VMs) to create a virtual

data center in a multitier environment. These VLANs often do not require any routing.

Figure 19 illustrates a typical virtual data center (vDC): there can be several isolated networks and fenced

networks (that is, connected via a gateway), so not all networks require a default gateway on the spine.

Figure 19. Example of a Vitual datacenter network

How many edge switches you can have in the same Layer 2 domain is bound to the maximum number of switch-

ids that have been validated by testing. These numbers are published in the release notes or in the configuration

limits in the documentation on the Cisco Nexus 5000 Series and Cisco Nexus 7000 Series.

Note: At the time of this writing, this number varies from 64 to 128 switch-ids, depending on the software

release and the platform.

For traffic whose VLANs are routed, the scalability is 16k-128k MAC addresses for routed traffic, which is provided

by the M1/F1 combination and is dependent on the traffic patterns.

Note: As previously mentioned at the time of this writing and with this specific design, should the links

connecting edge to spine break, forcing traffic through the asymmetric path connecting the spines, the number of

routed MAC addresses is reduced to 16k.

A functionality named mac-proxy, is on the roadmap for a future NX-OS SW release that will eliminating this

caveat. Once this functionality is available (currently planned for 1HCY12), it would truly be possible to consider

the MAC address scalability off a FabricPath domain as only dictated by the M1 MAC table space (minimum 128k)

even in presence of multiple link failures.


23/53


Notice that even if the total MAC count is exceeded, only routed traffic will experience flooding, while Layer 2 traffic

will be switched based on the switch-id information. For this reason, it wont be experiencing flooding.

For more information about scalability, see the FabricPath Scalability Considerations section at the end of thisguide.

Gateway Routing

The gateway protocols work in FabricPath the same way they work in regular Ethernet networks. The default

Active/Standby behavior of Hot Standby Router Protocol (HSRP) doesnt negate the value of FabricPath. Even

without any further configuration than the standard configuration of HSRP, the Layer 2 traffic (east-to-west) would

benefit from multipath forwarding.

Routed traffic (south-to-north) would instead be forwarded only to the active HSRP device, as shown in Figure 20:

Figure 20. The default HSRP configuration doesnt make use of both spines

Active/Active Gateway Routing

To perform traffic forwarding from multiple spines, you can use the Gateway Load Balancing Protocol (GLBP),

which hands out a different gateway MAC (up to four different MACs) in a round-robin fashion. A better idea is to

use the concept of vPC+ at the spine.

The spine devices are connected by an additional FabricPath link (which would be recommended anyway to

optimize multidestination tree forwarding) and by defining it as a vPC+ peer-link.

The vPC+ peer-link must be built using F1 ports and not M1 ports, since it must be configured as a FabricPath link

(Figure 21).


24/53


Figure 21. vPC+ makes active and standby HSRP devices active

Figure 21 illustrates how traffic originating from the server connected to the edge and destined to its default-gateway can use two equal cost paths. Notice that the Active and Standby HSRP roles refer only to the control

plane, as both aggregation devices are able to actively route traffic on the data plane.

The configuration to achieve this behavior is as simple as follows:

vpc domain 1

role priority 110

peer-keepalive destination 10.60.17.141

fabricpath switch-id 1000

interface port-channel10

switchport mode fabricpath

vpc peer-link

Even if the configuration requires the definition of a vPC domain, there is no real vPC port in the design described

in this topology. Notice that you need to configure a switch-id under the vPC domain configuration. This identifies

the emulated switch.

Because of this switch-id configuration, HSRP, VRRP, and GLBP announce their vMAC as coming from the

emulated switch-id instead of the switch-id of each individual spine. By doing this, the edge switches can forward

HSRP frames to either spine.

On both spines, the HSRP MAC address is programmed in the MAC tables with the special flag (G-flag) that

indicates that this traffic is meant to be routed.

DC1-Agg1# show mac address-table

Legend:


age - seconds since last seen, + - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

G 101 0000.0c07.ac01 static - F F sup-eth1(R)

Benefits of vPC+ at the Spine

In this design guide, the use of vPC+ at the spine is strongly recommended. vPC+ gives you the ability to forward

routed traffic to multiple routing engines as well as to optimize failover times.


25/53


As a result of this configuration, the edge switches learn the association of HSRP Virtual MAC addresses with the

emulated switch-id instead of the individual spine switch-id:

DC1-5500-1# show mac address-table | include 0c07

* 101 0000.0a00.0c07 dynamic 10 F F Eth1/5

* 101 0000.0c07.ac01 dynamic 0 F F 1000

As Figure 22 illustrates, the use of vPC+ allows the routed traffic between any two given endpoints to benefit from

both Layer 2 equal-cost multipathing and from the aggregated routing capacity of the spines.

Figure 22. Traffic load balancing for both Layer 2 and Layer 3 traffic

The existence of a dual-active path from each edge device to the emulated switch can be verified by looking at the

FabricPath routing table, as highlighted here:

DC1-5500-1# sh fabricpath route

FabricPath Unicast Route Table

a/b/c denotes ftag/switch-id/subswitch-id[x/y] denotes [admin distance/metric]

ftag 0 is local ftag

subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default




Dual-Active Exclude (Peer-Link Failure)

As a result of declaring the link that connects the spines as a vPC peer-link, the default behavior of vPC applies

whereby if the peer-link goes down, the SVIs on the vPC secondary device are shut down.

In the context of FabricPath designs, this behavior is not beneficial, because the FabricPath links are still

available, and theres no good reason to shut down the SVIs on the secondary.

It is thus recommended to complete the vpc domain configuration with the following command:

vpc domain 1

dual-active exclude vlan


26/53


Connecting FabricPath Edge or Leaf Layer to the Spine Layer

When connecting the FabricPath edge layer to the spine layer, you can use any ports on the spine layer F1 cards

unless the scalability of MAC addresses in the topology is a key design factor. In this case, maximizing the use ofthe SoC in the F1 card is beneficial.

In order to do this, you just have to distribute edge switches in a way that they use different SoCs. As an example,

you could connect edge-1 to ports 1 and 2 of the F1 card, edge-2 to ports 3 and 4, and so on (Figure 23).

Figure 23. Best way to connect edge to spine F1 card

Avoiding Flooding by Tuning ARP and Layer 2 Table

The default aging time on the Layer 2 forwarding table on the Cisco Nexus 7000 Series is 1800 seconds. The

default Address Resolution Protocol (ARP) timeout is 1500 seconds. In order to avoid flooding routed traffic, it is

beneficial to make sure that on the spine, the ARP timeout is more aggressive than the Layer 2 forwarding table

timeout, which is normally the default.

Tuning the MAC address table on the edge device is not necessary because in this architecture, the edge doesnt

perform learning based on the ARP. The edge device in a FabricPath topology learns MAC addresses only for

active conversations.

In the spine, the presence of the M1 card in the same VDC as the F1 card requires the F1 card to learn MAC

addresses for the sole purpose of rewriting the routed traffic with the FabricPath information. In this specific case,

the SoCs learn MAC addresses based on ARP, but they dont use this MAC information for Layer 2 traffic

forwarding.

Multicast Routing Considerations

Multicast routing and FabricPath switching do not require particular configuration or tuning on the aggregation

layer devices. Only PIM may require tuning if you want to reduce failover times, just like you would do in regularrouted designs.

Multicast Forwarding in FabricPath

As previously mentioned, FabricPath builds two multidestination trees with two different roots: one for FTag1 and

one for FTag2. At Layer 2, multicast traffic is hashed to either tree in order to utilize both of them. The hashing to

either multidestination tree is platform-dependent; for instance, it can include the VLAN field or the IP address

fields.

In order to maximize the efficiency of the traffic distribution, it is advisable to manually set the priority for the root

for the FTag1 and FTag2 at the spine:


27/53


Agg1:

fabricpath domain default

root-priority 66

Agg2:fabricpath domain default

root-priority 65

Traffic load-balancing can be configured with the following command:

DC1-Agg1(config)# fabricpath load-balance multicast ?

destination Include destination parameters

[[

source-destination Include source and destination parameters

symmetric Symmetric (default)

xor Include ex-or of source and destination parameters

IGMP snooping is used to build the Layer 2 multicast MAC forwarding table at the edge. The information is used to

prune traffic on a per-VLAN basis on the two multidestination trees.

You can verify the traffic path for a given multicast group using the following command:

DC1-Agg1# show fabricpath mroute

(ftag/1, vlan/101, 0.0.0.0, 239.1.1.5), uptime: 00:30:24, isis

Outgoing interface list: (count: 2)

Interface port-channel10, uptime: 02:15:06, isis


(ftag/1, vlan/101, 0.0.0.0, 239.1.1.6), uptime: 00:25:27, isis

Outgoing interface list: (count: 1)


Routed Multicast Configuration

Its outside of the scope of this document to describe all the details of the routed multicast configuration, since it

follows the same rules as in a non-FabricPath design:

Enable PIM under the SVI interface.

IGMP is automatically enabled under the SVI when PIM is enabled.

Define the rendezvous points (RP) in the network and/or configure the spine device for auto-RP. NX-OS

provides a specific functionality, based on RFC 4610, called Anycast-RP Using Protocol Independent

Multicast (PIM). This functionality makes it possible to provide RP redundancy in a simpler fashion than

when deploying Multicast Source Discovery Protocol (MSDP).

The following is a sample configuration:

ip pim anycast-rp 1.1.1.100 1.1.1.1

ip pim anycast-rp 1.1.1.100 1.1.1.2

ip pim rp-address 1.1.1.100 group-list 224.0.0.0/4

interface loopback100

description anycast-RP

ip address 1.1.1.100/32

ip router ospf 10 area 0.0.0.0

ip pim sparse-mode
http://www.ietf.org/rfc/rfc4610.txt


28/53


From an operational perspective, it may be advisable to align the PIM designated router (DR) priority with the

HSRP primary.

Note: The FTag selection for routed multicast traffic may change after routing. Post-routed traffic may beforwarded on a different FTag tree than the prerouted traffic.

PIM times are by default very conservative, and they are the cause of the slower convergence times that are

measured for multicast traffic. You can bring them down significantly by either tuning the PIM timers or by using

Bidirectional Forwarding Detection (BFD) for PIM.

Here is an example of the configuration:

bfd interval 50 min_rx 100 multiplier 3

interface Vlan101

[...]

ip pim sparse-mode

ip pim bfd-instance

ip pim dr-priority 10

ip pim hello-interval 1000

Note: PIM interval tuning should be considered with care because when deployed on too many interfaces, it

may increase the CPU utilization. BFD scales better. For information on scalability, please refer to the release

notes and/or the configuration limits.

Routed Multicast with vPC+

When using vPC+ in conjunction with the FTag priority settings that were previously described, the hashing of

routed multicast traffic is optimized, as illustrated in Figure 24.

Figure 24. Multicast routing with FabricPath

This means there is a sort of affinity behavior, based on which each vPC+ peer device will always use the

multicast destination tree rooted on itself to forward multicast traffic.

Summary Recommendations

This section summarizes the recommendation for building a Layer 3 spine in a FabricPath topology. Figure 25

illustrates these recommendations.

Configure VLANs in FabricPath mode.


29/53


Consider using manually configured switch-ids. This can help with managing and operating the topology

since you can define the numbering scheme.

Configure SVIs and HSRP as usual.

Configure vPC+ by defining a vPC peer-link and an emulated switch-id in the vPC domain. This is to allow

integration of legacy (non-FabricPath-enabled) edge layer devices and to provide Active/Active default-

gateway data-plane functionality.

Configure Layer 3 peering between routers over a FabricPath VLAN (for instance, on the same link that is

used to interconnect the spine devices, such as the vPC+ peer-link). This is required to reroute traffic

destined to the northbound Layer 3 domain if one of the aggregation devices loses physical connectivity to

the upstream devices.

Use FabricPath-enabled links for vPC peer-link purposes.

Configure dual-active exclude vlan for the list of SVIs (unless using vPC attached devices).

Distribute edge switches onto different SoCs on the F1 card for maximum MAC address scalability. Even if this is the default configuration, at the spine, you may want to verify that the ARP aging timeout is

faster than the Layer 2 forwarding table timeout.

Figure 25. Summary recommendations for FabricPath designs

Design Considerations for the Edge and Access Layer

The edge layer in FabricPath provides Classic Ethernet connectivity to servers and interfaces with FabricPath core

ports to the spine layer.

The edge layer performs unicast Layer 2 multipathing load balancing on equal-cost-path Layer 2 routes.

Flooded and unknown unicast traffic is sent on the multidestination tree associated with FTag1. Multicast traffic is

load balanced on the two multidestination trees (identified as usual with FTag1 and FTag2).

FabricPath VLANs

As with the spine layer, the access port in a FabricPath topology can be configured by assigning a FabricPath

VLAN to an access port.

You should start the configuration by making a VLAN a FabricPath type of VLAN:

DC1-5500-1# show vlan id 101


30/53


VLAN Name Status Ports

---- -------------------------------- --------- -------------------------------

101 FP_VLAN_101 active Eth1/5

VLAN Type Vlan-mode---- ----- ----------

101 enet FABRICPATH

The ports connecting the edge device to the spine or even to another edge device should be configured as

FabricPath core ports:

(config-if)# switchport mode fabricpath

The ports connecting to servers are configured as switchport mode access or trunk access. They can forward the

traffic with the regular Ethernet encapsulation.

To distribute unicast traffic via equal-cost multipath (ECMP), use the following command:

DC1-5500-2(config)# fabricpath load-balance unicast < destination, include-vlan,

layer-3, layer4, mixed, source, source-destination>

Note: By default, the unicast hashing is configured to include the source and destination IP addresses.

MAC Learning at the Access Layer

The edge layer by default performs conversational learning. Conversational learning spares MAC address table

consumption at the edge because remote MAC addresses are not learned in the Layer 2 forwarding table unless

theres an active conversation between a local node and a remote node.

If you configure an SVI on a FabricPath VLAN, conversational learning is disabled on this VLAN. It is therefore

normally recommended to not configure management SVIs on FabricPath VLANs.

Learning the HSRP MAC Address

The HSRP and GLBP MAC addresses - and in general the gateway vMAC addresses - are of particular interest

because theres never an active conversation between an end host and a vMAC. The reason is that when the

server sends traffic to the gateway, it does use the vMAC as the destination MAC address, but when the router

sends traffic to the end host, it never uses the vMAC as source MAC address; instead, it uses the burned-in

address (BIA).

If the edge switch was not learning the gateway vMAC address, all traffic destined to be routed would be

continuously flooded. The mechanism that solves this problem consists in considering HSRP, GLBP, or VRRP

hellos as part of the conversation with an end host. When a host sends traffic destined to the gateway vMAC

address, the edge switch stores it temporarily in the Layer 2 forwarding table. As soon as an HSRP hello is

received, it programs the switch-id associated with it.

If the spine is configured for vPC+, the source switch-id is the emulated switch-id. As a result, the MAC address

table of the edge switch will have the HSRP MAC associated with the emulated switch-id.

DC1-5500-2# show mac address-table vlan 101

Legend:


age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports

---------+-----------------+--------+---------+------+----+----------

* 101 0000.0c07.ac01 dynamic 0 F F 1000


31/53


Direct Connectivity between Edge Switches

One interesting point to observe is that edge switches can be connected directly to each other. This can be used

to provide a direct path between hosts. Consider Figure 26:

Figure 26. Direct connectivity between edge switches allows direct communication between servers

When DC1-5500-2 to DC1-5500-3 is connected, the two hosts can talk directly over this link without having to go

via the spine switches. This is unique to FabricPath. Spanning Tree Protocol would have blocked this link.

vPC+ at the Edge

The edge switches can be configured to perform vPC-like functions, which in the particular case of a FabricPath

infrastructure are referred to as vPC+ (as already discussed for providing Active/Active default-gateway

functionalities on the spine/aggregation devices).

Functions that are enabled by vPC+ at the edge include:

The ability to attach servers to edge switches with port-channel teaming

The ability to attach additional Classic Ethernet switches in vPC mode

The ability to attach Cisco Fabric Extenders in FEX Active/Active mode.

The key requirement for a vPC+ configuration to be operational in FabricPath is to define the emulated switch-id in

the vPC domain configuration. The other requirement is that the vPC peer-link must be a FabricPath link. All

VLANs on vPC+ links must be FabricPath VLANs.

When the peer-link goes down vPC+ operates similarly to vPC

When all the links of the peer-link fail, the primary switch will take no action, and the secondary switch willbring down all its vPC links.

In addition, vPC+ also intervenes to prevent attracting traffic to the vPC secondary switch. This function is

provided by IS-IS by ensuring that the secondary switch stops advertising reach ability information to the

emulated switch. This is essential to prevent the secondary switch from attracting traffic destined for the

emulated switch from the FabricPath cloud.


32/53


Figure 27 illustrates a design with vPC+:

Figure 27. Using vPC+ on the edge switches

In such a topology, the MAC addresses associated with a pair of edge devices running vPC+ are advertised to the

other switches that are part of the same FabricPath domain as belonging to the emulated switch-id, instead of the

individual switch-ids of the edge Cisco Nexus 5500 Switches.

Overall, with vPC+ most of the communications between devices that connect to the vPC+ peers is kept local and

doesnt involve the spines. This is true for both vPC connected devices and for orphan ports as well. Orphan portsin vPC+ can communicate over the peer-link as if this link were a regular FabricPath link (Figure 28):

Figure 28. Communication between devices connected to a vPC+ edge


33/53


In addition, even if the vPC+ peer-link were to fail, Host1 and Host4 would not lose connectivity because they

would still be able to communicate via the spines. In Figure 28, Host1 and Host4 send traffic to the switch-id of the

respective edge switches and not to the emulated switch-id. As for the north-to-south traffic flows destined to the

hosts supported by vPC links, they are sent to the vPC primary only as the vPC secondary device stops

advertising edge MAC address information associated to the emulated switch-id (Figure 29).

Figure 29. vPC+ behavior upon loss of the peer-link

Integration with FEX

You can extend FabricPath topologies by adding Cisco Fabric Extenders. Fabric extender ports are edge ports,

and if you are using vPC+, they allow creating vPCs between servers and FEX. Dual-layer vPC is also

implemented.

Summary

In summary, at the edge layer there is no major tuning necessary. The configuration is very simple:

You should configure VLANs as FabricPath VLANs.

Consider using manually configured switch-ids. This can help with managing and operating the topology

since you can define the numbering scheme.

To avoid disabling conversational learning, make sure not to have an SVI (like the management SVI) on a

FabricPath VLAN.

Consider the use of vPC+ if using port-channel teaming or FEX Active/Active or adapter-FEX/VM-FEX.

FabricPath Scalability Considerations

In terms of FabricPath scalability, there are three main elements you should consider in a given FabricPath

domain:

Number of switches

Number of VLANs

Number of MAC addresses


34/53


The maximum number of switches and VLANs supported (that is, tested by QA) in the same FabricPath domain is

continuously increasing at every Cisco NX-OS Software release. At the time of writing of this document, these are

the expected supported values:

Cisco Nexus 7000 Series (NX-OS Release 6.0): 128 switch-ids and between 2000 and 4000 VLANs

Cisco Nexus 5500 Switches (NX-OS Release 5.1(3)N1(1)): 128 switch-ids and between 2000 and 4000

VLANs

Note: For updated information on the maximum number of switch-ids supported within the same FabricPath

domain, please refer to the release notes and the configuration limits documentation for each specific software

release.

One of the benefits of FabricPath is the scalability both in terms of MAC addresses and in terms of

throughput/oversubscription ratios.

In terms of MAC address scalability, we should differentiate between edge and spine devices.

The edge switches learn only MAC addresses for which there are active conversations established (conversational

learning). The focus of this document is on using the Cisco Nexus 5500 Switches at the edge of the FabricPath

network. The current maximum value of MAC addresses supported on these devices is 24k.

For the spine devices, it is important to distinguish between east-to-west (switched) and north-to-south (routed)

traffic flows:

East-west traffic: These flows are switched by the spine devices based on the outer MAC header

information (FabricPath switch-id values) (see Figure 30). For the spine devices to perform this switching,

there is technically no need for them to learn the MAC addresses of the endpoints connected to the

FabricPath edge switches.

Figure 30. FabricPath spines switch traffic based on the Destination Switch-id (DSID)

North-south traffic: Traffic coming from the Layer 3 network domain and directed to an endpoint connected

to a FabricPath edge switch needs to be first handled by the M1 card (performing routing functionalities)

and eventually sent out to the FabricPath core through an F1 interface.


35/53


Figure 31. Traffic routing by M1 card and subsequent FabricPath encapsulation by F1 card

This essentially means that the MAC addresses scalability is mainly dictated by the CAM table available on the M1

card, which is capable of containing 128k addresses. The M1 card has all the information needed (MAC address,

switch-id of the edge device) to internally forward the traffic to the F1 interface that is properly connected to the

destination FabricPath edge switch.

However, as previously discussed in the section MAC Scalability Considerations, the endpoint MAC addresses

are also learned on the F1 card instead. This is because in the current implementation, the F1 card needs to

associate to each remote MAC address a specific value (named local ID [LID]) and insert it into the FabricPath

header before sending the frame toward the edge switch (Figure 32).

Figure 32. F1 learning MAC addresses for routed traffic purposes


36/53


The end result is that each SoC is currently learning all the remote MAC addresses for endpoints locally attached

to each FabricPath edge switch. This happens in an independent fashion, whether or not traffic belongs to the

same VLAN, which essentially brings the maximum MAC scalability values to 256k per F1 card (since it supports

16 SoCs, each capable of holding up to 16k MAC addresses). By properly assigning edge switches to spine ports,

it is theoretically possible to use the aggregated capacity of the spine F1 cards switch-on-chips (SoC) component

for up to 128k MAC addresses for routed traffic.

However, at the time of this writing, the maximum number of MAC addresses that can be stored for routing

purposes without incurring into flooding is 16k. The reason is described by the following examples. Lets assume

that one of the uplinks connecting a FabricPath edge switches to the spine fails, as shown in Figure 33:

Figure 33. Failure scenarios for routed traffic

Traffic that originated from the upstream Layer 3 core devices can still be sent to the left aggregation switch, but it

needs to use the transit connection (vPC+ peer-link) between the spines to be directed toward the FabricPath

edge switch. As a consequence, the MAC addresses of the endpoints connected to that specific FabricPath edge

switch will need to be installed on the MAC tables of the SoCs associated to these peer-link interfaces. This would

basically limit to 16k the total amount of MAC addresses that can be learned on the FabricPath edge switch, in

order to avoid flooding the routed traffic whose MACs have not been learned.

Obviously, the situation would become worse if multiple uplinks belonging to separate FabricPath switches fail at

the same time, leading to an extremely unlikely case where the maximum MAC scalability supported across the

entire FabricPath domain is reduced to 16k MAC addresses (Figure 34).


37/53


Figure 34. Impact of multiple links failure for routed traffic

If this unlikely scenario is a concern, it is possible to deploy port-channels (instead than single links) to

interconnect each FabricPath edge devices to the spine switches. In that way, a single link failure event would not

force the traffic on the peer-link.

It is also worth noting that a specific functionality, named mac-proxy, is on the roadmap for a future NX-OS SW

release that will help eliminating the issue described above, by removing the need for the F1 SoCs to learn

endpoint MAC address information. Once this functionality is available (currently planned for 1HCY12), it would

truly be possible to consider the MAC address scalability off a FabricPath domain as only dictated by the M1 MAC

table space (minimum 128k).

For Layer 2 traffic (that is, traffic that is not directed to the default gateway), the MAC address scalability is

dependent only on the MAC scalability of the edge devices. This is even more relevant for VLANs that do not

require routing, such as the isolated networks constituted by the virtual server building blocks of typical virtual

data centers (vDCs), like the ones depicted in Figure 35:


38/53


Figure 35. vDCs use many isolated VLANs

In such cases, the MAC address scalability for isolated VLANs is entirely decoupled from the F1 learning

component.

In all cases, east-to-west Layer 2 traffic doesnt use the MAC address table of the spine devices.

Finally, the ability to do unicast multipathing on up to 16 different equal-cost paths makes it possible to place up to

16 spines in the core, thus lowering the oversubscription and providing an aggregated bandwidth that equals the

sum of the bandwidth provided at the spine layer by each spine.

FabricPath Convergence Times

Cisco FabricPath improves significantly on the convergence times for any given failure. For most unicast failures,

the traffic loss is within a few hundred milliseconds. For routed traffic, the worst convergence time is the time that it

takes for routers to do an SPF calculation. Considering the various throttling mechanisms, this amounts to ~5

seconds, which can be further lowered by changing the default OSPF timers on the adjacent routing devices. The

configuration required to achieve this is very simple:

router ospf 10

timers throttle spf 10 100 5000

timers throttle lsa all 10 100 5000

timers lsa arrival 80

For Layer 2, multicast convergence times are also within a few hundred milliseconds, while for routed multicast,

PIM must be tuned via BFD and/or more aggressive timers than the defaults. Routed multicast with proper tuningcan converge within a couple of seconds.

Notice that these slower convergence times are not due to FabricPath but to default timers in routing protocols.

FabricPath has failure convergence times of a few hundred milliseconds.

For better failover times, consider the following:

Use vPC+ at the spine in order for HSRP to be advertised with the emulated switch-id.

Consider tuning the default OSPF SPF timers to avoid throttling SPF recalculations too much.

Consider tuning the default PIM timers and or using BFD.


39/53


Sample Configurations

This section illustrates how to configure the Cisco Nexus 7000 Series Switches and Cisco Nexus 5500 Switches

for a spine-edge design with the spine also providing the Layer 3 gateway function. The configuration samples

refer to the network topology shown Figure 36.

Figure 36. Reference Topology for the configurations provided in this document

Note that in regard to topology shown in Figure 36 and following configuration samples, the F1 line card is inserted

in slot 8, and the M1 line card is inserted in slot 10.

Cisco Nexus 7000 Series Spine 1 (DC1-Agg1)

install feature-set fabricpath

hostname DC1-Agg1

vdc DC1-Agg1 id 1

allow feature-set fabricpath

[]

feature-set fabricpath

[]feature ospf

feature pim

feature interface-vlan

feature hsrp

feature lacp

feature glbp

feature vpc

feature pong

feature bfd


40/53



vlan 101mode fabricpath

name FP_VLAN_101

vlan 102

mode fabricpath

name FP_VLAN_102

vlan 103

mode fabricpath

name FP_VLAN_103

vlan 333

mode fabricpath

name Aggr_IGP_Peering

vpc domain 1

role priority 110


delay restore 3600

dual-active exclude interface-vlan 101-104


interface Vlan101

no shutdown

ip address 10.20.0.2/16

ip ospf passive-interface


ip pim sparse-mode

ip pim bfd-instance

ip pim dr-priority 10


hsrp 1

preempt delay reload 300

priority 110

ip 10.20.0.1

interface Vlan102

no shutdown

ip address 10.30.0.2/16



ip pim sparse-mode

ip pim bfd-instance


hsrp 1



41/53


priority 110

ip 10.30.0.1

interface Vlan103no shutdown

ip address 10.40.0.2/16



ip pim sparse-mode

hsrp 1


priority 110

ip 10.40.0.1

interface Vlan333

no shutdown

description Aggr_IGP_Peering

no ip redirects

ip address 11.1.10.1/24


ip pim sparse-mode


description FP Link to DC1-5500-1

switchport



description vPC+ Peer-Link

switchport


vpc peer-link

interface Ethernet8/1

description vPC+ Peer-Link Member 1


channel-group 10 mode active

no shutdown


description vPC+ Peer-Link Member 2



no shutdown


description Port-channel to DC1-5500-1 Member 1


42/53




no shutdown


description Port-channel to DC1-5500-1 Member 2



no shutdown




no shutdown




no shutdown


description L3 Link to core

ip address 12.1.5.1/24


ip pim sparse-mode

no shutdown

interface mgmt0

ip address 10.60.17.140/24

interface loopback0

description local-loopback

ip address 1.1.1.1/32


interface loopback100

description anycast-RP

ip address 1.1.1.100/32


ip pim sparse-mode

router ospf 10

router-id 1.1.1.1

timers throttle spf 10 100 5000

timers throttle lsa all 10 100 5000

timers lsa arrival 80


43/53


fabricpath domain default

root-priority 66

topology 1

ip pim rp-address 3.3.3.3 group-list 224.0.0.0/4

ip pim ssm range 232.0.0.0/8


Cisco Nexus 7000 Series Spine 2 (DC1-Agg2)

install feature-set fabricpath

hostname DC1-Agg2

vdc DC1-Agg2 id 1

allow feature-set fabricpath

[]

feature-set fabricpath

feature vrrp

cfs eth distribute

feature ospf

feature pim

feature msdp

feature interface-vlan

feature hsrp

feature lacp

feature glbp

feature vpc

feature pong

feature bfd


vrf context management

ip route 0.0.0.0/0 10.60.17.254

vlan 101

mode fabricpath

name FP_VLAN_101

vlan 102mode fabricpath

name FP_VLAN_102

vlan 103

mode fabricpath

name FP_VLAN_103

vlan 333

mode fabricpath

name Aggr_IGP_Peering


44/53


vpc domain 1


delay restore 3600

dual-active exclude interface-vlan 101-104fabricpath switch-id 1000

interface Vlan101

no shutdown

no ip redirects

ip address 10.20.0.3/16



ip pim sparse-mode

ip pim bfd-instance


hsrp 1

ip 10.20.0.1

interface Vlan102

no shutdown

no ip redirects

ip address 10.30.0.3/16



ip pim sparse-mode

ip pim bfd-instance


hsrp 1

ip 10.30.0.1

interface Vlan103

no shutdown

no ip redirec

cisco fabric path-690079

Documents