vpc best practices and design on nx-osd2zmdbbm9feqrf.cloudfront.net/2017/usa/pdf/brkdcn-2378.pdf ·...

vPC Best Practices and Design on NX-OS

Nemanja Kamenica ([email protected])Engineer, Technical Marketing

BRKDCN-2378

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to chat with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

Cisco Spark spaces will be available until July 3, 2017.

cs.co/ciscolivebot#BRKDCN-2378


Session Goal

• To provide a thorough understanding of the Virtual Port Channel, design and best practices of vPC configuration.

• This session will examine best practice of vPC in environments with: Nexus 2000, firewalls, load-balancers, and vPC with Dynamic routing.

• Examine best practice of vPC with ACI, VxLAN, FCoE, FabricPath networks.

• This session will not examine in depth Nexus switch architecture, FCoE, Fabric Path, VxLAN, ACI, and Nexus 2000…

4

BRKDCN-2378 4

• Introduction to vPC

• Feature Overview

• Configuration Best Practices

• Design Best Practices

• vPC Operations and Upgrade

• vPC with Fabric Technologies

• Key Takeaways

Agenda


vPC Feature OverviewvPC Concept & Benefits

S1 S2

S3

• No Blocked Ports

• More Usable Bandwidth

STP

S2S1

S3

vPC Logical Topology

BRKDCN-2378 6

S3

S1 S2

vPC Physical Topology


Virtual Port Channel - vPC

• Eliminates Spanning Tree blocked ports by providing loop-free topology

• Better bandwidth utilization

• Provides device level redundancy with faster convergence

• MC-LAG on Cisco Nexus Devices

• Deployed by almost 95% of Nexus customers

Benefits

Unified Fabric

BRKDCN-2378 7


Data Center Technology Evolution

FabricPath with vPC+

2010

2009

VPC

2008

STP2013-2014

MPLS, OTV,

LISP

VXLAN

MPLS, OTV,

LISP

2014-2015

ACI

2010

FEX with vPC

Used to redundantly connect network entities at the edge of the Fabric

BRKDCN-2378 8







• Key Takeaways

Agenda


Feature OverviewvPC Terminology

Orphan

Device

Layer 3 Cloud

vPC Member Port

vPC

Orphan Port

vPCPeer

CFS

vPC DomainPeer-Link

vPC PeerKeepalive Link

S3

S1 S2

L3

P S

BRKDCN-2378 10


vPC Peer-keepalive link

• L3 link, connects vPC peers

• Carries periodic hart beet between vPC peers

• Uses UDP port 3200

• Sends keep-alive heart beets every secondvPC Domain

S3

S1 S2

L3

BRKDCN-2378 11

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 12BRKDCN-2378

vPC Peer-link

• vPC peer link is a port channel that carries:

• vPC VLANs

• CFS messages

• Flooded traffic from the other peer device

• STP BPDUs, HSRP hello messages and IGMP updates

• vPC imposes the rule that peer-link should never be blocking

vPC Domain

S3

S1 S2

L3


vPC

• Consists of port-channel member of vPC

• L2 port channel

• Ports in vPC can be in access or trunk mode

• VLANs allowed on vPC need to be allowed on peer-link

• LACP and Static port channel configuration

vPC Domain

S3

S1 S2

L3

BRKDCN-2378 13


Cisco Fabric Services Protocol

• Synchronization and consistency checking mechanism

• Runs on VPC Peer-link

• CFS protocols mechanism:

• Validation and comparison for consistency check

• Synchronization of MAC addresses for member ports

• Status of member ports advertisement

• STP management

• Synchronization of HSRP and IGMP snooping

• Enabled by default

vPC Domain

S3

S1 S2

L3

CFS

BRKDCN-2378 14


vPC Consistency check

• System configuration must be in sync

• Type 1 Consistency Check

• Graceful Consistency check – suspends:• Per-interface inconsistent parameters – vPC member ports on secondary peer set to down state

• Globally inconsistent parameters – misconfigured member ports on secondary peer suspended

• Parameters: STP mode, STP VLAN state, STP global settings, LACP mode, MTU…

• Type 2 Consistency Check

• Forwards traffic in case of inconsistency

• Possible undesirable traffic forwarding behavior

• Parameters: VLAN interface (SVI), ACL, QOS, IGMP snooping, HSRP…

BRKDCN-2378 15







• Key Takeaways

Agenda


vPC Configuration Best PracticesBuilding a vPC domain – Configuration Steps

CFS

1. Define domains

2. Establish Peer Keepalive connectivity

3. Create a Peer link

4. Create vPCs

5. Make Sure Configurations are Consistent

(Order does Matter!)

S1 S2

S3

L3

BRKDCN-2378 17


vPC Configuration Best PracticesvPC Domain-ID

• The vPC peer devices use the vPC domain ID to

automatically assign a unique vPC system MAC

address

• You MUST use unique Domain id’s for all vPC

pairs defined in a contiguous layer 2 domain

! Configure the vPC Domain ID – It should be unique within the layer 2

domain

NX-1(config)# vpc domain 20

! Check the vPC system MAC address

NX-1# show vpc role

<snip>

vPC system-mac : 00:23:04:ee:be:14

vPC Domain 10

vPC Domain 20

S1 S2

S3

S5

S4

BRKDCN-2378 18


vPC Configuration Best PracticesvPC Peer-Keepalive link

Preference Nexus 9500 / 7X00

series

Nexus 9X00 / 6000 /

5X00 / 3X00 series

1 Dedicated link(s)

(1GE/10GE Links)

mgmt0 interface

2 mgmt0 interface Dedicated link(s)

(1GE/10GE Links)

3 L3 infrastructure L3 infrastructure

Recommendations

(in order of

preference):

BRKDCN-2378 19


vPC Configuration Best PracticesvPC Peer-Keepalive link – Dual Supervisors

Standby Management Interface

Active Management Interface

vPC1 vPC2

vPC_PL

Management Network

Management Switch

vPC_PKLvPC_PKL• When using dual supervisors and mgmt0 interfaces

to carry the vPC peer-keepalive, DO NOT connect

them back to back between the two switches

• Only one management port will be active a given point

in time and a supervisor switchover may break keep-

alive connectivity

• Use the management interface when you have an out-

of-band management network (management switch in

between)

BRKDCN-2378 20


vPC Configuration Best Practices

vPC Peer-link should be a point-to-point connection

Peer-Link member ports can be 10/40/100GE interfaces

Peer-Link bandwidth should be designed as per the vPC

vPC imposes the rule that peer-link should never be blocking

vPC Peer-Link

S1 S2

S3

S1

S3

S2


Design Best Practices

• Always use identical switches and line cards on either sides of the peer link and vPC member ports !

Mixed Hardware across vPC Peers : Line Cards

Examples:

22

vPC Peer-link

S2

N9500

vPC Primary vPC Secondary

vPC

97160EX

9732EXS1

N9500

vPC Peer-link

S2


vPC

S19732EX

97160EX

9732EX 9536PQ

9732EX 9536PQ


Design Best PracticesMixed Hardware across vPC Peers : Line Card

X Y vPC

N9K-X9636PQ N9K-X9432PQ

N9K-X9564PX N9K-X9464PX

N9K-X9564TX N9K-X9464TX

N9K-X9536PQ N9K-X9736PQ

vPC Peer-link

S2

N9500


Y

vPC

X

X YS1

N9500

BRKDCN-2378 23



• Always use identical switches and line cards on either sides of the peer link and vPCmember ports !

• VDC type should match on both peer device

Mixed Hardware across vPC Peers : Line Cards

Examples:

24

vPC Peer-link

S2

N7700


F3

vPC

F3

F2E F2ES1

N7700M2M1

vPC Peer-link

S2


F3

vPC

F3

S1


Design Best PracticesMixed Hardware across vPC Peers : Chassis & Supervisors

• N7000 and N7700 in same vPC Domain - Supported

• vPC peers can have mixed SUP version*

• N9K: SUP-A, SUP-B

• N7K: SUP1, SUP2, SUP2E

• N5500 and N5600 in same vPC Domain – Not Supported

*Recommended only for short period

vPC Peer-link

S2

N7700


vPC

S1

N7000

vPC Peer-linkS2

N5600


vPC

S1

N5500

BRKDCN-2378 25



• Data plane loop prevention

• vPC peer forwards traffic locally when possible

• Traffic coming from vPC member port, crossing peer-link is NOT allowed to egress any vPCmember port

• Exception of the rule, when member port goes down

vPC Loop Avoidance

S3

vPC 1

S2

S1


vPC Configuration Best PracticesSpanning Tree (STP)

• All switches in Layer 2 domain should run either Rapid-PVST+ or MST

• Do not disable spanning-tree protocol for any VLAN

• Always define the vPC domain as STP root for all VLANs in that domain

STP is running to manage

loops outside of vPC domain,

or before initial vPC

configuration !

S1 S2

S4S3

S5

BRKDCN-2378 27


vPC Configuration Best PracticesvPC Peer-switch

Without Peer-switch:

• STP for vPCs controlled by vPC primary

• vPC primary send BPDU’s on STP designated ports

• vPC secondary device proxies BPDU’s to primary

With Peer-switch:

• Peer-Switch makes the vPC peer devices to appear as

single STP root

• BPDUs processed by the logical STP root formed by the

2 vPC peer devices

Nexus(config-vpc-domain)# peer-switch

BPDUs

P S

P S

BRKDCN-2378 28


Hybrid topology (vPC and non-vPC)

• Hybrid topology where vPC and non-vPC devices coexist in a vPC domain

• Need additional configuration parameters : spanning-tree pseudo-information

• STP pseudo configuration takes precedence over global STP configuration

• Not supported on Nexus 9200 and Nexus 9x00-EX

S1 S2

S3 S4


vPC1

Bridge Priority

VLAN 1 4K

VLAN 2 8K

Bridge Priority

VLAN 1 8K

VLAN 2 4K

STP Root

VLAN 1STP Root

VLAN 2

STP Root

VLAN 1

VLAN 2

VLAN 1

(blocked)

VLAN 2

(blocked)

peer-switch


vPC Configuration Best PracticesvPC Peer-Gateway

• Allows a vPC switch to act as the active

gateway for packets addressed to the peer

router MAC

• Keeps forwarding of traffic local to the vPC node

and avoids use of the peer-link

• Allows Interoperability with features of some NAS

or load-balancer devices

S1 S2

S4S3

Nexus(config-vpc-domain)# peer-gateway

BRKDCN-2378 30


vPC Configuration Best PracticesPVLAN on vPC

• PVLAN configuration across both vPC switches should be identical

• PVLAN configuration not supported on Peer-Link

• Type-1 Consistency Check

• Port mode is a type-1 check

• vPC member port brought down if PVLAN port mode differs between vPC peers

• Type-2 Consistency Check

• PVLAN will bring down mismatched couples

S1 S2


P P

PVLAN-

PROMISC

(3500, 3501)

PVLAN-

PROMISC

(3500, 3501)

C

Community

VLAN

BRKDCN-2378 31


S1 S2


S3

PVLAN

Isolated

Trunk

vPC Configuration Best PracticesPVLAN vPC Type 1 Consistency Check

PVLAN

Promiscuous

Trunk

S1 S2


S3

Type 1 Consistency

Failure

S1 S2


P P

S3

I I

TI

BRKDCN-2378 32


S1 S2


S3

vPC Configuration Best PracticesPVLAN vPC Type 2 Consistency Check

S1 S2


S3

S1 S2


P P

S3

I I

I

PVLAN-

PROMISC

(10, 201)

Secondary

Trunk (2,31)

(3,30), (4,100)

PVLAN-

PROMISC

(10, 201)

Secondary

Trunk (2,31)

(3,30), (4,100)

I

Secondary

Trunk (3,31)

(2,30), (4,100)

Secondary

Trunk (2,31)

(3,30), (4,100)

Type 2 Consistency

Failure

BRKDCN-2378 33

Failure Scenarios


vPC Failure Scenario vPC member port down

Secondary vPC peerS

P Primary vPC peer

On of the vPC member ports fails (optics failure or cable failure)

• vPC primary and secondary peer remain primary and secondary, no change in roles

• Result in change in path, and traffic with destination to other peer, will cross peer-link to get to destination

• This is not desirable behavior, and peer-link can be oversubscribed

S2

P

SW3 SW4

vPC1 vPC2

vPC_PLink

vPC Peer-keepalive

S1

S


vPC Failure Scenario vPC peer-keepalive link down

Keepalive Heartbeat

Secondary vPC peerS

P Primary vPC peer

vPC peer-keepalive link failure (link loss):

• vPC peer-link up

• No role change

• Status of other vPC peer known

• Both peers forwarding

• No down time in the network

SW3 SW4

vPC1 vPC2

vPC_PLink

vPC Peer-keepaliveP S

S2S1


vPC Failure Scenario vPC Peer-Link down

Keepalive Heartbeat

Secondary vPC peerS

P Primary vPC peer

vPC peer-link failure (link loss):

• vPC peer-keepalive up

• Status of other vPC peer known

• Both peers are active

• Secondary vPC peer disables all vPC’s

• Traffic flows over vPC primary

• Traffic from orphan devices connected to secondary peer will be black-holed

SW3 SW4

vPC1 vPC2

vPC Peer-Link

vPC Peer-keepaliveP S

Suspend secondary

vPC Member Ports

S2S1

BRKDCN-2378 37


vPC Failure Scenario – Dual Active

1. vPC peer-keepalive DOWN

2. vPC peer-link DOWN

3. DUAL-ACTIVE or SPLIT BRAIN scenario

• vPC primary peer remains primary and secondary peer becomes operational primary role

• Result in traffic loss / uncertain traffic behavior

• When links are restored, the operational primary (former secondary) keeps the primary role & former primary becomes operational secondary

vPC Peer-Keepalive down followed by vPC Peer-Link down

Secondary vPC peerS

P Primary vPC peer

P

SW3 SW4

vPC1 vPC2

vPC_PLink

vPC Peer-keepalive

Traffic Loss / Uncertain Traffic

Behavior

S1 S2

P

Additional Features



• Single attached devices to vPCdomain, will black-hole traffic if peer-link fails

• With Orphan Port Suspend feature, will suspend orphan ports on vPC secondary peer

• When peer-link is restored, vPCsecondary restores orphan ports

vPC Orphan ports suspend

Nexus(config-if)# vpc orphan-ports suspend

S2S1

S3

P S

Active or Standby

Active or Standby

BRKDCN-2378 40

Secondary vPC peerS

P Primary vPC peer



• When peer device goes down or peer link goes down, SVIs are suspended

• After restore of the peer device, or peer link, ARP table is empty -traffic blac-kholed

• Before bringing up SVI, peer devices synchronize ARP table over CFS

• Reduces convergence time

vPC ARP sync

L2

L3

ARP TABLE

IP1 MAC1 VLAN 100

IP2 MAC2 VLAN 200

ARP TABLE

IP1 MAC1 VLAN 100

IP2 MAC2 VLAN 200

SVI

200

SVI

100SVI

200

SVI

100

ARP TABLE

IP1 - -

IP2 - -

Nexus(config-vpc-domain)# ip arp synchronize

CFS

BRKDCN-2378 41



• After vPC peer reload, traffic might be black-holed, before L3 connectivity is reestablished

• vPC link bring up can be delayed to allow L3 routing protocol convergence

• Default time 30 seconds

vPC Delay Restore

L2

L3

Nexus(config-vpc-domain)# delay restore <1-3600 sec>

OSPF

BRKDCN-2378 42


vPC Configuration Best PracticesvPC auto-recovery

1. vPC peer-link down : S2 - secondary shuts all its vPC member ports

2. S1 down : vPC peer-keepalive link down : S2 receives no keepalives

3. After 3 keepalive timeouts, S2 changes role and brings up its vPC

S2S1

S3

P S

S2S1

S3

P S

S2S1

S3

Secondary vPCS

P Primary vPC

Nexus(config-vpc-domain)# auto-recovery

BRKDCN-2378 43

Operational

PrimaryP


S2S1

S3

• Until peer adjacency is reestablished between vPC devices, vPC member ports are suspended

• vPC auto-recovery reload delay allows “alive” vPC peer to assume primary role after delay time is expired

• Delay timer can be tuned

vPC Configuration Best PracticesvPC auto-recovery reload delay

Nexus(config-vpc-domain)# auto-recovery reload-delay <240-3600 seconds>

BRKDCN-2378 44


vPC Configuration Best PracticesvPC auto-recovery

Auto-recovery addresses two cases of single switch behavior

• Peer-link fails and after a while primary switch (or keepalive link) fails

• Both VPC peers are reloaded and only one comes back up

How it works

• If Peer-link is down on secondary switch, 3 consecutive missing peer-keepalives

will trigger auto-recovery

• After reload (role is ‘none established’) auto-recovery timer (240 sec) expires

while peer-link and peer-keepalive still down, autorecovery kicks in

• Switch assumes primary role

• VPCs are brought up bypassing consistency checks


S2S1

S3

P S

3. S2 takes over as operational Primary and S1 is isolated from the vPC domain

vPC Configuration Best PracticesvPC Self-Isolation

1. Error Triggered : All Line cards Fail or All Vlans’s down on peer-link

2. S1 sends “self-isolation” message through the peer-keepalive

S2S1

S3

P S

Secondary vPCS

P Primary vPC

Error

Triggered

Self- Isolate

Operational

PrimaryPISOLATED

S2S1

S3

BRKDCN-2378 46


vPC Configuration Best PracticesExample Configuration and Verification on Nexus 7x00

vPC domain 100

peer-keepalive destination

10.126.216.44

peer-gateway

self-isolation

vPC domain 100


10.126.216.41

peer-gateway

self-isolation

Switch# show vPC brief

<snip>

vPC domain id : 100

<snip>

vPC role : primary

<snip>

Self-isolation : Enabled


<snip>

vPC domain id : 100

<snip>

vPC role : secondary

<snip>

Self-isolation : Enabled

2015 Sep 29 22:33:03 S1 %$ VDC-1 %$

%vPC-2-ENTER_SELF_ISOLATION: Local

switch goes into self isolation

state due to all linecards failure.

Please resume failed linecards and

do shut/no shut on peer-link to exit

self-isolation state

2015 Sep 30 10:33:14 S2 %$ VDC-1 %$

%vPC-2-ENTER_SELF_ISOLATION: Remote

switch goes into self isolation

state due to all linecards failure.

Please resume failed linecards and

do shut/no shut on peer-link to exit

self-isolation state

BRKDCN-2378 47


vPC Configuration Best PracticesvPC Self-Isolation

• vPC self-isolation is turned OFF by default

• No Impact on vPC operation if sellf-isolation enabled

• Functional only when enabled on both vPC peers.

• Not part of vPC type-1 and type-2 consistency checks

BRKDCN-2378 48


vPC Configuration Best PracticesWhy Object-Tracking ?

Primary Secondary

• Modules hosting peer-link and uplink fail on

the vPC primary

• Peer-Link is down and vPC Secondary shut all its vPC

• Auto-Recovery does not kick in as peer-keepalive link is active

• Traffic is black holed

S1 S2

S3

S5S4

L2

L3

BRKDCT-2378 49


vPC Configuration Best PracticesObject-tracking

S1 S2

• Object Tracking triggered when the track object goes down

• vPC object tracking, tracks both peer-link and uplinks in a list of Boolean OR

• Traffic forwarded over the remaining vPC peer! Track the vpc peer link

track 1 interface port-channel11 line-protocol

! Track the uplinks

track 2 interface Ethernet1/1 line-protocol

track 3 interface Ethernet1/2 line-protocol

! Combine all tracked objects into one.

! “OR” means if ALL objects are down, this object will go down

track 10 list boolean OR

object 1

object 2

object 3

! If object 10 goes down on the primary vPC peer,

! system will switch over to other vPC peer and disable all local vPCs

vpc domain 1

track 10

• Suspends the vPCs on the impaired device

S4 S5

S3

L2

L3

BRKDCN-2378 50


vPC Configuration Best PracticesSpanning Tree Bridge Assurance

Root

Blocked

BPDUs

Network

Network Network

Network

BPDUs

EdgeEdge

Network

Network

BPDUs

Malfunctioning

switch

Stopped receiving BPDUS!

BA Inconsistent

BA Inconsistent

%STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port Ethernet2/48

VLAN0700

switch# show spanning vl 700 | in -i bkn

Eth2/48 Altn BKN*4 128.304 Network P2p *BA_Inc

Stopped receiving BPDUS!

BRKDCN-2378 51


Spanning Tree Bridge Assurance

• Turns STP into a bidirectional protocol

• Ensures spanning tree fails “closed” rather than “open”

• All ports with “network” port type send BPDUs regardless of state

• If network port stops receiving BPDUs, port is placed in BA-Inconsistent state (blocked)

Almost like a routing protocol…

%STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port Ethernet2/48 VLAN0700.switch# sh spanning vl 700 | in -i bknEth2/48 Desg BKN*4 128.304 Network P2p *BA_Inc

BRKDCN-2378 52


vPC Configuration Best PracticesvPC & Bridge Assurance (BA)

• STP Bridge Assurance is enabled by default on vPC Peer-Link

• DON’T disable Bridge Assurance on vPC Peer-link

• NO Bridge Assurance on vPC member ports (even with peer-switch)

BRKDCN-2378 53



• Light-weight Layer 2 failure detection protocol

• Designed for detecting:

• One-way connections due to physical or soft failure

• Miss-wiring detection (loopback or triangle)

• Cisco proprietary, but listed in informational RFC 5171

• Runs on any single Ethernet link, even inside bundle

• Centralized implementation in switching platforms

• Message interval: 7 - 90 sec (default: 15 seconds)

• Detection: 2.5 x interval + timeout value (4 sec) ~ 41 sec

Unidirectional Link Detection (UDLD)

TxRx

TxRx


vPC Configuration Best PracticesUDLD with vPC

• UDLD NOT recommended on vPC peer-link

• UDLD NOT recommended on vPC member ports if LACP is used

• UDLD only in normal mode on vPC member ports if required

BRKDCN-2378 55







• Key Takeaways

Agenda


Design Best PracticesFHRP with vPC

• FHRP in Active/Active mode with vPC

• Primary peer should be the HSRP/VRRP active device

• Best Practice : Use default FHRP timers

FHRP

“Standby”:

Active for

shared L3 MAC

FHRP

“Active”:

Active for

shared L3 MAC

S1 S2

S3 S4

L2

L3

BRKDCN-2378 57


Design Best PracticesASA Cluster

ASA Cluster Mode

• Use unique vPC for ASA Cluster Data Links to vPC domain

• Use vPC per ASA device for Cluster Control Link (CCL) to vPC domain

• Leverage peer-switch configuration

Cluster

Control Link

Cluster

Data Link

BRKDCN-2378 58


Nexus 2000 (FEX) Straight-Through Deployment with vPC

• Port-channel connectivity from the server

• Two Nexus switches bundled into a vPCpair

• Suited for servers with Dual NIC and capable of running Port-Channel

BRKDCN-2378 59

FEX

101

Fabric Links

FEX

102

vPC

HIF

S1 S2

HIF


Nexus 2000 (FEX) Active-Active Deployment with vPC

• Fabric Extender connected to two Nexus

5X00 / 6000 / 7x00 / 9300

• Suited for servers with Single NIC or Dual NIC

not having port-channel capability

• Scale implications of less FEX per system and

less vPC

Note :

• This design is currently not supported on

Nexus 9500

BRKDCN-2378 60

Fabric Links

HIF

Fex 101Fex 102

S1 S2

HIF


Nexus 2000 (FEX) Active-Active Scale & Limitations (N7X00)

• N7X00 can support up to 64 FEXs

• N7X00 supports only 15 Active-Active FEX in 7.2(0)D1(1)

• N7X00 supports only 32 Active-Active FEX in 7.3(0)D1(1)

• Straight-Through FEX and Active-Active FEX cannot exist on the same ASIC instance

• Layer 3 HIF ports are not supported with Active-Active FEX

• Active-Active FEX is not supported with vPC+

BRKDCN-2378 61


Nexus 2000 (FEX) - Enhanced vPC

• Port-channel connectivity to dual-homed

FEXs

• From the server perspective a single access

switch with port-channel support – each line

card supported by redundant supervisors

• Ideal design for a combination of single

NIC and Dual NIC servers with port-

channel capability

Note :

This design is currently not supported on N7000 / N7700 and

N9X00

Fabric Links

Fex 100 Fex 101

S1 S2

HIFHIF

BRKDCN-2378 62


Physical Port vPC – Nexus 7x00

vPC domain

FEX101 FEX102

e101/1/1 e102/1/1Po1

Port-channel vPC

VPC1 VPC1Po1

vPC domain

FEX101 FEX102

e101/1/1 e102/1/1

Physical port vPC

VPC1 VPC1

interface e101/1/1

switchport

vpc 1

lacp mode active

• vPC configuration on a physical Layer 2 port as opposed to a port-channel

• Front panel ports and FEX ports connected to F2/F2e/F3 only

• Improves scaling as separate port-channel interface not created for single-link vPC member port

• Key benefit: more than 1000 host facing vPCs with FEX

BRKDCN-2378 63

Data Center Interconnect -vPC


Data Center Interconnect - DCI

• DCI provides connection of distant date centers

• Extend VLANs between data centers

• Technologies for DCI:

• Overlay Transportation Virtualization – OTV (Multiple DC Interconnect)

• Virtual Port Channel – vPC (Two DC Interconnect)

• vPC DCI:

• STP Isolation Between DC

• Easy to Configure

• Resilient Solution


vPC - Data Center Interconnect (DCI)

Long Distance

Dark Fiber

DC 1 DC 2C

OR

EA

GG

R

AC

CE

SS

Server Cluster

CO

RE

AG

GR

AC

CE

SS

Server Cluster

vPC domain 10 vPC domain 20

vPC domain 21vPC domain 11

E

- -

-

- -

E

E

E

E

F

F

F

F-

-

- -

-

--

B

N N

N

NN

N

R

R

-

RRRR

RR

NN

B

-

E

Rootguard

B

F

N

E

BPDUguard

BPDUfilter

Network port

Edge or portfast

- Normal port type

R

802.1AE (Optional)

BRKDCN-2378 66


Design Best PracticesvPC - Data Center Interconnect (DCI)

• vPC Domain id for vPC layers should be UNIQUE

• BPDU Filter on the edge devices to avoid BPDU propagation

• STP Edge Mode to provide fast Failover times

• No Loop must exist outside the vPC domain

• No L3 peering between Nexus devices (i.e. pure layer 2)

BRKDCN-2378 67

Layer 3 over vPC


Dynamic Routing over vPCProblem?

1) Packet arrives at R1

2) R1 does lookup in routing table and sees 2 equal paths going north (to S1 & S2)

3) Assume it chooses S1 (ECMP decision)

4) R1 now has rewrite information to which router it needs to go (router MAC S1 or S2)

5) L2 lookup happens and outgoing interface is port-channel 1

6) Hashing determines which port-channel member is chosen (say to S2)

7) Packet is sent to S2

8) S2 sees that it needs to send it over the peer-link to S1 based on MAC address

R1

S2

S3

Po1

Po2

S1


Dynamic Routing over vPCProblem?

R1

S2

S3

Po1

Po2

S1

9) S1 performs lookup and sees that it needs to send to S3

10) S1 performs check if the frame came over peer link & is going out on a vPC.

11) Frame will ONLY be forwarded if:

• Outgoing interface is NOT a vPC or

• Outgoing vPC doesn’t have active interface on other vPC peer (in the example S2)

Note:

• Use of Peer-Gateway allows routing/forwarding traffic for the peer-router MAC locally, but does NOT Enable Dynamic Routing on vPC VLANs

• Even with Peer-Gateway Routing protocols (e.g. OSPF) TTL expiry when traversing in transit the peer vPC Router device.

BRKDCN-2378 70


SVI 1

IP Y

VIP A

SVI 1

IP Z

VIP A

S1 S2

Router

Dynamic Routing over vPC

• Not recommended to attach L3 devices to vPC domain via L2 port-channel

• Common workarounds:

• Individual L3 links for routed traffic

• Static route to FHRP VIP

Devices without L3 over vPC support

SVI 2

IP X

A

Router

L3 ECMP

SVI 1

IP Y

VIP A

SVI 1

IP Z

VIP A

SVI 2

IP X

S1 S2

B

Router

SVI 1

IP Y

VIP A

SVI 1

IP Z

VIP A

SVI 2

IP X

Static Route to VIP A

S1 S2

BRKDCN-2378 71


P

OSPF/EIGRP

Primary

vPC

Secondary

vPC

OSPF/EIGRP

VLAN 99

72BRKDCN-2378


• Point-to-point dynamic routing protocol adjacency between the vPC peers to establish a L3 backup path to the core through PL in case of uplinks failure

• Define SVIs associated with FHRP as routing passive-interfaces in order to avoid routing adjacencies over vPC peer-link

• A single point-to-point VLAN/SVI (aka transit VLAN) will suffice to establish a L3 neighbor

• Alternatively, use an L3 point-to-point link between the vPC peers to establish a L3 backup path

Backup Routing Path

Routing Protocol Peer

P

Use one transit VLAN to establish L3 routing

backup path over the vPC peerlink in case L3

uplinks were to fail, all other SVIs can use

passive-interfaces

S2S1

S4S3

S5

P

P

P

L2

L3

Dynamic Routing over vPC


• Dynamic peering between Layer 3 device and vPC peers over vPC VLAN

• Dynamic unicast routing for IPv4

• Traffic does not get decremented TTL if travers peer-link

• “Peer-Gateway” should be enabled.

Dynamic routing over vPCConfiguration

Nexus(config-vpc-domain)# layer3 peer-router

PP

P

S1

vpc domain 200


10.10.12.42 source 10.10.12.52

peer-gateway

layer3 peer-router

S2

vpc domain 200


10.10.12.52 source 10.10.12.42

peer-gateway

layer3 peer-router

S1 S2

BRKDCN-2378 74


Dynamic routing over vPCExample Configuration and Verification on Nexus 7x00

PP

P

S1

vpc domain 200

peer-keepalive

destination 10.10.12.42

source 10.10.12.52

peer-gateway

layer3 peer-router

S2

vpc domain 200

peer-keepalive

destination 10.10.12.52

source 10.10.12.42

peer-gateway

layer3 peer-router


<snip>

vPC domain id : 100

<snip>

Peer Gateway : Enabled

<snip>

Operational Layer3 Peer : Enabled


<snip>

vPC domain id : 100

<snip>

Peer Gateway : Enabled

<snip>

Operational Layer3 Peer : Enabled

BRKDCN-2378 75


Benefits of Dynamic Routing over vPC

• No Static routes

• No Parallel links

• No design changes

• Route peering across vPC’s over existing infrastructure

• Routing between vPC DCI

BRKDCN-2378 76


Dynamic routing over vPCUse Case 1 : Firewall at Aggregation layer

• Peering Firewalls in routed mode over vPC

• Firewalls may be in active-standby mode

• Static routing / L3 P2P links NOT required

• External and internal traffic traverse same

port channel to firewall.

L3 Cloud

S1 S2

FW-A FW-B

Dynamic Peering Relationship

BRKDCN-2378 77


Dynamic routing over vPCUse Case 2 : Remote Orphan Site Peering in DCI Deployment

• vPC as Data Center Interconnect (DCI)

• Each Switch has routing adjacency with bothvPC device in other DC

• Each DC connected to a remote site byorphan port

• Remote sites forms routing adjacency withboth peers of its directly connected DC

Remote Site 1 Remote Site 2

S1 S2

S3 S4

BRKDCN-2378 78

Dynamic Routing over vPCSupported Topologies


Dynamic routing over vPCSupported Designs

P

PP

PP

PP

Note : Supported on Nexus 9X00 in ACI and NX-OS mode

Supported only in Nexus 7X00 on M3, F3, and F2E Line Cards, and Nexus 5x00

Currently not supported on Nexus 3X00, Nexus 7000 M1, M2, and F2 series Line card

Layer 3 services devices with vPC Layer 3 over DCI - vPC

BRKDCN-2378 80



P

PP

P

PP

STP inter-connection using a vPC VLAN Orphan device with vPC peers over vPC VLAN

BRKDCN-2378 81

Note : Supported on Nexus 9X00 in ACI and NX-OS mode

Supported only in Nexus 7X00 on M3, F3, and F2E Line Cards, and Nexus 5x00

Currently not supported on Nexus 3X00, Nexus 7000 M1, M2, and F2 series Line card



Peering with vPC peers over FEX vPC host interfaces

PP

P

Note : Supported only in Nexus 7X00 on F3 and F2E Line Cards starting from release 7.2.

Supported on Nexus 9X00 in ACI mode

Currently not supported on Nexus 5X00, Nexus 3X00, Nexus 9X00 (NX-OS mode), Nexus 7000 M-series Line card

BRKDCN-2378 82

Dynamic Routing over vPCUnsupported Topologies


Dynamic routing over vPCUnsupported Design

• The routing metric on S1 is less than the routing

metric on S2 (preferred path using S1).

• Traffic from A to B may hash to S2. This traffic

will need to traverse to peer-link to get to B

through S1.

• Due to the vPC loop avoidance rule S1 will not

allow traffic to flow to B.Router1

Router2

Po1

Po2

A

B

Po100

Int VLAN 10

Metric 20

Int VLAN 10

Metric 10

Int VLAN 10

Int VLAN 20

Int VLAN 20Int VLAN 20

S2

SVI

SVI

S1

Peering across vPC interfaces with unequal L3

metrics

Multicast over vPC


Design Best PracticesvPC and Multicast

Source

• vPC supports PIM-ASM (on all platforms)

• vPC supports PIM-SSM (on N9000 and N5600)

• vPC uses CFS to sync IGMP state

• Sources in vPC domain

− both vPC peers are forwarders

− Duplicates avoided via vPC loop-avoidance logic

• Sources in Layer 3 cloud

− Active forwarder elected on unicast metric

− vPC Primary elected active forwarder in case

metric are equal

− Active forwarder concept is per multicast

group/sourceReceivers

S2S1

Source


• vPC Terminology and Roles





• Key Takeaways

Agenda



• Isolates a switch from the vPC complex to:

• Debug

• Troubleshoot

• Physically isolate

• Minimal disruption of traffic flows

• “no shutdown” brings switch up

• Part of configuration, persistent after reload

• Recommended to have “peer-switch” enabled

vPC Shutdown

Primary Secondary

vPC

S1 S2

S3

switch# configure terminal

switch(config)# vpc domain 100

switch(config-vpc)# shutdown


vPC vPC

system mode maintenance

One command!Pre-change System Snapshot

Change window begins

Graceful Insertion and Removal - GIR

BRKDCN-2378 89


vPC vPC

One command!Pre/Post-change Snapshot Comparison

Change window complete

system mode normal

Graceful Insertion and Removal - GIR

BRKDCN-2378 90


Graceful Insertion and Removal

• Flexible framework providing a comprehensive, systemic method to isolate a node.

• Configuration profile foundation in NX-OS

• Initial support for:

• vPC/vPC+

• ISIS

• OSPF

• EIGRP

• BGP

• Interface

• Per VDC on Nexus 7x00

Platform Release

Nexus 5x00/6000 NX-OS 7.1

Nexus 7x00 NX-OS 7.2

Nexus 9000 NX-OS 7.X


ISSU with vPC

• ISSU (In Service Software Upgrade) recommended way system upgrade in a vPCenvironment

• vPC system can be independently upgraded

• Upgrade must be run one peer at a time

• Start with vPC primary switch

• Configuration is locked on “other” vPC peer during ISSU

• vPC run seamlessly with two different version of software

• Aggressive timers not supported

5.2(x) / 6.2(x)







• Key Takeaways

Agenda

vPC with VXLAN


Virtual Extensible LAN - VXLAN

• VXLAN is a network overlay technology

• VXLAN builds Layer-2 & Layer-3 overlay network on top of an IP routed network

• VXLAN uses MAC in IP-UDP encapsulation (UDP dest. port 4789)

• VLAN scale – VXLAN extends the L2 segment ID field to 24-bits

Benefits

MAN/WAN


VXLAN Packet Format

• VXLAN is a Layer 2 overlay scheme over a Layer 3 network.

• VXLAN uses Ethernet in UDP encapsulation

• VXLAN uses a 24-bit VXLAN Segment ID (VNI) to identify Layer-2 segments

FCSOuter

Mac Header

Outer

IP HeaderUDP Header VXLAN

HeaderOriginal L2 Frame FCS

Dst.

MA

C A

ddr.

Src

.

MA

C A

ddr.

VLA

N T

ype

0x8100

VLA

N I

D

Ta

g

Eth

er

Type

0x0800

IP H

eader

Mis

cD

ata

Pro

tocol

0x11

Header

Checksum

Oute

r

Src

. IP

Oute

r

Dst.

IP

UD

P

Src

. P

ort

VX

LA

N P

ort

UD

P L

ength

Checksum

0x0000

VX

LA

N

RR

RR

1R

RR

Reserv

ed

VN

I

Reserv

ed

14 Bytes

(4 bytes optional) 20 Bytes 8 Bytes 8 Bytes

48 48 16 16 16 72 8 16 32 32 16 16 16 16 8 24 24 8


VXLAN Terminology

• VXLAN terminates its tunnels on VTEPs (Virtual Tunnel End Point)• VTEP has two interfaces :

• Bridging functionality for local hosts• IP identification in the core network for VXLAN encapsulation / de-encapsulation

VTEP – Virtual Tunnel End Point

End SystemEnd System

VTEP

IP Interface

Local LAN Segment

End SystemEnd System

VTEP

IP Interface

Local LAN Segment

Transport IP Network


VXLAN

VLAN

vPC VTEP vPC VTEP

98BRKDCN-2378

vPC and VXLAN

• When vPC is enabled an ‘anycast’ VTEP address is programmed on both vPC peers

• Multicast topology prevents BUM traffic being sent to the same IP address across the L3 network (prevents duplication of flooded packets)

• vPC peer-gateway feature must be enabled on both peers

• Backup SVI, configured with PIM

• VXLAN header is ‘not’ carried on the vPCPeer link

vPC VTEP


VXLAN & vPCvPC Configuration

vtep

1

vtep

2

vtep

3

vtep

4

H110.10.10.10

VLAN 10(vpc)

H210.10.10.20

VLAN 10(vpc)

VTEP1vlan 10

vn-segment 10000

interface loopback 0

ip address <VTEP individual IP – orphan)

ip address <VTEP anycast IP – per VPC domain> secondary

!

interface nve1

source-interface loopback0

member vni 10000 mcast-group 235.1.1.1

!

Vlan 99

!

Interface vlan 99

ip address 99.1.1.1/24

ip ospf cost 10

ip router ospf 1 area 0.0.0.0

ip pim sparse-mode

!

vpc nve peer-link-vlan 99

Map VNI to VLAN

VXLAN Tunnel Interface

Source Interfaceindividual IP is used for single attached Hostsanycast IP is used for VPC attached Hosts

VTEP2vlan 10

vn-segment 10000

interface loopback 0

ip address <VTEP individual IP - orphan>

ip address <VTEP anycast IP – per VPC domain> secondary

!

interface nve1

source-interface loopback0

member vni 10000 mcast-group 235.1.1.1

!

Vlan 99

!

Interface vlan 99

ip address 99.1.1.2/24

ip ospf cost 10

ip router ospf 1 area 0.0.0.0

ip pim sparse-mode

!

vpc nve peer-link-vlan 99

Vlan for VXLAN vPC peer-link

SVI for the VXLAN vPC peer-link

Enable the VLAN on the VXLAN vPC peer-link

BRKDCN-2378 99


VXLAN & vPCDual attached Host to dual attached Host (Layer-2)

vtep

1

vtep

2

vtep

3

vtep

4

H110.10.10.10

VLAN 10(vpc)

H210.10.10.20

VLAN 10(vpc)

• Host 1 (H1) and Host 2 (H2) are dual

connected to a vPC domain

• As H1 is behind a VPC interface, the

anycast VTEP IP is the source for

the VXLAN encapsulation

• As H2 is behind a VPC interface, the

anycast VTEP IP is the target

vtep

20vtep

30

BRKDCN-2378 100

vPC with ACI


Nexus 9000 + APIC = ACI

APICAPIC

APIC

BRKDCN-2378 102


External

Network

App DBWeb

QoS

Filter

QoS

Service

QoS

Filter

ACI uses a policy based approach that focuses

on the application.

BRKDCN-2378 103


vPC and ACI

• No dedicated peer-link between vPCpeers:

• Fabric itself serves as the peer-link

• No out-of-band mechanism to detect peer liveliness:

• Due to rich fabric-connectivity (leaf-spine), it is very unlikely that peers will have no active path between them

• CFS (Cisco Fabric Services) is replaced by Zero Message Queue (ZMQ)

• As ACI fabric is VXLAN-based, an anycastVTEP is shared by both leaf switches in a vPC domain

ACI fabric utilised for control-planevPCDomains

vPCpeers

ACI

fabric

vPC vPC

vtep

1

vtep

2

vtep

3

vPC with FCoE


Fiber Channel over Ethernet - FCoE

• Fiber Channel traffic over Ethernet

• Ethernet standards to support FCoE:

• Priority Flow Contol – PFC

• Enhanced Transmission Selection – ETS

• Data Center Bridging Exchange – DCBX

FC PayloadEthernetHeader F

CSFCoE

HeaderFC

Header EO

F

CR

C


vPC with FCoEUnified Fabric Design

• vPC with FCoE is supported between hosts Nexus 9000, Nexus 7X00, Nexus 5X00 and N5X00 & N2X00 pairs.

• vPC and FCoE only on the first hop

• Each vPC peer must be part of separate fabric.

• Best Practice: Use static port channel rather than LACP with vPC and boot from SAN.

VLAN 10,30

VLAN 10,20

STP Edge Trunk

VLAN 10 ONLY HERE!

Fabric A Fabric BLAN Fabric

Nexus 5000

FCF-A

Nexus 5000

FCF-B

vPC contains only 2 X 10GE

links – one to each Nexus 5X00

vPC with FabricPath


Switch(config-if)# switchport mode fabricpath

109BRKDCN-2378

FabricPath: an Ethernet Fabric

• Spanning Three Protocol independence

• High MAC address scalability with conversation learning on Edge ports

• Unique Switch ID (SID) identifies switches in FabricPath fabric

• IS-IS for control plane information exchange

• Multi destination Trees for BUM traffic

• Loop mitigation with TTL

• Simple CLI configuration

Shipping on Nexus 7x00, Nexus 600x and Nexus 5x00

FabricPath


vPC vs vPC+

• Physical architecture of vPC and vPC+ is the same from the access edge

• Functionality/Concepts of vPC and vPC+ are the same

• Key differences are addition of Virtual Switch ID and Peer Link is a FP Core Port

• vPC+ is not supported on Nexus 9X00 & Nexus 3X00 Series

Architecture of vPC and FabricPath with vPC+

FP VLAN’s

vPC+vPC

FP Port

FP

CE Port

CE VLAN’s

CE


Dynamic Routing over vPC+

• Layer 3 devices can form routing adjacencies with both the vPC+ peers over vPC

• The peer link ports and VLAN are configured in FabricPath mode

• PIM-SSM multicast

• L3 peering with vPC+ plus devices is not supported on N7X00

P Routing Protocol Peer

N55xx, N56xx, N6000

Router/ Firewall

Dynamic Peering Relationship

Fabricpath Link

vPC

P P

P

FabricPath







• Key Takeaways

Agenda


Key Takeaways

• Full bandwidth of the network without Spanning Three

• High availability and improved convergence for downtime reduction

• Dynamic routing over vPC for appliance connectivity

• Dual-homing device in ACI, VXLAN, FCoE and FabricPath environment

• WHO HAS LEARN SOMETHING NEW TODAY?


Related Sessions

Session Id Session Name

BRKACI-1001 Your first 7 days of ACI

BRKDCN-2404 VXLAN deployment models - A practical perspective

BRKDCN-2304L4-L7 Service Integration in Multi-Tenant VXLAN EVPN

Data Center Fabrics

BRKDCN-2458Nexus 9000/7000/6000/5000 Operations and

Maintenance Best Practices

BRKDCN-2378 114


• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 gift card.

• Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us.

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.

http://www.ciscolive.com/us

http://ciscolive.com/Online

http://ciscolive.com/Online

http://www.ciscolive.com/Online


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKDCN-2378 116

Thank you

vpc best practices and design on nx-osd2zmdbbm9feqrf.cloudfront.net/2017/usa/pdf/brkdcn-2378.pdf ·...

Documents