brkdct-3144

Troubleshooting Cisco Nexus 7000 Series Switches BRKDCT-3144

Dipl.-Ing. Andreas la Quiante [email protected]

Nexus Product Management Cisco Data Center Group

Level 3

11:30

Chapter 0: Housekeeping

Our ASICs are

starting with zero,

so do we today ;-)

© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-3144 Cisco Public

Housekeeping Contribution & Great Help from:

Matt

Martin Ron

Roland

Dmitry Ronald

Adam

Need help like me? Terri

They provided feedback, answers to my questions.

Also I borrowed some content…Danke!

Anver

Q&A Moderator


Housekeeping Icons

N7K

Switch

Router

PC

Layer 3

Layer 2

Focus

Notes

N7004-Berlin#

sh int e 3/12 CLI

my notes

Geek

content

Error/Failure/Challenge

Cisco TAC

Partner request:

Include „real cases/examples“

Reference Slide Hidden Slides

148 in total [28-JAN-14]

Interface


8

Housekeeping The Rides

1

2

3

4

5

6

Strategy

Tools & System

Data-Plane

Layer 2

Backup

Data-Plane

Layer 3 Cisco Live 2014: 90 min

Control-Plane

Inband

Control-Plane

ARP

1

Layout of each chapter, like a train/subway line with a color code

25

20

12

12

8

1 1

WebEx 120 min for YOU

12-FEB-14 10:00 CET opt-in:

[email protected]

“subject BRKDCT-3144”

11:35

mailto:[email protected]

© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-3144 Cisco Public 9

Housekeeping Adjacent Sessions

Breakouts Avoiding any content overlap since we have only 90 minutes today

please consider the following sessions to complement your skills:

BRKDCT-2049 Overlay Transport Virtualization

BRKDCT-2048 Deploying Virtual Port Channel in NX-OS (vPC)

BRKDCT-3144 Troubleshooting Cisco Nexus 7000 Series Switches

BRKDCT-2121 VDC Design and Implementation Considerations with Nexus 7000

BRKNMS-2695 Admin.and Mon.of the Cisco DataCenter with Cisco Prime DCNM

BRKDCT-3346 End-to-End QoS Implementation and Operation with Cisco Nexus

BRKDCT-3445 Building scalable data center networks with NX-OS and Nexus 7000

BRKDCT-3145 Adv - Troubleshooting Cisco Nexus 5000 / 2000 Series Switches

BRKARC-3470 Advanced - Cisco Nexus 7000/7700 Switch Architecture

BRKDCT-2081 Cisco FabricPath Technology and Design

BRKDCT-3103 Advanced OTV – Config/Troubleshooting OTV in the network

0

Chapter 1: Strategy, Tools and System

Strategy

CLI

Ethanalyzer

ELAM

ELAME

System


Strategy Three areas

Direction System Troubleshooting - Core, CPU, Memory, Interface/Vlan behaving odd, hardware challenges

1

Data Plane Troubleshooting - Packets are lost - your primary questions is “where” - 100% loss or partial loss - consistent or periodically

Control Plane Troubleshooting - Something is flapping - Convergence challenges - start at the process (log)

ELAM(E)

“Anything better

than checking

everything is an

improvement”

ALQ 2014

L2

Inband ARP

TCAM L3

L3

ELAM(E) Ethanaylzer

ELAM: Embedded Logic Analyzer

Memory leaks not common on NX-OS

Each process has an max-mem limit N7K# sh sys internal memory-alert-log


Strategy System, Data-Plane, or Control-Plane

I/O Module

(Forwarding

Engine)

I/O Module

(Forwarding

Engine)

System

Control-

Plane

Data-

Plane

1

2 3 6

4 5

Reference Point 2

Reference Point 1

Supervisor

(Control-Plane)

Tools

13 11:40


Tools NX-OS Logging and a Powerful CLI

NX-OS Value

NX-OS is build up

with most extensive,

fine granular logging

capabilities

NX-OS CLI, SNMP

XML, Python

GUI, OF

OnePK

Chef, Puppet

High Performance

Feature Rich Switching

Logging

Switching

Logging

Configuration

1

NX-OS:

Build in

Flight Recorder

Standard CLI

Engineering CLI

Internal keyword

output is not

documented


Tools Show Tech

Show tech ABC Always try to use the detailed version show tech detail

Feature

Event history

States (PSS,...)

HW states

Always redirect to a file

Always use a separate file per show tech

1

Suggestion for VDC: BRKDCT-2121

VDC Design and Implementation Considerations

Global Service

VDC-1 Default

Feature

“project binary logger”

Significan time saver Show tech all-binary

Avoiding also

“we need show tech A”

after a while doing RCA

“we need show tech B”

For use by

TAC/BU/ENG

t0 t2 t3 t1

t0 to t2 trigger failure

Immediately collect data!

Then start

troubleshooting


17

Tools Custom ASICs provide detaild counters via CLI

ASICs Some „error“ counters are part of a normal operation

(e.g. dropping packets at ingress trunk if the marked VLAN is not

known (CBL drops), diag packets, extra flooded packets)

One of TAC‘s favourite commands. Use „all“ to look for all

modules / ASICs

1

N7004-Berlin# show hardware internal errors module 3

|------------------------------------------------------------------------|

| Device:Clipper MAC Role:MAC Mod: 3 |

| Last cleared @ Mon Nov 25 21:41:37 2013

| Device Statistics Category :: ERROR

|------------------------------------------------------------------------|

Instance:2

Cntr Name Value Ports

----- ---- ----- -----

0 GD GMAC bad character interrupt 0000000000000002 12 -

1 GD GMAC sequence error interrupt 0000000000000002 12 -

2 GD GMAC transition from nosync to sync int 0000000000000002 12 -

3 GD GMAC transition from sync to nosync int 0000000000000001 12 -

4 PL ingress_cbl_drop 0000000000003426 12 -

GD GMAC Build in MAC Controller

Our innovative ASICs

provide many counters

ASICs are a great

source of information

(esp. for the Data Path)

e.g.

F1/F2/F3

Non-Zero Counter


Tools CLI-Tools: How can I select what I need?

Tips & Tricks

N7004-Berlin# show system internal pktmgr interface

<SNIP>

Vlan1, ordinal: 38 Hash_type: 1

SUP-traffic statistics: (sent/received)

Packets: 2769 / 1896

Bytes: 1619370 / 241310

Instant packet rate: 1 pps / 0 pps

Packet rate limiter (Out/In): 0 pps / 0 pps

Average packet rates(1min/5min/15min/EWMA):

Packet statistics:

Tx: Unicast 1123, Multicast 1641

Broadcast 5

Rx: Unicast 163, Multicast 1730

Broadcast 3

N7004-Berlin# show system internal pktmgr interface |i or|I

<SNIP>

Vlan1, ordinal: 38 Hash_type: 1



port-channel100, ordinal: 72 Hash_type: 1



If I am only interested in

parts of the output I can ask

for just those items

You save time by having to

read less

Nexus# sh ver | ?

egrep Egrep -

grep Grep -

head Displ 1st ln

last Displ last

less Filter

no-more

sed

wc Count

begin Begin with

count Count

exclude Exclude ln

include Include ln


Tools Log Window (upcoming)

1

N7K# source logw.py 15/01/2014 12:24:55 100

starting with empty stats stats init done Logw system check port version

0.060813

Time range 2014-01-15 12:24:55 ... 2014-01-15 12:26:35 Got 343 show

... event-history clis

244 clis left after pre-filtering

collecting outputs...done, collected 2602 events in 96.197735 seconds sorted

<snip>

Trigger

logw.py [-h] [-v] [-f FILTERS] [-t TRUNCATE] [-n

MAX_EVENTS] [-s] start_date start_time duration

Logfile

10MB

logfile

NVRAM

On-board

Event

History

Tip: show log log immediately displays the logfile output, and is

faster than show log which has to read the logging severity settings

On-Board: Major state changes, MTS transactions, useful for module troubleshooting

It is a good idea to synchronize all devices in

your network to one time source

11:45


Tools Accounting Log or who did it?

Audit Recording

N7004-Berlin# show accounting log | last 3

Mon Dec 2 03:33:05 2013:type=update:id=console0:user=admin:cmd=switchto ;

configure terminal ; interface port-channel110 ; shutdown (SUCCESS)


configure terminal ; interface port-channel110 ; no shutdown (REDIRECT)


configure terminal ; interface port-channel110 ; no shutdown (SUCCESS)

N7004-Berlin(config)# terminal log-all

N7004-Berlin(config)# show accounting log all | last 2


show accounting log all | last 2 (SUCCESS)


show hardware internal errors all (SUCCESS)

Only configuration commands are captured by default. Enable all commands to be captured with terminal log-all

(feature requires 5.x NX-OS or higher)

1

Trigger

With the log informaton you see what happend

With the accounting-log you see who triggered it and more

importantly which action triggered it

logw.py


Tools Ethanalyzer

Guidance In production networks usable (low risk)

1

Use “display” mode so simply verify packet is present

Use pcap-file for detail analysis outside the device at a later time

Ethanalyzer Kernel N7004# ethanalyzer local interface inband decode-internal limit-frame-size 150

display

Capturing on inband

2013-12-07 16:04:07.855965 Cisco_b5:26:49 -> PVST+ STP 96 RST. Root = 327

68/1/00:0c:30:8b:a0:40 Cost = 2 Port = 0x9063

<SNIP>

Allows quickly to

narrow the

failure domain

Ethanalyzer captures traffic to/from the control plane and not the data plane

Inband Reference Point 1

Reference Point 2 ELAM(E)

Ethanaylzer Ethanalyzer captures traffic

for Control-Plane Troubleshooting

The second reference point serves

traffic for both Control- and Data-

plane Troubleshooting Ingress


Tools ELAM & ELAME

ELAM & ELAME

Embedded Logic

Analyzer Module

1

It is widely used by engineering, QA, TAC and escalation teams

ELAM is an unsupported and internal tool

ELAM requires a great deal of platform architecture and ASIC

knowledge to use. This limits the audience of the raw tool.

Identifying the appropriate FE, creating triggers, and interpreting

ELAM data for complex flows requires full architectural and

forwarding knowledge

Good news: ELAME makes ELAM easy to use

skill

ELAME

F-Series

M-Series


Tools ELAM/ELAME Motivation

workflow

1

Determin the FE

Configure Trigger

Start ELAM

Analyze

ELAM allows you to verify if a packet is present and/or to analyze

ELAME allows you to verify quickly if a packet is present,

especially in a complicated setup it saves you TIME!

Use cases:

1) Determining the failure

domain

2) Analyze the System

behaviour

IP 42.42.42.1

MAC aaaa.bbbb.cccc IP 42.42.42.2

MAC aaaa.bbbb.dddd

You MUST know the source and destination

MAC/IP pairs involved for troubleshooting.

Is the source and/or destination dual-homed?

Is the source and/or destination real or virtual?

Êureka, Lamira

Orion, Clipper, Flanker

11:50

Attach to the module


Tools ELAME, Part 1

ELAME

1

N7004-Paris# source sys/elame 10.0.2.2 224.0.0.5

elam helper, version 1.015

... source 10.0.2.2, destination 224.0.0.5

... getting current vdc ... 4

... ingress interface derived from source address

... ingress interface list is Ethernet4/1

... expanded ingress interface list is Ethernet4/1

... FE instance list is 4/1/1

... setting trigger...

... elam trigger set

... starting capture...

... elam capture started

... no packet captured so far

press [enter] when packets in question are known to have been sent…

... packet captured at FE: 4/1/1

... capture instance 4/1/1 (slot/type/instance)

Since NX-OS 6.2(2) we include „elame.tcl“ in the

distribution:

Berlin

10.0.2.2/24

Paris

10.0.2.4/24

Do we receive OSPF

packets from our

neighbor on E 4/1?

E 4/1

M-Series line card

skill

ELAME

F-Series

M-Series Because ELAM especially on M-Series is complicated

this example show how easy it is to use ELAME

ELAME works on F2 and M-Series line cards with IPv4

You just specify source and destination address

the tool determines the correct FE to programm

even on M-Series Modules


Tools ELAME, Part 3

ELAME

1

N7004-Paris# source sys/elame 10.0.2.2 224.0.0.5

<SNIP>

... packet captured at FE: 4/1/1

... capture instance 4/1/1 (slot/type/instance)

+++ IPv4 packet: 86 bytes from MAC 4055.390f.5642 / IP 10.0.2.2 to MAC

0100.5e00.0005 / IP 224.0.0.5 TTL 1

+++ protocol OSPF

+++ packet received on interface Eth4/1 vlan 0 (source index 0x00030)

... rbus: ccc 0x0 cap1 0x1 cap2 0x1 flood 0x1 dest_vlan 0 dest_index

0x00032 l2_fwd 0x0

+++ packet is flooded to BD 50 / vlan 0

... destination index is NOT from L2 table lookup

+++ copy of the packet is sent to CPU

... lamira OFE: rdt 0x0 dest_index 0x010c7 flood 0x0 l2fwd 0x0 ofe_drop 0x0

+++ lamira OFE exception(s): CPP_LIF (0x200000000)

... FE instance 4/1/1 context after analysis: pb2 retried

... done

DBUS and RBUS captured,

easy tool even on M-Series line cards (here N7K-M224)

skill

ELAME

F-Series

M-Series

E 4/1

LTL 0x30

SUP

LTL 0x10C7

Paris Berlin

Lamira

Eureka

The lines beginning with +++ are the important once

ELAM(E)

Ethanaylzer


Tools ELAM

ELAM F2

Embedded Logic

Analyzer Module

1

F2 no PB for ELAM (:= more simple but the recommendation is

to still use ELAME like the pros)

Clipper: Layer 2 ELAM and/or Layer 3 ELAM

module-3# elam asic clipper instance 2

Module-3(clipper-elam)# layer 3

module-3(clipper-l3-elam)# trigger dbus ipv4 if source-ipv4-address

42.42.42.142

module-3(clipper-l3-elam)# trigger rbus ofe if trig

module-3(clipper-l3-elam)# start

module-3(clipper-l3-elam)# status

<SNIP>

L2

L3

Clipper FE2

E3/12

OFE

IFE

OFE := Outgoing „Pipeline“

IFE := Incomming „Pipeline“

Status: Armed := waiting for the packet

Status: Triggered := we have captured


Tools ELAM, DBUS

ELAM F2

Embedded Logic

Analyzer Module

1

42.42.42.142

E 3/12

F-Series line card

module-3(clipper-l3-elam)# show dbus

--------------------------------------------------------------------

Clipper Instance 02 - Capture Buffer On L3 DBUS:

<SNIP>

--------------------------------------------------------------------

L3 DBUS CONTENT - IPV4 PACKET

--------------------------------------------------------------------

<SNIP>

l2-packet-length : 0x52 ingress-lif : 0xfca

vlan-id : 0x2a ilm-addr : 0x32

source-index : 0x402 destination-index : 0x0

frame-type : 0x5 sequence-number : 0x94

l2-frame-type : 0x0 l4-protocol : 0x59

recirc-preserve-acos: 0x0

recirc-multicast-bridge-disable: 0x0

ipv4_l4_info_elsewhere_1: 0x0

ipv4_l4_info_elsewhere_2: 0x0

destination-mac-address: 0100.5e00.0005

source-mac-address: 0010.7be8.53b0

source-ipv4-address: 42.42.42.142

Destination-ipv4-address: 224.0.0.5

Berlin


Tools ELAM, RBUS

ELAM F2

Embedded Logic

Analyzer Module

1

42.42.42.142

E 3/12

F-Series line card

module-3(clipper-l3-elam)# show rbus

--------------------------------------------------------------------

Clipper Instance 02 - Capture Buffer On L3 RBUS:

<SNIP>

--------------------------------------------------------------------

L3 RBUS OFE CONTENT

--------------------------------------------------------------------

OFE valid: 0x1

trig : 0x1 l2-l3-acos : 0x0

<SNIP>

dvif : 0x0 vlan : 0x2a

md-di-valid : 0x0 redirect : 0x0

ccc : 0x4 l2-forward : 0x1

routed : 0x0 eid-select : 0x0

lif-status-enable : 0x1 bcn-compatible : 0x0

VID 42:= 0x2a

Berlin

11:55


Tools ELAM

ELAM F2

Embedded Logic

Analyzer Module

1

module-3# elam asic clipper instance 2

Module-3(clipper-elam)# layer 2

Module-3(clipper-l3-elam)# trigger dbus ipv4 if destination-ipv4-address

42.42.42.142

Module-3(clipper-l3-elam)# trigger rbus ingress if trig

L2

L3

Clipper FE2

E1/12

egr ingr

Since the former example indicated no Layer 3 rewrite

we look now into Layer 2 ELAM (still looking for Layer 3

information)

module-3(clipper-l2-elam)# show rbus

<SNIP>

inner-cos : 0x0 acos : 0x0

di-ltl-index : 0x8015 l3-multicast-di : 0x0

source-index : 0x402 vlan-id : 0x2a

index-direct : 0x0 eid-sel : 0x0

vqi : 0xfa v5-fpoe-idx : 0xf9

l3-fpoe-idx : 0x0 l3-multicast-v5 : 0x0

dft : 0x0 dfst : 0x0

System Troubleshooting

42


System Troubleshooting Is my Interface operational and who owns my Interface?

Ethernet IF

E 3/12

1

N7004-Berlin# show int eth 3/13

Ethernet3/13 is down (SFP not inserted)

N7004-Berlin# show int eth 3/12

Ethernet3/12 is up

The Interface could be described as the Port-ASIC including the

MAC Controller

Another view would be the Software Process in

the Control Plane Ethpm (:= Ethernet Port Manager)

An up-to-date

network drawing helps Ethpm

VID 1

VID 42 STP

Vlan Mgr show system internal vpcm event interaction


System Troubleshooting Interface shutdown timeout

Ethernet IF

E 1/27

1

Ethpm

Phy_off

802.1X

PIXM

ACL

QOS

L2FM

STP

N7K(config)#

interface e1/27

N7K(config-if)# shut

N7K# show inter e1/27

Ethernet1/27 is down

(Internal-Fail

errDisable,

libeventseq:

sequence timeout)

Processes and Services are

depending on each other

Collect information about

the whole environment: (e.g. Show tech )

As you likely don‘t know all

dependent processes

Ethpm is interacting with each

service sequencially

(Request and Response)

OK, how about shutting

down a port (e.g. e1/27)? N7K(config-if)# shut


System Reducing MTTR

Core Files

1 Collect cores form „all“ locations on the active (don‘t forget your

standby SUP) and attach them to a TAC case right away

N7004# show cores vdc-all

VDC Module Instance Process-name PID Date(Year-Month-Day Time)

--- ------ -------- --------------- -------- -------------------------

VDC Module Instance Process-name PID Date(Year-Month-Day Time)

--- ------ -------- --------------- -------- -------------------------

1 17 1 pixmc 2134 2013-10-28 16:52:48

1 8 1 pixmc 2134 2013-10-28 16:52:50

1 16 1 pixmc 2134 2013-10-28 16:52:50

If you find a file, be prepared to send it (a selection)

or attach them right away to the case to save time

SR 123

2010 Jul 17 00:30:18 vrt001 %$ VDC-1 %$ %SYSMGR-SLOT8-2-SERVICE_CRASHED:

Service "mtm" (PID 1600) hasn't caught signal 6 (core will be saved). SUP-A

active

SUP-B

stdby

VDC-1 VDC-1

VDC-2 VDC-2

Here you see „slot 8“ := you know the line card and MTM is a line

card process

look in dir bootflash:core (of both SUPs) to make sure...and clean up...

12:00


Strategy Tools and System Troubleshooting Summary & Take Away

Flight Recorder

Strategy CLI

Ethanalyzer

ELAM

Baseline your network, know your counters and have a show tech in a good state, have an up-to-date drawing

Right after the „challenge“ collect a show tech detail

if feasible

Traffic to/from the control plane: Wireshark DataPlane capture with Nexus 7000: ELAME

Look at „normal“ counters, logs and if you need more: remember show hardware internal errors all

1

ELAME

System

Chapter 2: Data- Plane Layer 2

2 MAC Table

PIXM

L2FM

STP


Data-Plane Failure Domain

Failure Domain I am loosing packets between A and B?

How can I determine „where“

2

Determine Failure Domain Quickly

100% traffic loss:

• Table not

progammed

• Wrongly

programmed

• Inconsistency

100% traffic loss?

X % traffic loss?

ELAME

X % traffic loss:

• Congestion?

• Periodically?

Timer/Aging event

(e.g. MAC Table)

A B


Architecture Three Stage Fabric,

Troubleshooting

At the ingress

forwarding engine

for unicast

multicast replication

occures at the

egress line card

Congestion

F-Series (Ingress)

M-Series (Egress)

Ingress Module

First Stage Egress Module

Third/Last Stage

EALR 8

SoC Xbar Xbar

Xbar

EALR 8

SoC

Suggestion for QoS & Queueing: BRKDCT-3346

End-to-End QoS Implementation and Operation


Data-Plane Congestion: X % packet drops

Troubleshooting

N7009-Lagos# show hardware internal errors all

|------------------------------------------------------------------------|

| Device:Sacramento Xbar ASIC Role:FABRIC Mod: 9 |

| Last cleared @ Fri Nov 15 02:19:12 2013

| Device Statistics Category :: ERROR

|------------------------------------------------------------------------|

Instance:0

ID Name Value Ports

-- ---- ----- -----

2129 FB09-P21 LOW_BP_CNT_IN 0000000000000099 1-48 I1-2

|------------------------------------------------------------------------|

| Device:Clipper XBAR Role:QUE Mod: 9 |

| Last cleared @ Fri Nov 15 05:18:38 2013

| Device Statistics Category :: CONGESTION

|------------------------------------------------------------------------|

Instance:0

ID Name Value Ports

-- ---- ----- -----

132 VQ credited pkt replica VOQ tail drops 0000000000000189 1-4 -

137 VQ credited pkt replica drop count 0000000000000189 1-4 -

9602 VQ VQI 204 CCOS 3 drop count 0000000000000189 1-4 -

Clipper

Sacramento

BP :=

Backpressure

Not the first command to use

Displays non-zero error counters

CMD – test with x packets - CMD again

System

FPGA Version on

FAB2 needs to be

PM 0.007 for SUP-2/2E

Q

Verify our System status

before troubleshooting

12:05


Data-Plane Line Card Architecture

LC Families

EARL based Line

Cards

M-Series

(:= M1, M2)

SoC based Line

Cards

F-Series

(:= F1, F2, F3)

M2

2 x per LC

SoC

e.g. F2E

Clipper

up to 60 mpps

per SoC

Fabric

ASIC

Fabric

ASIC

EARL 8

Up to

60mpps

L2

L3

P

R

Q

Q:= Queuing Engine

R:= Replication Engine

P:= Port ASIC

FE .= Forwarding Engine

F1

16 x SoC

F2/F2E

12 x SoC

F3 N7K

(and all

1G/10G)

6 x SoC

F3 N77

12 x SoC

Q R P

FE

M-Series F-Series

Suggestion: BRKARC-3470

Advanced - Cisco Nexus 7000/7700 Switch Architecture


Data-Plane

Forwarding

Similar: show platform hardware

capacity

forwarding on C6K

N7004-Berlin# show hardware internal forwarding engine usage

slot 4

Forwarding Engine Usage

-----------------------

Module inst pps peak pps

4 1 0 4 @Tue Nov 26 20:17:33 2013

N7004-Berlin# show hardware internal statistics module 3 rates

Hardware statistics on module 03:

+ =============================

+ Clipper MAC Instance 0

+ =============================

|-- Ingress IN

| |--- Packets/sec

| | |--- 2: 0

| | |--- 1: 0

| | |--- 3: 0

| | |--- 4: 0

| | |--- sum: 0

| |--- Bytes/sec

| | |--- 2: 3

| | |--- 1: 75

| | |--- 3: 91

| | |--- 4: 0

| | |--- sum: 169

|-- Egress OUT

| |--- Packets/sec

| | |--- 2: 0

| | |--- 1: 0

| | |--- 3: 0

| | |--- 4: 0

| | |--- sum: 0

| |--- Bytes/sec

| | |--- 2: 3

| | |--- 1: 76

| | |--- 3: 87

| | |--- 4: 73

| | |--- sum: 239

This command works for

M-Series line cards

This command works for

F-Series line cards

FE 0

E 3/1

vPC PKA

E 3/2 & 3/3

vPC PL

Module 3: F2


Data-Plane Line Card Components

LC Internals

module-1# show hardware internal dev-port-map

--------------------------------------------------------------

CARD_TYPE: 12 port 100G

>Front Panel ports:12

--------------------------------------------------------------

Device name Dev role Abbr num_inst:

--------------------------------------------------------------

> Flanker Eth Mac Driver DEV_ETHERNET_MAC MAC_0 12

> Flanker Fwd Driver DEV_LAYER_2_LOOKUP L2LKP 12

> Flanker Xbar Driver DEV_XBAR_INTF XBAR_INTF 12

> Flanker Queue Driver DEV_QUEUEING QUEUE 12

> Sacramento Xbar ASIC DEV_SWITCH_FABRIC SWICHF 2

> Flanker L3 Driver DEV_LAYER_3_LOOKUP L3LKP 12

> EDC DEV_UNDEFINED PHYS 12

+-----------------------------------------------------------------------+

+----------------+++FRONT PANEL PORT TO ASIC INSTANCE MAP+++------------+

+-----------------------------------------------------------------------+

FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF

1 0 0 0 0 0,1

2 1 1 1 1 0,1

3 2 2 2 2 0,1

4 3 3 3 3 0,1

5 4 4 4 4 0,1

<SNIP>

EDC0 EDC1

Flanker

0

Flanker

1

SAC0 SAC1

000c.308b.a040


Layer 2 Hardware Learning

Layer 2

Berlin

PO 110

MAC Address Table (16K, 64K, or 128K)

2

N7004-Berlin# show mac address-table vlan 1

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link,

(T) - True, (F) - False

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

G 1 0000.0c9f.f001 static - F F sup-eth1(R)

G 1 4055.390f.5642 static - F F sup-eth1(R)

* 1 4055.390f.5643 static - F F vPC Peer-Link

* 1 000c.308b.a040 dynamic 0 F F Po110

MAC Address Table

000c.308b.a040

N7004-Berlin# show hardware internal forwarding f2 l2 table utilization

L2 entries: Module inst total used mcast ucast lines lines_full

3 0 16384 15 0 15 512 0

N7004-Berlin# show hardware internal forwarding l2 table utilization

L2 entries: Module inst total used mcast ucast lines lines_full

4 1 131072 22 8 14 8192 0

Sync via CFS


Layer 2 Hardware Learning, Moves and Aging

Layer 2

MAC A

MAC Index Flag

A PO1 PI_E

C 3/3

MAC Index Flag

A PO1 PI_E

C 3/3

MAC Index Flag

A PO1

C 3/3 PI_E

MAC C

E 1/1 E 2/2 E 3/3

Line Card 1 Line Card 2 Line Card 3

PO1

2 L2FM show mac address-table …

show hardware mac address-table …

Learning and Aging optimized for physical and logical ports

(:= PC Port Channel) with additional signaling via L2FM

L2FM

N7004-Berlin(config)# logging level l2fm 6

2013 Dec 17 02:52:46 N7004-London %$ VDC-3 %$ %L2FM-4-L2FM_MAC_MOVE: Mac

f0de.f1f2.c804 in vlan 42 has moved from Eth3/37 to Eth3/41

2013 Dec 17 02:53:00 N7004-London %$ VDC-3 %$ %L2FM-4-L2FM_MAC_MOVE: Mac

f0de.f1f2.c804 in vlan 42 has moved from Eth3/41 to Eth3/37

12:10


Layer 2 Looking for the internal history of a MAC address

Layer 2

L2FM

Looking back in time for a specific MAC Address

2

65

N7004-London(config)# show system int l2fm l2dbg macdb address f0de.f1f2.c804

Legend

Db: 0-MACDB, 1-GWMACDB, 2-SMACDB, 3-RMDB, 4-SECMACDB

Src: 0-UNKNOWN, 1-L2FM, 2-PEER, 3-LC, 4-HSRP

5-GLBP, 6-VRRP, 7-STP, 8-DOTX, 9-PSEC 10-CLI 11-PVLAN

12-ETHPM, 13-ALW_LRN, 14-Non_PI_MOD, 15-MCT_DOWN, 16 - SDB

17-OTV, 18-Deounce Timer, 19-AM, 20-PCM_DOWN, 21-MCT_UP, 22-L2VPN

Slot:0 based for LCS 19-MCEC 20-OTV/ORIB

VLAN: 42 MAC: f0de.f1f2.c804

Time If/swid Db Op Src Slot FE

Sat Dec 14 22:18:20 2013 0x1a124000 0 INSERT 3 2 9

Sat Dec 14 22:18:20 2013 0x1a124000 0 RESET_LL_UNDERWAY 2 0 15

Sat Dec 14 22:18:51 2013 0x1a124000 0 NON_PI_MOD 3 2 15



Sat Dec 14 22:19:31 2013 0x1a124000 0 FLUSH 12 0 15

Sat Dec 14 22:19:31 2013 0x1a124000 0 DELETE 0 0 15

Sat Dec 14 22:19:36 2013 0x1a128000 0 INSERT 3 2 10

12

3

6

N7004-London# show interface snmp-ifindex |i 1a124000 Eth3/37 !Port 437403648 !IFMIB (0x1a124000) !IFINDEX


Layer 2 Internal Header with Meta Data to make Forwarding Decision and steer Frame

Layer 2

LTL := Local Target Logic (e.g. Source Index (SI) and

Destination Index (DI) e.g. 0x00402)

BD := Bridge Domain

E 3/1

Internal Header added by

PORT ASIC or SoC

(FE)

Ingress L2 Logic learns

MAC Address in HW

(M & F-Series)

Header Packet

DI = 402h VLAN, ...

Internal Header contains

SI, DI, VLAN

SI = BAh

402h

Org Packet

We add an internal header to carry

needed information (e.g. Index, VLAN)

2

N7004-Berlin# show hardware mac address-table 3 address 000c.308b.a040

!reformatted!

FE | Valid| PI| BD | MAC | Index| Stat| SW | Modi| Age| Tmr| GM|

---+------+---+------+---------------+-------+-----+-----+-----+----+----+---

0 1 0 17 000c.308b.a040 0x00402 0 0x009 0 121 1 0

2 1 1 17 000c.308b.a040 0x00402 0 0x009 0 121 1 0

+

removed

Packet


Layer 2 Internal Indices

Layer 2

0402h

8011h

BD – VLAN

VDC 2:17 := 1

DB

PO110

A interface is assigned

one or more indices

One port gets assigned one or more index values, internally we

use the concept of bridge domains (which map to VLAN ID)

2

67

N7004-Berlin# show system internal pixm info ltl 0x00402

PC_TYPE PORT LTL RES_ID LTL_FLAG CB_FLAG MEMB_CNT

------------------------------------------------------------------------------

Normal Po110 0x0402 0x1600006d 0x00000000 0x00000002 1

Member rbh rbh_cnt

Eth3/12 0x000000ff 0x08

CBL Check States: Ingress: Enabled; Egress: Enabled

VLAN| BD| BD-St | CBL St & Direction:

--------------------------------------------------

1 | 0x11 | INCLUDE_IF_IN_BD | FORWARDING (Both)

Member info

------------------

Type LTL

----------------------

PORT_CHANNEL Po110

FLOOD_W_FPOE 0x8011

N7004-Berlin# show vlan internal bd-info

bd-to-vlan 17

VDC Id BD Id Vlan Id

------ ------- -------

2 17 1

How to convert a

BD (in dec)

to a VLAN ID

11h = 17 STP ingress/egress


Layer 2 Internal Indices Table

PIXM

000bh

E 3/12

0402h

8011h PO110

10C7h

10C8h SUP

LTL setup (here) for SUP-2 and NX-OS 6.2(5.41)

2

N7004-London# show system internal pixm info ltl-region

===========================================================

PIXM VDC 1 LTL MAP Version: 2

Description: LTL Map for N7K SUP2 Silverstone (all flavors)

===========================================================

LTL_TYPE SIZE START END

========================================================================

LIBLTLMAP_LTL_TYPE_PHY_PORT 1024 0x0 0x3ff

LIBLTLMAP_LTL_TYPE_PC 3204 0x400 0x1083

LIBLTLMAP_LTL_TYPE_SUP_FUTURE 67 0x1084 0x10c6

LIBLTLMAP_LTL_TYPE_SUP_ETH_INBAND 64 0x10c7 0x1106

-------------------------------------------------------------------

SUB-TYPE LTL

-------------------------------------------------------------------

LIBLTLMAP_LTL_TYPE_SUP_INBAND_HQ 0x10c7

LIBLTLMAP_LTL_TYPE_SUP_INBAND_LQ 0x10c8

<SNIP>


Layer 2 STP

STP

STP

STP

root

Config

BPDU

DP

DP := Designated Port

RP := Root Port

BPDU := Bridge Protocol

Data Unit

RP TCN

BPDU

Know your port states in a stable condition

(:= before the troubleshooting, prepare yourself)

Two BPDU types: Configuration BPDU’s and TCN BPDU’s

2

Tracking Port Role Changes, Root Changes via SYSLOG

For vPC with peer switch

configuration both devices

are sending BPDUs as root.

NX-OS 4.2(6), 5.0(2a)

logging level spanning-tree 6

%STP-6-PORT_ROLE: Port Ethernet2/1 instance VLAN0001 role changed to designate

12:15


Layer 2 Spanning Tree, Data Loop

STP

2

Symptoms for a Data Loop

High link utilization (100%)

High CPU and fabric traffic utilization

Constant MAC Address re-learning and flapping

Exessive output drops on an interface

N7004-Berlin# show interface e 3/7 | i rate

30 seconds input rate 24 bits/sec, 0 packets/sec

30 seconds output rate 304 bits/sec, 0 packets/sec

300 seconds input rate 104 bits/sec, 0 packets/sec

300 seconds output rate 424 bits/sec, 0 packets/sec

Verify each switch on the redundant path

Someone who is supposed to block is forwarding...

No loop in my lab

today…

In the real world

we see loops

created by blade

servers, teaming-

nic’s and

hypervisors

(:= virtual swiches)


Layer 2 Spanning Tree, Data Loop

STP

2

Verifying systematically the path

N7004-Berlin# show spanning-tree interface ethernet 3/7 detail

Port 391 (Ethernet3/7) of VLAN0042 is designated forwarding

<SNIP>

BPDU: sent 1972, received 5

N7004-Paris# show spanning-tree interface ethernet 4/2 detail

Port 514 (Ethernet4/2) of VLAN0042 is root forwarding

<SNIP>

BPDU: sent 5, received 2007

N7004-Berlin# show system internal pktmgr interface ethernet 3/7

Ethernet3/7, ordinal: 80 Hash_type: 2

SUP-traffic statistics: (sent/received)

Packets: 2217 / 82

Bytes: 139163 / 17376



Average packet rates(1min/5min/15min/EWMA):

Packet statistics:

Tx: Unicast 0, Multicast 2217

<SNIP>

Paris

Berlin

VID 42

Moscow

London

E4/17

STP

STP

pktmgr

pktmgr

Ethanaylzer

Ethanaylzer

ELAME


STP

75

Layer 2 Spanning Tree

STP

2

N7004-London(config-if)# spanning-tree port type edge

Warning: Edge port type (portfast) should only be enabled on ports

connected to a single

host. Connecting hubs, concentrators, switches, bridges, etc... to this

interface when edge port type (portfast) is enabled, can cause temporary

bridging loops. Use with CAUTION

N7004-London(config-if)# show spanning-tree vlan 1 detail

VLAN0001 is executing the rstp compatible Spanning Tree protocol

Bridge Identifier has priority 32768, sysid 1, address 4055.390f.5643

Configured hello time 2, max age 20, forward delay 15

Current root has priority 32769, address 000c.308b.a040

Root port is 4195 (port-channel100), cost of root path is 2

Topology change flag not set, detected flag not set

Number of topology changes 2 last change occurred 0:15:50 ago

from port-channel100

<SNIP>

What is our STP role? Are we stable? TCN send or received?

If yes through which Interface did we received last TCN?

In case of an access port enable port-fast

STP

DP

RP

TCN

BPDU


Layer 2 STP event history

STP

STP

2

N7004-London(config-if)# sh spanning-tree internal event-history tree 1 interface

port-channel 110

VDC03 VLAN0001 <port-channel110>

0) Transition at 795697 usecs after Sat Dec 14 21:20:53 2013

State: DIS Role: Unkw Age: 0 Inc: no [STP_PORT_EV_UP]

<SNIP>

5) Transition at 800886 usecs after Sat Dec 14 21:20:53 2013

State: FWD Role: Root Age: 0 Inc: no [STP_PORT_ROLE_CHANGE]

Looking back in time for STP: Event-History

800886 us – 795697 us

= 5189 us ~ 5.2 ms

12

3

6


Layer 2 Summary

Flight Recorder Show tech vlan

Show tech stp

Show tech lacp

Show tech l2fm

Show tech forwarding l2 unicast

2 MAC Table

PIXM

L2FM

STP

Always look for

the detail option

and use it

12:20

Chapter 3 Data-Plane Layer 3

3

uRIB

Ex SPAN

LC

Control-Plane


Layer 3 Unicast Routing Architecture

3 Areas to verify

FIB

Manager

uFDM

uRIB

OSPF

route adj

IS-IS RIP IP BGP

mRIB • RIB fully resolved and used for

packets originated by the control plane

Is control plane state

as expected

(route exists, points to

expected next hop)?

Is control plane stable?

Is control plane

consistent with data

plane

(route programmed in

forwarding plane,

consistent with control

plane)?

Data-Plane

Control-Plane

Forwarding Hardware

• Neighbor management

• Protocol database

• Add/Delete prefixes

• Translate routes to

hardware format

• Program hardware

forwarding engine

• Push routes to platform

• Route download

Control-Plane

Data-Plane


Layer 3 Unicast

L3

Paris

42.42.42.4

Ip ospf-42

42.42.42.142

11.0.0.1/32

VID = 42

3

N7004-Paris# show ip ospf 42 internal txlist urib

ospf 42

ospf process tag 42

ospf process instance number 1

ospf process uuid 1090519321

ospf process linux pid 7746

<SNIP>

OSPFv2->URIB transmit list: version 0x10

85

N7004-Paris# show processes cpu sort |i PID|7746

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

7746 10450 502752 0 0.00% 0.01% 0.01% - ospf

uRIB route adj

ARP 5

13: 42.42.42.0/24

14: 11.0.0.1/32

15: 10.0.2.0/24

16: 10.0.4.0/24

16: RIB marker

OSPF-42

SAP 320

Assumption: Control-Plane is stable, OSPF receives LSAs we look at the flow of information from OSFP to HW, if not good check a) configuration (hidden slides before this one)

b) look for Control Plane

Control-Plane


Layer 3 Unicast

L3

uRIB

OSFP

route adj

N7004-Paris# show ip route ospf-42 detail

<SNIP

255.255.255.255/32, ubest/mbest: 1/0

*via sup-eth1, [0/0], 01:59:22, broadcast

11.0.0.1/32, ubest/mbest: 1/0

*via 42.42.42.142, Vlan42, [110/41], 01:57:18, ospf-42, inter

N7004-Paris# sh ip arp 42.42.42.142

<SNIP>

IP ARP Table

Total number of entries: 1

Address Age MAC Address Interface

42.42.42.142 00:03:39 0010.7be8.53b0 Vlan42

OSPF Routes in URIB

Administrative distance assigned

Is there a route to the destination ?

Do we have a resolved

Layer 2 address?

N7004-Paris# sh ip ospf 42 route

<SNIP>

11.0.0.1/32 (inter)(R) area 0.0.0.0

via 42.42.42.142/Vlan42 , cost 41 distance 110

(D) route is directly attached

(R) route is in RIB

Control-Plane


uFDM

uRIB

client

route adj

87

Layer 3 Unicast

L3

Forwarding Hardware

FIB

Manager

3

N7004-Paris# show forwarding ipv4 route 11.0.0.1 module 4

IPv4 routes for table default/base

------------------+------------------+----------------------+-----------------

Prefix | Next-hop | Interface | Labels

------------------+------------------+----------------------+-----------------

11.0.0.1/32 42.42.42.142 Vlan42

N7004-Paris# show forwarding adjacency 42.42.42.142 module 4

IPv4 adjacency information

next-hop rewrite info interface

-------------- --------------- -------------

42.42.42.142 0010.7be8.53b0 Vlan42

N7004-Paris# show ip arp 42.42.42.142


42.42.42.142 00:08:56 0010.7be8.53b0 Vlan42

Is adjacency

consistent with ARP

In the control plane?

Verifying on the ingress line card

Hardware forwarding (FIB)

information on per-module basis

Displays hardware adjacency

table information

Data-Plane

12:25


uFDM

uRIB

client

route adj

Layer 3 Unicast

L3

Forwarding Hardware

FIB

Manager

Verifying on the ingress line card

3

N7004-Paris# show system internal forwarding route 11.0.0.1 module 4 detail

RPF Flags legend:

S - Directly attached route (S_Star)

V - RPF valid

M - SMAC IP check enabled

G - SGT valid

E - RPF External table valid

11.0.0.1/32 , Vlan42 , No of paths: 1

Dev: 1 , Idx: 0x2603 , RPF Flags: V , DGT: 0 , VPN: 7

RPF_Intf_5: Vlan42 (0x35 )

AdjIdx: 0xa038 , LIFB: 0 , LIF: Vlan42 (0x35 ), DI: 0x0

DMAC: 0010.7be8.53b0 SMAC: 4055.390f.5644

N7004-Paris# show system internal forwarding adjacency entry 0xa038 module 4

Device: 1 Index: 0xa038 dmac: 0010.7be8.53b0 smac: 0055.390f.5644 e-vpn: 7

e-lif: 0x35 packets: 0 bytes: 0

88

Data-Plane


Layer 3 Unicast

Verificaton Location L2/L3 reachability for multicast and max. unicast

3

N7004-Paris# ping multicast 224.0.0.5 interface vlan 42

PING 224.0.0.5 (224.0.0.5): 56 data bytes

64 bytes from 42.42.42.5: icmp_seq=0 ttl=254 time=0.836 ms



<SNIP>




<SNIP>

N7004-Paris# ping 42.42.42.142 packet-size 1472

PING 42.42.42.142 (42.42.42.142): 1472 data bytes




<SNIP>

Why not 1500?

1500 – 20 (IP) -8 (ICMP) = 1472

Ethanalyzer ELAME Debug Better alternatives

OSFP

Debug

Ethanalyzer

ELAME

CoPP

RL

ICMP

Data-Plane

Q


Layer 3 Inconsistency

Example

3

N7K# test forwarding inconsistency

N7K# show forwarding inconsistency

IPV4 Consistency check : table_id(0x13) Execution time : 14327 ms ()

No inconsistent adjacencies.

Inconsistent routes:

1. slot(1), vrf(default), prefix (172.31.38.6/32), Route extra in FIB Software

2. slot(1), vrf(default), prefix (172.31.38.2/32), Route extra in FIB Software

Test for inconsistency

Data-Plane

N7K# show ip route 172.18.144.2 IP Route Table for VRF "default"

<SNIP>

172.18.144.0/24, ubest/mbest: 1/0

*via 172.31.38.2, [200/0], 1d22h, bgp-65000, internal, tag 64949

N7K# show ip fib route 172.18.144.2

<SNIP>

------------------+------------------+----------------------+--------

Prefix | Next-hop | Interface | Labels

------------------+------------------+----------------------+---------

*172.18.144.0/24 0.0.0.0 Null0

How can we recover?

(show forwarding ipv4

route 172.18.144.2

module 1)

FIB

Manager

uRIB route

12:30

IDS

91


Layer 3 Internal Security Check

Security Check

This checking drops

various ‘illegal’ packets

These drops can be

also seen in show hardware internal

errors but there they

might look a bit more

cryptic

The checks can be

disabled via ‘hardware

ip verify …’ – in

default VDC (for all

VDCs)

3

N7004-Paris# show hardware forwarding ip verify module 4

IPv4 IDS Checks Status Packets Failed

-----------------------------+---------+------------------

address source broadcast Enabled 0

address source multicast Enabled 0

address destination zero Enabled 0

address identical Disabled --

address reserved Disabled --

address class-e Disabled --

checksum Enabled 0

protocol Enabled 0

fragment Disabled --

length minimum Enabled 0

length consistent Enabled 0

length maximum max-frag Enabled 0

length maximum udp Disabled --

length maximum max-tcp Enabled 0

tcp flags Disabled --

tcp tiny-frag Enabled 0

version Enabled 0

<SNIP>

IDS and how do we identify the source or sender?

Data-Plane


Layer 3 & Tools Exception SPAN

Examples 1

Forwarding Engine

Line Card

DI := SUP

DI := drop

Exception

Redirect

Table

SPAN Engine

ERSPAN

SPAN

E 3/37

DI := SUP

DI := drop

Use inband SPAN

- MTU failures

- TTL errors

- ICMP redirect

Use exception SPAN

- IP Option fail

- IP check

- RPF

- Unsupported RW

3

N7004-Berlin(config)# monitor session 1

N7004-Berlin(config-monitor)# source interface sup-eth 0 both

or

N7004-Berlin(config-monitor)# source exception [layer 3|fabricp | other | all]

Destination Index

:= Drop can be

changed to SPAN

Engine

Data-Plane


Data-Plane Layer 3

Flight Recorder show tech-support routing ip unicast

show tech-support ip

Show tech-support ospf

show tech forwarding l3 unicast mod 4

Show tech eltm

Show tech ethpm

3

uRIB

Ex SPAN

LC Data-Path

L2|L3|OVT|MPLS Unicast|Multicast

Chapter 4: Control Plane

4

Inband Concept

Trigger

CoPP

Netstack RL

Inband

12:35


Architecture Inband Path

Two Tasks

Looking for dropped

packets which are

targeted for the

Control Plane

Management

Port

1G 10G

PID

Multiple

CPU

Cores

Inband

Forwarding

Engine

CoPP

RL

OSPF SUP Line Card

System

Controller

High CPU due to:

Punted traffic

ACL processing

Control Plane tasks

Indentifying from

where/what is being

send from/to the

CPU

Ethanalyzer

Kernel

ELAME

Reference Point 2

Reference Point 1


Control-Plane High Inband traffic (CoPP customizing slides hidden for reference)

OSPF

192.251.19.22

Syslog messages report

OSPF neighbor failures

4

40.9.0.0

2011 Mar 26 15:38:56.395 N7K-1-VDC2 %OSPF-5-NBRSTATE: ospf-6467 [3981]

Process 6467, Nbr 192.251.19.22 on Vlan19 from INIT to DOWN, DEADTIME


Process 6467, Nbr 192.251.19.22 on Vlan19 from DOWN to INIT, HELLORCVD


Process 6467, Nbr 192.251.19.22 on Vlan19 from INIT to DOWN, DEADTIME


Process 6467, Nbr 192.251.19.22 on Vlan19 from DOWN to INIT, HELLORCVD

The trigger or why you start looking:


99

Control-Plane Stable environment?

L3 Resources

How long since the

route was added?

How long since ARP

has been updated?

How long have

adjacency stayed up?

Can we find previous

incarnations of

adjacency here?

Log of recent routing

changes (can filter out

prefix in question)?

4

N7004-Paris# show ip route 11.0.0.1

<SNIP>

11.0.0.1/32, ubest/mbest: 1/0

*via 42.42.42.142, Vlan42, [110/41], 02:50:20, ospf-42, inter

N7004-Paris# show ip arp 42.42.42.142



42.42.42.142 00:01:29 0010.7be8.53b0 Vlan42

N7004-Paris# show ip ospf neighbors

OSPF Process ID 42 VRF default

Total number of neighbors: 2

Neighbor ID Pri State Up Time Address Interface

42.0.0.5 1 FULL/BDR 02:52:48 42.42.42.5 Vlan42

200.0.0.10 1 FULL/DR 02:51:44 42.42.42.142 Vlan42

Are we stable?


Control-Plane Protocol Flapping

Failure Domain

Determine with

Etheranalzer the

failure domain

From Prozess point

of view:

Do I get enough?

Do I get too much?

Ingress MAC Drops?

Ethanalyzer

HWRL Drops?

CoPP Drops?

Inband Drops or FC?

Packet Manager?

IPv4/IPv6

ARP/AM

uRIB

Line Card

ELAME

OSFP

Do we receive the

packet?

Do we receive the

packet (e.g. BPDU

or LSA at the CPU?

CPU?

MEM?

We verified on the other

side we are sending

LDP, BGP, OSPF, …

One real world example

in session 5 (ARP)

12:40


Control-Plane Too much inband traffic

OSPF

192.251.19.22

Syslog messages report

OSPF neighbor failures

CPU states high utilization

caused by OSPF and

Netstack process

4

40.9.0.0

Here two processes OSPF and NETSTACK

are using most resources.

How much do they use usually?

How does my base line look like?

N7K-1-VDC2# show system resources

Load average: 1 minute: 2.92 5 minutes: 2.38 15 minutes: 2.27

Processes : 1267 total, 4 running

CPU states : 34.0% user, 42.5% kernel, 23.5% idle

Memory usage: 4115232K total, 3638780K used, 476452K free

N7K-1-VDC2# show processes cpu sort

PID Runtime(ms) Invoked uSecs 1Sec Process

----- ----------- -------- ----- ------ -----------

3981 127 276 462 43.2% ospf

3841 267 78 3427 16.4% netstack

2941 34146488 7377876 4628 0.9% platform

3982 118 245 485 0.9% ospfv3

(CoPP customizing slides hidden for reference)

+ statistics per Core for SUP-

2/SUP-2E and with newer NX-

OS for SUP-1


Environment

Control-Plane You can Customize CoPP but don not turn it off

40.9.0.0/16

OSPFv2

224.0.0.5

224.0.0.6

Module 1

CoPP

N7K-1# show policy-map interface control-plane module 1

class copp-system-class-ospf-test

control Plane

service-policy input: copp-system-policy

class-map copp-system-class-ospf-test (match-any)

match access-grp name copp-system-acl-malicious

police cir 100 bps , bc 200 ms

module 1 :

conformed 0 bytes; action: drop

violated 0 bytes; action: drop

N7K-1# show policy-map interface control-plane module 2

class copp-system-class-ospf-test

control Plane

service-policy input: copp-system-policy

class-map copp-system-class-ospf-test (match-any)

match access-grp name copp-system-acl-ospf-test

police cir 100 bps , bc 200 ms

module 2 :

conformed 0 bytes; action: drop

violated 1799505072 bytes; action: drop

Module 2

CoPP

Generic: show policy-map interface control-plane you determine the

affected class, and with N7K# show class-map type control-plane you determine what is classified for those classes.

4


RL

Control-Plane Next to CoPP we restrict some traffic via RL (AKA HWRL)

As with CoPP policers,

modifying the default rates

should be carefully planned

before any configuration

changes.

Rate-limiters can prevent overwhelming the control-plane

CoPP

RL

Multiple

CPU

Cores

4

N7004-Berlin# show hardware rate-limiter

Units for Config: packets per second

Allowed, Dropped & Total: aggregated since last clear counters

Module: 3

R-L Class Config Allowed Dropped Total

+----------------+--------+-------------+-------------+----------------+

L3 mtu 500 0 0 0

L3 ttl 500 0 0 0

L3 control 10000 0 0 0

L3 glean 100 0 0 0

<SNIP>

L2 storm-ctrl Disable

access-list-log 100 0 0 0

copy 30000 1423 0 1423

receive 30000 8540 0 8540

L2 port-sec 500 0 0 0

L2 mcast-snoop 10000 2 0 2

<SNIP>


Inband

Control-Plane Inband

SUP-2 / NX-OS 6.2 (5.41)

B

P

D

U

Q0 Q1

Clipper

R2D2

CPU

4

BDR-529-Berlin# show system inband queuing status

Weighted Round Robin Algorithm

Weights BPDU - 64, Q0 - 16, Q1 – 4

BDR-529-Berlin# show system inband queuing statistics

Inband packets unmapped to a queue: 0

Inband packets mapped to bpdu queue: 2078

Inband packets mapped to q0: 1339

Inband packets mapped to q1: 4

In KLM packets mapped to bpdu: 0

In KLM packets mapped to arp : 0

In KLM packets mapped to q0 : 0

In KLM packets mapped to q1 : 0

In KLM packets mapped to veobc : 0

Inband Queues:

bpdu: recv 2078, drop 0, congested 0 rcvbuf 2097152, sndbuf 4194304 no drop 1

(q0): recv 1339, drop 0, congested 0 rcvbuf 2097152, sndbuf 4194304 no drop 0

(q1): recv 4, drop 0, congested 0 rcvbuf 2097152, sndbuf 4194304 no drop 0


Inband

Control-Plane Inband

CPU

4

N7004# show hardware internal cpu-mac inband events

1) Event:TX_PPS_MAX, length:4, at 382147 usecs after Fri Jan 10 20:04:37

2014 new maximum = 191

2) Event:RX_PPS_MAX, length:4, at 382147 usecs after Fri Jan 10 20:04:37

2014 new maximum = 195

How to determine the max pps rate to/from the CPU, if we run out of

buffer and it’s occurrence

How to determine the time of the max pps rate to correlate against

your logs?

N7004-Berlin# show hardware internal cpu-mac inband stats | in rate|buffer

Rx no buffers .................. 0

Packet rate limit ........... 64000 pps

Rx packet rate (current/max) 85 / 195 pps

Tx packet rate (current/max) 85 / 191 pps

Goal: Compare against logs Possible next step: logw.py

α

tα

α

12:45


Control-Plane Software Architecture

NX-OS

Packet

Manager

NetStack

IP

Clients

NetStack VDC -1

L3

L2

„ip input“

ARP

OSPF System

manager

OSPF ARP

System Manager starts

and controls / monitors

If the heatbeat fails

core sig6 -> system

troubleshooting N7004-Berlin# debug pktmgr frame

2014 Jan 10 20:14:40.061027 pktmgr: In 0x0800 82

7 4055.390f.5645 -> 0100.5e00.0005 Eth3/6

STP

BGP

Clients

Ethanalyzer ELAME Debug

Packet

Manager

NetStack

IP

NetStack VDC-2

L3

L2

„ip input“

System


Control-Plane Flight Recorder

Data Collection show tech-support sysmgr (depreciated)

Show tech-support ha

show tech-support netstack detail

show tech-support pktmgr

show tech-support <service>

4

Inband Concept

Trigger

CoPP

Netstack RL

Inband

Chapter 5 Control Plane ARP

5 ARP glean


Control-Plane Address Resolution Protocol and Adjacency Manager

Layer 2/3 ARP Incomplete...

5

N7004-Berlin# show ip arp

Flags: * - Adjacencies learnt on non-active FHRP router

+ - Adjacencies synced via CFSoE

# - Adjacencies Throttled for Glean

D - Static Adjacencies attached to down interface

IP ARP Table for context default



IP ARP Table for context default



192.168.0.3 00:04:41 4055.390f.5643 Vlan1

10.0.3.5 00:06:35 4055.390f.5645 Ethernet3/6

10.0.2.4 00:07:14 4055.390f.5644 Ethernet3/8

20.0.0.13 00:00:14 INCOMPLETE Ethernet3/14

192.168.0.254 - 0000.0c9f.f001 Vlan1

E 3/13 E 3/14

20.0.0.0/24

.13 .14

VRF

uRIB

(253) route adj AM ARP



5

N7004-Berlin# show ip arp statistics ethernet 3/14

ARP packet statistics for interface: Ethernet3/14

Sent:

Total 10, Requests 9, Replies 0, Requests on L2 0, Replies on L2 0,

Gratuitous 1, Tunneled 0, Dropped 0

Send packet drops details:

MBUF operation failed : 0

Context not yet created : 0

Invalid context : 0

Invalid ifindex : 0

Invalid SRC IP : 0

Invalid DEST IP : 0

Destination is our own IP : 0

Unattached IP : 0

<SNIP>

E 3/13

20.0.0.0/24

.13 .14

VRF ARP

E 3/14

N7004-Berlin# debug ip arp packet

2014 Jan 5 21:51:40.477507 arp: (context 1) Sending packet on interface Ethernet3/14, (prty 0) Hrd

type 1 Prot type 800 Hrd len 6 Prot len 4 OP 1, Pkt size 28

2014 Jan 5 21:51:40.477629 arp: Src 4055.390f.5642/20.0.0.14 Dst ffff.ffff.ffff/20.0.0.13

2014 Jan 5 21:51:40.481061 arp: (context 4) Receiving packet from interface Ethernet3/13, (prty 6)

Hrd type 1 Prot type 800 Hrd len 6 Prot len 4 OP 1, Pkt size 46

2014 Jan 5 21:51:40.481131 arp: Src 4055.390f.5642/20.0.0.14 Dst ffff.ffff.ffff/20.0.0.13

Consider the use of Debug-Filter and send to a file

12:50



5

Received:

Total 1, Requests 0, Replies 0, Requests on L2 0, Replies on L2 0

Proxy arp 0, Local-Proxy arp 0, Tunneled 0, Fastpath 0, Snooped 0, Dropped 1

Received packet drops details:

Appeared on a wrong interface : 0

Incorrect length : 0

Invalid protocol packet : 0

Invalid context : 0

Context not yet created : 0

Invalid layer 2 address length : 0

Invalid layer 3 address length : 0

Invalid source IP address : 0

Source IP address is our own : 0

No mem to create per intf structure : 0

Source address mismatch with subnet : 0

Directed broadcast source : 0

<SMIP>

E 3/13

20.0.0.0/24

.13 .14

VRF ARP

N7004-Berlin# show ip arp statistics vrf ALQ

<SNIP>

Received:

Total 13, Requests 0, Replies 0, Requests on L2 0, Replies on L2 0

Proxy arp 0, Local-Proxy arp 0, Tunneled 0, Fastpath 0, Snooped 0, Dropped 13

<SNIP>

Invalid source MAC address : 0

Source MAC address is our own : 13

<SMIP>

N7004-Berlin# show ip arp statistics ethernet 3/14



5

Check CoPP and/or HWRL:

Customer# show class-map type control-plane copp-system-p-class-normal

class-map type control-plane match-any copp-system-p-class-normal

match access-group name copp-system-p-acl-mac-dot1x

match exception ip multicast directly-connected-sources

match exception ipv6 multicast directly-connected-sources

match protocol arp

class-map copp-system-p-class-normal (match-any)

violate action: drop

module 5: violated 20557632224 bytes,

5-min violate rate 4154397 bytes/sec

module 9: violated 0 bytes,

5-min violate rate 0 bytes/sec

Customer# show hardware rate-limiter | i Module|R-L|glean

Module: 5

R-L Class Config Allowed Dropped Total

+------------------+--------+---------------+-------------+-----------------+

L3 glean 100 4904 2935 7839

L3 glean-fast 100 863401 1539316 2402717

SWT-1 SWT-2

ARP INCOMPLETE

It worked before

no new deployment

Ethanalyzer verifies ARP

packets are being send

by SWT-1 but not

received

On SWT-2 ARP is being

Received and Send

customer# show vpc brief

vPC keep-alive status : peer is not reachable through peer-keepalive


Control-Plane Flight Recorder

Data Collection

show tech-support arp

show tech-support adjmgr

show tech-support hsrp

5 ARP glean

Summary


Inband

127

Nexus 7000 Troubleshooting N7K offers complete visibility, excessive logging and integrates great testing tools

Supervisor

(Control-Plane)

Fabric

I/O Module

(Forwarding

Engine)

Data

Plane

Data Center OS

NX-OS

I/O Module

(Forwarding

Engine)

ELAME

Control Plane

Management Plane ethanalyzer

PI := Platform Independent

PD := Platform Dependent

RL CoPP

Netstack

ASIC counters

ASIC counters

show

debug/filter

logging

ASIC counters

12:55


Call to Action…

Visit the World of Solutions:-

Cisco Campus

Walk-in Labs

Technical Solutions Clinics

Meet the Engineer

Lunch Time Table Topics, held in the main Catering Hall

Recommended Reading: For reading material and further resources for this session, please visit www.pearson-books.com/CLMilan2014

128

http://www.pearson-books.com/CLMilan2014




Complete your online session evaluation

Complete four session evaluations and the overall conference evaluation to receive your Cisco Live T-shirt

Complete Your Online Session Evaluation

129


130

Offering The Rides via WebEx

1

2

3

4

5

6

Strategy

Tools & System

Data-Plane

Layer 2

[TCAM]

Optional

Data-Plane

Layer 3

Control-Plane

Inband

Control-Plane

ARP

1 25

20

12

12

8

1st WebEx 120 min for YOU

12-FEB-14 10:00 CET opt-in:

[email protected]

“subject BRKDCT-3144”

11:35

mailto:[email protected]

Chapter 6: ACL’s Optional

6

TCAM


TCAM Ternary Content Addressable Memory

Result Types

6

N7004-London# show system internal access-list feature bank map vlan ingress

<SNIP>

slot 3

=======

_________________________________________________________________________

Feature Rslt Type T0B0 T0B1 T1B0 T1B1

_________________________________________________________________________

QoS Qos X

RACL Acl X

PBR Acl X

VACL Acl X

DHCP Acl X

ARP Acl X

Netflow Acl X X

Netflow (SVI) Acl X X

Netflow Sampler Acc X

Netflow Sampler (SVI) Acc X

SPM WCCP Acl X X

BFD Acl X

SPM OTV Acl X

ACLMGR ERSPAN (source) Acl X

Per bank only one

result type can be

used:

for “VLAN” & “Ingress”

either QoS or NF

Sampler

“I can’t configure QoS…the system

rejects my configuration…”

T0B0

T0B1

T1B0

T1B1


6

TCAM

TCAM Statistics per Entry & Logging

N7K# sh ip access example

IP access list example

statistics per-entry

10 permit ip any 10.1.2.100/32 [match=3452]

20 deny ip any 10.1.68.101/32 [match=49920]

30 deny ip any 10.33.2.25/32 [match=232324]

40 permit tcp any any eq 22 [match=9881]

50 deny tcp any any eq telnet [match=442]

60 deny udp any any eq syslog [match=87112]

70 permit tcp any any eq www [match=4345667]

80 permit udp any any eq snmp [match=234222]

ACL logging is enabled by including the log

keyword in an ACL rule (show log log).

The Sup receives a copy of the packet. The

original packet is forwarded/dropped in

hardware with no performance penalty.

Statistics per Entry

The CPU is protected by using one of the available rate limiters. Forwarding engine hardware enforces rate to avoid saturating inband interface CPU. hardware rate-limit access-list-log command adjusts rate (def 100 pps)

ACL Logging can be a useful tool during troubleshooting. Use ACL logging to sample specific packets from data plane.Use onboard ethanalyzer (wireshark) to analyze sampled packets


Statistics per entry results in no optimization and no merge

activity. Instead a 1:1 mapping of configured ACE to CL

TCAM will be seen

6

TCAM Space „...when using ACL stats per entry on the 7K the TCAM

utilization goes up to 47%, when removed, it dropped to 7%...“

object groups do NOT offer ANY optimization in terms of

CL (:= Classification) TCAM utilization

Statistics per entry are often used for troubleshooting with

„host ACEs“

TCAM Utilization

ACLs Statistics are

NOT enabled by default

(fundamental difference

vs. IOS) because they

require the ACEs NOT

to be merged and this

affects the TCAM

utilization.


TCAM Utilization

ACL’s

6

N7004(config)# show hardware access-list vdc 3 input statistics module 3

VDC-3 Ethernet3/30 :

====================

INSTANCE 0x7 / Tcam 1 resource usage:

----------------------

Label_b = 0x2

Bank 0

------

IPv4 Class

Policies: BFD() [Merged]

Netflow profile: 0

Netflow deny profile: 0

Entries:

[Index] Entry [Stats]

---------------------

[0058:000e:000e] prec 1 redirect(0x64)-routed udp 0.0.0.0/0 0.0.0.0/0 eq 3784

ttl eq 255 flow-label 3784 [1891]

[0059:000f:000f] prec 1 redirect(0x64)-routed udp 0.0.0.0/0 0.0.0.0/0 eq 3785

ttl eq 254 flow-label 3785 [63895]

[005a:0010:0010] prec 1 permit-routed ip 0.0.0.0/0 0.0.0.0/0 [128276]

<SNIP>

Specific applications

(dhcp, bfd) may install

their own ACLs which

must merge with user

configured

racl,vacl,pacl

3/30

L3

BFD

London FE7


TCAM Bank Management

TCAM

6

N7004(config)# hardware access-list resource feature bank-mapping

N7004(config)# show system internal access-list feature bank-class map ingress

slot 3

=======

Feature Class Definition:

0. CLASS_QOS :

QoS,

1. CLASS_INBAND :

Tunnel Decap, SPM LISP,

2. CLASS_PACL :

PACL, Netflow,

3. CLASS_DHCP :

DHCP, Netflow, Netflow (vlan), ARP,

4. CLASS_RACL :

RACL, RACL_STAT, Netflow (SVI), ARP,

<SNIP>

Feature Class Combination (Ingress)

0. CLASS_PACL, CLASS_QOS_INTF, CLASS_EMPTY, CLASS_EMPTY

1. CLASS_PACL, CLASS_NF_SMPL_INTF, CLASS_EMPTY, CLASS_EMPTY

<SNIP>

33. CLASS_EMPTY, CLASS_EMPTY, CLASS_NF_SMPL, CLASS_QOS

“now I can configure

QoS and NF Sampler”


TCAM Utilization

Flight Recorder show tech-support aclmgr

show tech-support aclqos

6

TCAM

brkdct-3144

Documents