cisco asr 9000 architecture - andovercg.com · cisco asr 9000 architecture dennis cai, erbu...

BRKARC-2003

Cisco ASR 9000 Architecture

Dennis Cai, ERBU Technical Marketing Architect

CCIE # 6621

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 2

Session Goal

To provide you with a thorough understanding of the Cisco ASR 9000 Router architecture, RSP, fabric, and line card design, packet flows, and key forwarding engine functions and features

This session will not examine baseline IOS-XR, for example, IOS-XR control plane and management plane protection, modular OS design, software package, SMU installation, IOS-XR routing configuration, etc. Please refer to TECARC-2001 (Cisco IOS-XR - Cisco Next Generation Operating System) for IOS-XR related technical information

2


Building Carrier Ethernet Services Using Cisco Ethernet Virtual Circuit Framework - BRKSPG-2204 (EVC SW Infrastructure)

Redundancy Mechanisms for Carrier Ethernet and Layer 2 VPN Services -BRKSPG-2207 (L2VPN Resiliency)

Deploying Next Generation Carrier Ethernet: Services, Architectures and Operations - TECOPT-2100 (Carrier Ethernet design)

End-to-End Video Architecture and Design Part 1: Video Fundamentals, HeadendDesign, and Video Optimized Transport - TECSPV-1001 (Video solution)

End-to-End Video Architecture and Design: Part 2 - Access Networks and Video Service Delivery - TECSPV-1002 (Video solution)

Next Generation Assurance for IP Video Delivery Networks - BRKSPV-2111(video inline morning and management)

Deployment challenges with Interconnecting Data Centers - BRKDCT-3060 (Data center interconnect)

And more …

Related Sessions

http://vsearch.cisco.com/main.cfm?m=browse.au&auid=18513































































ASR 9000 Functional EvolutionNext-Generation SP Edge and Aggregation, DC Edge

2012

Mark

et F

un

cti

on

ali

ty

High Scale

Ethernet

Subscriber

Awareness

2011

Videoscape & IPv6

Advanced

Services

Available

Layer 2

Carrier

Ethernet

Rich L3 VPN Services

& Legacy Interfaces

IP RAN Backhaul

For Ethernet & TDM

Mobile

Backhaul on

ATM & CEoPS

Application

Service

Integration

Broadcast &

On-Demand

Video Delivery

Data Center

PE &

Interconnect

Network

Virtualization

(nV)


ASR 9000 Portfolio Evolution NxDS0 – 120 Gbps LC

TODAY

# of Slots 10 slots 6 slots

Max. Linecards per

Chassis8 LC + 2 RSP 4 LC + 2 RSP

Max. Linecard Bandwidth 120 Gbps 120 Gbps

Max. Slot Bandwidth 180 Gbps 180 Gbps

Chassis Bandwidth 2.8 Tbs 1.4 Tbs

ASR 9010 ASR 9006

2H 2011

# of Slots 10 slots 6 slots

Max. Linecards per

Chassis8 LC + 2 RSP 4 LC + 2 RSP

Max. Linecard Bandwidth 240 Gbps 240 Gbps

Max. Slot Bandwidth 440 Gbps 440 Gbps

Chassis Bandwidth 6.4 Tbs 3.2 Tbs

1H 2012

# of Slots 22 Slot 10 Slot 6 slots Pizza Box“Iron Man”

9000v

Satellite

Max. Linecards per

Chassis20 LC 8 LC 4 LC 2 IO Slots Fixed Ports

Max. Linecard

Bandwidth360 Gbps 360 Gbps 360 Gbps

Modular +

4x10GE

44xGE +

4x10GE

Max. Slot Bandwidth 1.2 Tbps 440 Gbps 440 Gbps

Chassis Bandwidth 48 Tbps 6.4 Tbps 3.2 Tbps 120 Gbs 80 Gbs

NxDS0 – 240 Gbps LCNxDS0 – 360 Gbps LC

Iron ManASR 9922

ASR 9000v

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 6Foundation for Next-Generation Edge

Increased Service

VelocityQuickly deploy new

services

Multi-dimensional

ScaleSystem and services

scale

ASR 9000 “nV System”

ASR

9000v

ASR 9000

ASR

9000v

nV

Satellite

nV

clusterNetwork

Cloud

Client

Simplify OperationsReduce overall TCO

Integrated A to Z

Management

ASR 9000 Network Evolution – nV TechnologySuper, Simple

http://www.cisco.com/web/solutions/sp/asr9000.html

http://www.cisco.com/web/solutions/sp/asr9000.html


Agenda

Hardware Architecture

‒ Chassis, RSP, Line Cards and Switch Fabric

Software Architecture

‒ Packet Forwarding Overview

‒ L3 Forwarding

‒ L2 Forwarding

‒ Load Balancing (BGP Multi-path, ECMP, Link Bundle)

‒ HA and Fast Convergence

‒ QoS Architecture

Data Plane Troubleshooting


ASR 9010 and ASR 9006 ChassisIdentical HW components across two chassis*

Integrated cable

management

with cover

RSP (0-1)

Line Card

(0-3, 4-7)

System fan trays

Six Modular

Power Supplies

Front-to-

back airflow

System fan trays

Three Modular

Power Supplies

RSP (0-1)

Line Card

(0-3)

Air draw

Side-to-back

airflow

cable

management

* Fan tray is different


Power and CoolingExisting Power Supply and Fan are ready for 400G/slot

2.1/1.5 kW

2.1 kW

3 kW

DC Supplies

AC Supplies

AB

AB

3 kW

A

B

Fans unique to chassis

Variable speed for

ambient temperature variation

Redundant fan-tray

Low noise, NEBS and OSHA compliant

6 & 10 slot use same power supplies

Single power zone

All power supplies run in active mode

Power draw shared evenly

50 Amp DC Input or 16 Amp AC

for Easy CO Install

ASR 9010 Fan Tray

Power Supply

ASR 9006 Fan Tray


Useful CLIs

RP/0/RSP0/CPU0:asr1(admin-config)#power budget reservation standby-rsp disable to

disable power reservation for secondary RSP if only single RSP in the system

RP/0/RSP0/CPU0:asr1(admin-config)#power budget enforcement disable to disable the

power budget enforcement. Be very cautious, if system run out of power under certain condition,

it may cause undeterminstic state

RP/0/RSP0/CPU0:asr1(admin)#show environment power-supply

<snip>

Power Shelves Type: AC

Total Power Capacity: 3000W

Usable Power Capacity: 3000W

Supply Failure Protected Capacity: 0W

Worst Case Power Used: 2200W

Slot Max Watts

---- ---------

0/RSP0/CPU0 235

0/RSP1/CPU0 235

0/0/CPU0 630

0/1/CPU0 350

0/FT0/SP 375

0/FT1/SP 375

Worst Case Power Available: 800W

Supply Protected Capacity Available: Not Protected


Available power is checked when:

–An LC card is inserted

–An LC card is powered up via the CLI

–An LC card is reset via “hw-mod reload”

If the system does not have enough available power to accommodate the LC, then the LC becomes “UNPOWERED”

Installing new power supplies will not automatically power up any UNPOWERED line cards. The user can force a recheck using:

“hw-mod reload loc <>”

RSP and Fan Tray cards are given priority allocation of power budget

LC power budget is checked in numeric order until it is exhausted. The actual power up of the LC's is in parallel, and asynchronous

Power Check and Rules


RSP Engine

Performs control plane and management functions

Dual Core CPU processor with 4GB or 8GB DRAM (4G8G is not field upgradable)

2MB NVRAM, 4GB internal bootdisk, 2 external compact flash slots

Dual Out-of-band 10/100/1000 management interface

Console & auxiliary serial ports

Hard Drive: 70G HDD

Console Port

AUX PortManagement

Ethernet

ALARMBITS

Compact Flash

Slots

Status LED

Status light


Front Panel CPU Complex

RSP Engine Architecture

4/8GB MEM

I/O FPGA

HDD

Mgt Eth

CF card

Console

NVRAM Boot Flash

EOBC/Internal GE switch

EtherSwitch

Timing Domain

ClockTimeFPGA

BITS

4G disk

Mgt Eth

Aux

Alarm

Switch fabric

PuntFPGA Arbitration

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

FIA

CPU

Arbiter


RSP Engine ArchitectureFile system

RP/0/RSP0/CPU0:asr9k#dir ?

/all List all files

/ena Recognize sub directories

/recurse Recursively list subdirectories encountered

WORD file name

bootflash: bootflash: file system boot image

compactflash: compactflash: file system

compactflasha: compactflasha: file system

disk0: disk0: file system XR image, configurations

disk0a: disk0a: file system

disk1: disk1: file system

disk1a: disk1a: file system

harddisk: harddisk: file system SYSLOG, core dump

harddiska: harddiska: file system

harddiskb: harddiskb: file system

location Fully qualified location specification

nvram: nvram: file system ROMMON variables


RSP Operations Impact Fabric ?Guarantee “0” packet loss for RSP failover or OIR

Switch fabric ASIC reside on the RSP blade physically

Switch fabric ASIC is controlled by low level hardware, it operates separately from RSP function

All fabric ASIC run in active mode regardless of the RSP status

RSP SW switch over, reload, crash including kernel crash have NO impact on fabric operation

RSP OIR has no traffic impact due to long/short pin backplane design and instant fabric switch over

–Short pin trig the control signaling for fabric switchover in hardware

–Long pin is used for data packet. It can continue draining the in-flight packets from the fabric during the extended short period of time


Ethernet Line Card FamilyShipping

A9K-40G A9K-4T A9K-8T/4 A9K-2T20G A9K-8T A9K-16T/8

Common HW ASIC and HW architecture

Identical SW features

Line card Memory options for QoS scale


NP0PHY

NP2PHY

NP3PHY

NP1PHY

FIA0

CPU

NP4PHY

NP6PHY

NP7PHY

NP5PHY

FIA1

Example: A9K-8T

B0

NP: Network ProcessorMain forwarding ASIC

L2 & L3 forwarding, features (QoS, ACL, etc),

control plane policing, mcast replication, etc

10Gbps bi-directional with features applied

CPU (same as RSP)Program HW forwarding tables

Distributed Control planesSW switched packets

Inline Netflow

FIA: Fabric Interface ASICProvide non-blocking data connection to switch fabric

Internal system queues/VoQIntelligent mcast replication

B: Bridge FPGAProvide non-blocking data

connection between NP and FIAInternal System queues

Intelligent mcast replication

10G PHY for one or two 10G port, or

10x1G port

Line Card Architecture – Hardware Components

Note, Bridge FPGA provide non-blocking

connection between NP and the FIA. Functionally it

does the HW conversion due to different interface

format on NP and FIA. It‟s part of the switch fabric

connection. To make it logically simple, it will be

removed from the remaining slides.

B1

B0

B1


RP/0/RSP1/CPU0:SJC#show controllers fabric ?

Arbiter Arbitration ASIC show screens.

Crossbar XBAR ASIC show screens.

fia Show command for fabric interface asic

RP/0/RSP1/CPU0:SJC#show controllers fabric fia bridge stats location 0/0/cpu0

RP/0/RSP1/CPU0:SJC#show controllers fabric fia stats location 0/0/cpu0

RP/0/RSP1/CPU0:SJC#show controllers np ?

counters Display contents of global stats counters

crashinfo Display NP Crash info

drvlog Display Driver Logging

fabric-counters XAUI counters dump

interrupts Show NP interrupt data

memory NP Raw Memory Dump

portMap Show port mapping on NP

ports Shows physical ports associated with each np

<snip>

Line Card HW Components – Counters

FIA

NP

Crossbar

Fabric

ASIC

Arbiter

FIA B0

B0


4xNPs Line Card Family

NP0PHY 3

NP2PHY 1

NP3PHY 0

NP1PHY 2

NP0PHY 3

NP2

NP3

NP1

PHY 7

PHY 2

PHY 6

PHY 1

PHY 5

PHY 0

PHY 4

NP0

NP2

NP3

NP1

NP0PHY

NP2

NP3

NP1PHY

A9K-4T-E/B/L

A9K-8T/4-E/B/L

A9K-2T20G-E/B/L

A9K-40G-E/B/L

Oversubscribed line card Up to

60Gbps (~57Gbps) bandwidth

FIA

FIA

FIA

FIA


8xNPs Line Card Family

PHY

PHY

PHY

PHY

PHY

PHY

PHY

NP2

NP1

NP0

NP6

NP5

NP4

NP7

NP3PHY

NP2

NP1

NP0

NP6

NP5

NP4

NP7

NP3

PHYPHY

PHYPHY

PHYPHY

PHYPHY

PHYPHY

PHYPHY

PHYPHY

PHYPHY

A9K-8T-E/B/L A9K-16T/8-B

Oversubscribed line card

Up to 120Gbps (~117Gbps)

bandwidth

FIA0

FIA1

FIA0

FIA1


Line Card Memory OptionsDifferent Queue Scale for Different Deployments

3 memory options for each line card: Extended (or high queue),

Base (medium queue), Low (low queue)*

Different memory option has different QoS queue/policer scale, L2

sub-interface scale and ACL scale

All other system wide scale is the same across different type of

the line cards, including FIB, MAC address, Bridge-domain, L3

sub-interface, VRF, etc

All line cards have the same HW Identical features

Mixed different type of line cards are supported on the same

chassis with same system wide scale and identical features

Low

Medium

High

* A9K-16T/8 only have “B” option


L/B/E Line Cards – What’s the Difference?

-

Forwarding ASIC

STATS MEMORY

FRAME MEMORYLOOKUP

MEMORY

TCAM

FIB MAC

Each NPU has Four Main Associated memories TCAM , Search/Lookup memory , Frame/buffer memory and statistics memory

–TCAM is used for VLAN tag, QoS and ACL classification

–Lookup Memory is used for storing FIB tables, Mac address table and Adjacencies

–Stats memory is used for all interface statistics, forwarding statistics etc

–Frame memory is buffer memory for Queues

E/B/L line card have different TCAM , Stats and Frame Memory size, which give different scale number of the QoS queues and L2 sub-interfaces per line card

Lookup Memory is the same across line card s why?

–To support mix of the line cards without impacting the system wide scale including routing, multicast, MAC address, L3 interface, MPLS label space scale

NP complex

Same lookup memory size across different type of line

cards mix-and-match LCs, same system level scale

Different memory size for different QoS

queuing/policing, L2 EFP and ACL scale


Optics

TenGE – XFP OpticsLR, ZR, & ER

GE - SFP OpticsT, S, L, & Z

Standard, CWDM, & DWDM XFPs/SFPs/SFP+ available

IPoDWDM G.709 FEC/EFEC

Optics official support matrix:


A9K-SIP-700

Powerful and Flexible QFP Processor

• Flexible ucode Architecture for Feature Richness

• L2 + L3 Services: FR, PPP, HDLC, MLPPP, LFI, L3VPN, MPLS, Netflow, 6PE/VPE

Strong QoS

• 128+K Queues,

• 128+K Policers

• H-QoS

• Color Policing

Scalability

• Distributed control and data plane; 20G; 4 Bays

• L3 i/f, route, session, protocol – scaled up for MSE needs

High Availability

• IC-Stateful Switch Over capability,

• MR-APS

• IOS-XR base for high scale and reliability

SIP/SPA – Non-Ethernet Support

SPA support list:

http://www.cisco.com/en/US/partner/prod/collateral/routers/ps9853/data_sheet_c78-573452.html

SPA Support: ‒ChOC-12: T1 / T3, MLPPP, LFI, IC-SSO, MR-APS‒ChOC-3/STM1 + ChOC-12: DS0 / STM1 + ChOC-48: DS3 / E3 / STM4‒POS: OC3/STM1, OC-12/STM4, OC-48/STM16, OC-192/STM64

SPA Roadmap (2011):Ch T1/E1, Ch T3/E3, CEoPs, ATM





Dual-FIA 8xNPs

Linecard

FIA0

FIA1

RSP1

Single-FIA4xNPs

Linecard

Switch Fabric Overview

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

RSP0

4x23Gbps =92Gbps with dual RSP

2x23Gbps=46Gbps with single RSP

Active-active load balancing: Unicast: per-packet load balancing, Multicast: per (S,G) load balancing

Arbiter for fabric access control. Arbiter is in active/standby mode, which is controlled by low level hardware signalling

Frame format over fabric: super-frame, it can aggregate multiple small packet into a big sup-frame to improve the fabric throughput

23Gbps per fabric channel

8x23Gbps =184Gbps with dual RSP

4x23Gbps=92Gbps with single RSP

FIA

Arbiter

Arbiter


Dual-FIA 8xNPs

Linecard

FIA0

FIA1

RSP1

Single-FIA4xNPs

Linecard

Fabric RedundancyGuarantee “0” packet loss during RSP failover and OIR

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

RSP0

Separate data and arbitration(control) paths

All fabric data channels run in active mode for extra fabric bandwidth and instant fabric switch over

Both Arbiters work in parallel – both answer all requests, FIAs follow active Arbiter, FIAs switch to backup if needed instant control path switchover

Arbiter switchover is controlled by low level hardware signalling

FIA

Arbiter

Arbiter


NP0PHY

NP2

NP3

NP1

FIA0

CPU

NP4

NP6

NP7

NP5

FIA1

B0

System Internal BandwidthNO bottleneck inside the system

B1

B0

B1

15Gbps bi-

directional30Gbps and 25M pps

(combined ingress and egress)

PHY

PHY

PHY

PHY

PHY

PHY

PHY

RSP1

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Arbiter

RSP0

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Arbiter

Each FIA has one fabric channel

which is 23 Gbps bi-directional, to

each of the switch fabric ASIC

A9K-16T/8-B

~120Gbps raw bandwidth

A9K-8T/4

~60Gbps raw bandwidth

30Gbps bi-

directional

60Gbps bi-

directional

30Gbps bi-

directional

Use A9K-8T line card as example


Agenda





‒ L3 Forwarding

‒ L2 Forwarding






NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHYFIA

CPUNP0PHY

NP2PHY

NP3PHY

NP1PHY

FIA

CPU

Egress LCIngress LC

Switch Fabric

1

Ingress NP look up Get egress NP information (added into the NP/fabric

header), apply ingress features

Egress NP look up Get egress logical port, VLAN, MAC, ADJ information, etc for packet rewrite, apply egress

features

IOS-XR Two-Stage Packet ForwardingFully Distributed Forwarding on Line Cards

2

Packet is forwarded to the egress NP based on the information in the

NP/fabric header

Each line card has independent AIB only for local interfaces

Each line card has independent Interface DB for local interfaces

Both Ingress and Egress FIB – allows forwarding features to be independently applied on LCs


NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHYFIA

CPU

NP0PHY

NP2PHY

NP3PHY

NP1PHY

FIA

CPU

NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHYFIA

CPU

LC2

LC3

LC1

IGMP joins

Multicast Source

IGMP joins

Switch Fabric

1

FPOE

2

3

MGID

Fabric Replicationreplicate single copy to

each LC which has IGMP join, based on FGID table

in switch fabric

FIA Replication replicate single copy to each NP which

has IGMP join, based on MGID table in FIA

NP Replication replicate single copy per each

receiver based on multicast FIB table

1 2 3

MFIB

Multicast Packet ForwardingIntelligent Multicast Replication

FGID – Fabric Group ID

MGID – Multicast Group ID

MFIB – Multicast Forwarding Information Base

Efficiency

Line rate

Scalable

Simple

MFIB


IGMP joins

NP0

NP2

NP3

NP1

FIAB0

B1

CPUPHY

PHY

PHY

PHY

NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHY

FIA

CPU

NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHY

CPU

LC2

LC3

IGMP joins

Multicast Source

Switch Fabric

1

FGID

Fabric Replicationreplicate single copy to

LCs which receive IGMP join, based on FGID table

in switch fabric

FIA Replication replicate single copy to NPs which

receive IGMP join, based on MGID table in FIA

NP Replication replicate copy per receiver based on

multicast FIB table

1

2

4

Intelligent Multicast ReplicationSwitch Fabric and Egress LC Replication

FGID – Fabric Group ID

MGID – Multicast Group ID

MFIB – Multicast Forwarding Information Base

B0

B1

FIAB0

B1

IGMP joins

Bridge Replication similar as FIA replication, single copy

to NP

3

2

2

3

3

4

MGIDMGID

MFIB1 0000000000

2 0000000001

3 0000000011

…

N 1111001111

hit

FGID=3

FPOE Table

1Kx10 bitmap representing

all combination of the

egress linecards

1 00

2 01

3 10

…

N 10

hitMGIDReplicate to the first

Bridge only

MGID Table in FIA

Each entry represent one

mroute, 2 bits indicating the

down stream ASICs


Initial lookup

Internal

Replication

Engine

Second lookup

From Fabric

To egress

port

NP

Initial Lookup

– Original packet is lookup in the MFIB/L2FIB returning the number of copies to be replicated on the given NPU, each copy per output logical interface

– Packet is sent to a special internal replication engine. HW chips replicates the packets and send the copy to the processing engine

Second Lookup

–Each replicated copy is processed. Egress feature like QoS, ACL is applied at this pass

Intelligent Mulitcast Packet ReplicationInside NP

14Gbps

replication

capacity per NP


ASR 9000 Flexible Ethernet SW Infrastructure(“EVC” SW Infrastructure, BRKSPG-2204 )

VLAN tag local significant

Flexible VLAN

tag classification

Flexible VLAN

tag rewrite

Flexible

Ethertype (.1Q,

QinQ, .1ad)

Flexible service mapping and multiplexing. Support all standard based

services concurrently on the same port:Regular L3, L2 interface/sub-interfaceIntegrated L2 and L3 – IRB/BVIMixed L2 and L3 sub-interfaces on the same port

1

2

(H-)VPLS

EoMPLS PW

EoMPLS PW

EoMPLS PW

L3 SubI/F

X

Bridging

Bridging

Routing

IRB

Routing and Bridging

IRB

X

L2 or L3 sub-interfaces(802.1a/qinq/.1ad)





L2VPN P2P

L2VPN P2P service configuration example

l2vpn

xconnect group cisco

p2p service1 local connect

interface gig 0/0/0/1.101


p2p service2 VPWS


neighbor 1.1.1.1 pw-id 22

p2p service3 PW stitching



EFP configuration example

Interface gig 0/0/0/1.101 l2transport

encapsulation dot1q 101 second 10

rewrite ingress pop 2 Symmetric


encapsulation dot1q 101



encapsulation dot1q 102-105

rewrite ingress push dot1q 100 Symmetric


CLI Comparison – L2VPN P2P

ASR 9000 7600

Local Connect interface GigabitEthernet4/1/0service instance 101 ethernet

encapsulation dot1q 101 second 10rewrite ingress tag pop 2 Symmetric

interface GigabitEthernet4/1/1service instance 100 ethernet

encapsulation dot1q 100rewrite ingress tag pop 1 Symmetric

connect eline-101 GigabitEthernet4/1/0 101

GigabitEthernet4/1/1 100

EoMPLS interface GigabitEthernet4/1/1

service instance 11 ethernet

encapsulation dot1q 101 second-dot1q 60-70

xconnect 10.0.0.3 101 encapsulation mpls

PW stitching l2 vfi tac-training point-to-point

neighbor 10.0.2.3 3001 encapsulation mpls

neighbor 10.0.2.2 3000 encapsulation mpls

[note] require BGP configuration if it‟s for inter-

AS

EFP configuration under interface

Including VLAN tag encapsulation, tag rewrite, Qo/ACL features, etc

Interface gig 0/0/0/1.101 l2transportencapsulation dot1q 101 second 10rewrite ingress tag pop 2 Symmetric

Interface gig 0/0/0/2.101 l2transportencapsulation dot1q 101rewrite ingress tag pop 1 Symmetric

Service configuration under “l2vpn”

l2vpnxconnect group cisco

p2p service1 local connectinterface gig 0/0/0/1.101interface gig 0/0/0/2.101

p2p service2 EoMPLSinterface gig 0/0/0/3.101neighbor 1.1.1.1 pw-id 22

p2p service3 PW stitchingneighbor 2.2.2.2 pw-id 100neighbor 3.3.3.3 pw-id 101


L2VPN Multi-Point

L2VPN MP service configuration example

l2vpn

bridge group cisco

bridge-domain domain1 local bridging

Interface gig 0/0/0/1.101

split-horizon group no bridging among same SHG


split-horizon group

bridge-domain domain2 vpls



vfi cisco



bridge-domain domain3 h-vpls


neighbor 192.0.0.3 pw-id 100 spoke PW

vfi cisco for core PWs

neighbor 192.0.0.1 pw-id 100 core PW


EFP configuration example









rewrite ingress push dot1q 100 Symmetric


CLI Comparison – L2VPN MP Bridging

ASR 9000 7600

Local Bridging interface GigabitEthernet4/1/0service instance 101 ethernet

encapsulation dot1q 101-1000

bridge-domain 100 split-horizon

interface GigabitEthernet4/1/1

service instance 101 ethernetencapsulation dot1q 101-1000


(H-)VPLS &

SVI based

EoMPLS

interface GigabitEthernet4/1/0service instance 2 ethernet




service instance 2 ethernetencapsulation dot1q 20


l2 vfi vpls-20 manual VPLS

vpn id 120

neighbor 10.0.0.1 encapsulation mpls core PWneighbor 10.0.0.2 encapsulation mpls no-split-

horizon spoke PW

neighbor 10.0.0.4 encapsulation mpls

interface Vlan20 (H-)VPLS

xconnect vfi vpls-20

interface Vlan20 SVI EoMPLS

xconnect 10.0.0.1 101 encap mpls

EFP configuration under interface

Including VLAN tag encapsulation, tag rewrite, Qo/ACL features, etc

Same as L2VPN P2P services

Service configuration under “l2vpn”

l2vpn

bridge group cisco

bridge-domain domain1 local bridging


split-horizon group


split-horizon group

bridge-domain domain2 SVI EoMPLS




bridge-domain domain3 vpls & h-vpls


neighbor 192.0.0.3 pw-id 100 spoke PW

vfi cisco core PWs under VFI config




L2VPN E-Tree Using Split-horizon

L2VPN E-Tree service configuration example

l2vpn

bridge group cisco

bridge-domain domain1


split-horizon group leaf port

Interface gig 0/0/0/2

split-horizon group leaf port

Interface gig 0/0/0/3.101 root port

Interface gig 0/0/0/4 root port

Leaf

RootLeaf

Root

MPLS

E-Tree Forwarding Rules:

• Root can talk with root

• Root can talk with leaf or leaf

can talk with root

• but leaf can‟t talk with leaf

BD

VFI

MPLS

Root bridge port

Leaf bridge port


split-horizon group leaf access PW

neighbor 192.0.0.4 pw-id 100 root access PW

vfi cisco vfi PW is root PW ONLY



Root


Integrated L2 and L3 Support – BVI vs. SVI

7600 SVI exampleASR 9000 IRB/BVI example



Interface bvi 20

ipv4 address 1.1.1.1 255.255.255.0

l2vpn

bridge group cisco




routed interface bvi 20

interface gig 1/2

switchport

switchport mode trunk

switchport trunk allow vlan 50-1000


service instance 2 ethernet


rewrite ingress tap pop 1 sym

bridge-domain 50

Interface vlan 50

ip address 1.1.1.1 255.255.255.0

vfi cisco




xconnect 192.0.0.3 100 en mpls

interface vlan 50

ip address 1.1.1.1 255.255.255.0

xconnect vfi cisco

l2 vfi cisco manual

vpn id 50



neighbor 192.0.0.3 encapsulation mpls no-split-horizon

VLAN is global

VLAN is local significant, no

relationship with BVI number


Mixed L2 and L3 on the Same Port

L2 trunk port, use SVI for L3

interface gig 1/2

switchport



Interface vlan 50

ip address 1.1.1.1 255.255.255.0

Native L2 and L3 sub-interface on the

same port



ip address 1.1.1.1 255.255.255.0

Interface gig 0/0/0/0/1.51 l2transport

encapsulation dot1q 51rewrite ingress tag pop 1 Symmetric

l2vpn define L2 servicebridge group test

<snip>

7600/6500

ASR 9000

SVI SVI

Native L2 and L3 sub-interface support on the

same physical port

No such limitations of the L3 features on the SVI

L2 trunk with

SVI for L3

Native L2 + L3 port


CLI Comparison – EVC vs. L2 switchport “trunk”



…


l2vpn

bridge group cisco




<snip>




interface gig 1/1

switchport



interface gig 1/2

switchport



ASR 9000 (EVC)

configuration example Switchport trunk

configuration

Switchport trunk model is easy for the L2

trunk configuration, especially with hundreds or

thousands of VLANs

EVC per sub-interface model has advantage

of providing per VLAN features, local VLAN

significant, and no global 4K VLAN limit, etc

EVC model is complex for provisioning if per

VLAN feature is not required. CLI enhancement

is under investigation

VLAN is global

VLAN is local port

significant


Flexible VLAN Tag Classification

RP/0/RSP0/CPU0:PE2-asr(config)#int gig 0/0/0/4.100 l2transport

RP/0/RSP0/CPU0:PE2-asr(config-subif)#encapsulation ?

default Packets unmatched by other service instances

dot1ad IEEE 802.1ad VLAN-tagged packets

dot1q IEEE 802.1Q VLAN-tagged packets

untagged Packets with no explicit VLAN tag

RP/0/RSP0/CPU0:PE2-asr(config-subif)#encapsulation dot1q 100-200,205 ?

comma comma

exact Do not allow further inner tags

RP/0/RSP0/CPU0:PE2-asr(config-subif)#encapsulation dot1q 100 second-dot1q 10-20,35 ?

comma comma


RP/0/RSP0/CPU0:PE2-asr(config-subif)#encapsulation dot1ad 20 dot1q 10-20 ?

comma comma


Double tag

Single tag

Multiple tag

Range of tag

.1q and/or .1ad

Loose or exact match

Untagged

Default tag


Flexible VLAN Tag Rewrite

RP/0/RSP0/CPU0:PE2-asr(config)#int gig 0/0/0/4.100 l2transport

RP/0/RSP0/CPU0:PE2-asr(config-subif)#rewrite ingress tag ?

pop Remove one or more tags

push Push one or more tags

translate Replace tags with other tags

RP/0/RSP0/CPU0:PE2-asr(config-subif)#rewrite ingress tag pop ?

1 Remove outer tag only

2 Remove two outermost tags

RP/0/RSP0/CPU0:PE2-asr(config-subif)#rewrite ingress tag push ?

dot1ad Push a Dot1ad tag

dot1q Push a Dot1Q tag

RP/0/RSP0/CPU0:PE2-asr(config-subif)#rewrite ingress tag push dot1q 100 ?

second-dot1q Push another Dot1Q tag

symmetric All rewrites must be symmetric

RP/0/RSP0/CPU0:PE2-asr(config-subif)#rewrite ingress tag translate ?

1-to-1 Replace the outermost tag with another tag

1-to-2 Replace the outermost tag with two tags

2-to-1 Replace the outermost two tags with one tag

2-to-2 Replace the outermost two tags with two other tags

Pop tag 1 or 2

Push tag 1 or 2

Tag translation

1-1

1-2

2-1

2-2


Flexible Service and VLAN Tag ClassificationDeployment Example – Consolidated DC Edge

VFI

MC-LAG

interface bundle-ethernet1.10 L3 sub for internet access


ipv4 address 1.1.1.1 255.255.255.0

interface bundle-ethernet1.20 L3 sub for L3VPN


vrf customer1

ipv4 address 20.1.1.1 255.255.255.0

interface bundle-ethernet1.100 l2transport VLAN range for DCI1

encapsulation dot1q 100-1000 selective QinQ

interface bundle-ethernet1.200 l2transport VLAN range for DCI2

encapsulation dot1q 1001-2000 selective QinQ

L2vpn

bridge group DCI

bridge-domain DCI1

interface bundle-ethernet1.100

vfi DCI1

neighbor …

bridge-domain DCI2

interface bundle-ethernet1.200

vfi DCI2

neighbor …

VFI VFI VFI

vPC

Data Center 1

DC2DC3

L3VPN

Internet

DCIt


ASR 9000 L3 “At-a-Glance”Extend IOS-XR L3/MPLS to the ASR9000

IOS-XR routing and MPLS Widely deployed, proven

stability and feature rich

Built-in HA NSF/GR & NSR

& SSO

Distributed control plan High Multi-dimensional control plane scale, for example 5000

BGPs + 10,000 BFDs + etcHierarchical FIB

structure and Fast FIB programming

Prefix independent fast network convergence

IPv6 ready Dual stack, 6PE/6VPE,

high scale, line-rate IPv6 forwarding performance

Distributed inline Netflow1M entries/LC , 100Kpps /LC, flexible sampling and IPv4/IPv6/MPLS feature

support Mature IOS-XR

L3 Routing & MPLS

SW Infrastructure


L3 Control Plane Overview

LDP RSVP-TEBGP

ISIS

OSPF

EIGRP

Static

FIB Adjacency

LC NPU

ARP

LSD RIB

AIB

SW FIB

LC CPU

RP

AIB: Adjacency Information Base

RIB: Routing Information Base

FIB: Forwarding Information Base

LSD: Label Switch Database


L3 Forwarding – Hierarchical Data Structure FIB and ADJ Tables Overview

RecursivePrefix Leaf

Table

NR PrefixLeaf Table

RecursiveLDI

NR LDI 0

NR LDI 1

…

NR LDI N

0 1 2 3 . . . … nVRF table

Search Tree

Leaf Table

(FIB table)

LDI/ADJ table

BGP PIC

TE-FRR PIC

Link bundle PIC

PIC: Prefix independent convergence

LAG: Link aggregation group

LDI: Load information

OutI/FProtected

TE adj

OutI/FBackupTE adj

OutI/F

OutI/F

LAG

LAG

adj

adj


Show CLI Examples –Validate the L3 Forwarding Path

show ip route 10.0.0.4

show cef 10.0.0.4/32 RSP‟s view

Show cef 10.0.0.4/32 location 0/2/cpu0 Local Line card‟s view

Show cef 10.0.0.4/32 detail location 0/2/cpu0detailed view on the line card

show mpls forwarding prefix 10.0.0.4/32

show arp 10.0.13.3

show adjacency ten 0/2/0/1 detail location 0/2/cpu0


3-Level Evenly Load BalancingBGP Multi-path, ECMP, Link Bundle

BGP multi-path z.z.z.za.a.a.a

1.1.1.1

1.1.2.2

1.1.8.8

11.1.1.1

11.x.x.32

BGP multi-paths

8-way (RLDI)

1.1.1.1

1.1.2.2

…

1.1.8.8

ECMP 32-way

(NR-LDI)

11.1.1.1

11.2.2.2

…

11.x.x.32

Link bundle, maximum 64

member ports (LAG)

Gig 0/0/0/1

Gig 0/1/0/2

…

Gig 0/3/0/39

z.z.z.z

Dynamic hashing table,

256 hashing buckets,

Evenly load balancing

regardless of odd or even

number of BGP

paths/ECMPs/bundle

member ports

Note, 8-way dynamic BGP multi-

path supported from 4.1.1 release

Level 1Level 2

Level 3


L3 Unicast – Load balancing Parameters

Incoming Traffic Type Parameters

IPv4 Unicast Source IP, Destination IP, Source port (TCP/UDP only), Destination port (TCP/UDP

only), Router ID

IPv6 Unicast Source IP, Destination IP, Source port (TCP/UDP only), Destination port (TCP/UDP

only), Router ID

MPLS

- IP Payload, with < 4 labels

Source IP, Destination IP, Source port (TCP/UDP only), Destination port (TCP/UDP

only), Router ID

- IP Payload, with > 4 labels 4th MPLS Label (or Inner most) and Router ID

- Non-IP Payload Inner most MPLS Label and Router ID


L3 Multicast – Load balancing Parameters


IPv4 Multicast (Bundle load-

balancing)

Egress LC selection Destination IP (a.k.a, multicast group address)

Egress NP selection Source IP, Destination IP, Source port (TCP/UDP only), Destination port (TCP/UDP

only), Router ID

Egress NP selection Destination IP (a.k.a, multicast group address) (From 4.1 release for optimization)

Egress member selection

within an NP


only), Router ID

IPv6 Multicast

Egress LC selection Destination IP (a.k.a, multicast group address)

Egress NP selection Source IP, Destination IP, Source port (TCP/UDP only), Destination port (TCP/UDP

only), Router ID

Egress NP selection Destination IP (a.k.a, multicast group address) (From 4.1 release for optimization)

Egress member selection

within an NP


only), Router ID


L2 Services – Load balancing Parameters


Bundle load-balancing

Default Source MAC, Destination MAC

EFP-based with auto mode XID of the xconnect

EFP-based with user hash User provided hash

Flow-based with IP src-dst Source IP and Destination IP

Flow-based with MAC src-dst Source MAC and Destination MAC

PW load-balancing over

ECMP

Default PW VC Label

Flow-based with IP src-dst Source IP and Destination IP

Flow-based with MAC src-dst Source MAC and Destination MAC

Note: Both Unicast and flood traffic follows the same load-balancing


Cisco ASR 9000 L2It does FULL L2, but not a classic switch

ASR 9000 support full L2 service (multi-point, point-to-point, E-Tree, native L2 or L2 over MPLS). It‟s fully MEF certified.

However, the behavior and configuration is quite different than traditional L2 switch

By default, it‟s Router, it doesn‟t do any bridging function unless you configure it specifically

There is no “switchport “ concept. All L2 features are done with “EVC” SW infrastructure

There is no global VLAN concept. All VLAN is local port significant. By default VLAN 10 on port A doesn‟t talk to VLAN 10 on port B. It require “bridge-domain” configuration to bridge port/VLAN together

ASR 9000 doesn‟t support SVI. No “int vlan” concept. Instead, it use “IRB/BVI”. It‟s different CLI but provide the same function

By default, STP is not enabled on any L2 port unless you “explicitly” configure it under STP process

NO global VLAN (no 4K limit)

Per-VLAN features, more flexible, more secure, more scale


ASR 9000 L2 “At-a-Glance”Built on New Cisco EVC SW Foundation

Flexible Cisco “EVC” Carrier Ethernet SW infrastructure

VLAN tag matching, manipulation, service

mapping

Distributed control plan High Multi-

dimensional L2 service scale, VLAN, PW,

Bridge-domain, VPLS, MAC address scale, etc

Distributed HW based MAC learning,

aging/flushing, 4-5Mpps

Rich L2 fast convergence

features

Standard compliant, MEF certified,

802.1q/qinq/.1ad/.1ah EoMPLS, H-VPLS/VPLS supported, L2 feature rich

E-OAM: CFM, Y.1731, Link OAM

Cisco “EVC”

Carrier Ethernet

Foundation


ASR 9000 MAC Learning Overview

NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHYFIA

CPUNP0PHY

NP2PHY

NP3PHY

NP1PHY

FIA

CPU

Switch Fabric

MAC learning/aging/flushing is done by hardware

and fully distributed on each NP independently

Distributed HW based MAC Learning

–MAC learning /aging/flushing is fully distributed to the NP, independently

–Each NP can learn around 4-5 Millions MAC addresses per second in hardware

Data-plane MAC table synchronization

–MAC address are synchronized across all NPs in the entire system, regardless

of the bridge-domain or bridge port configuration.

–MAC address synchronization is achieved by special MAC notification

messages which is implemented in the data-plane microcode


Displaying MAC Table –How to handle large scale MAC address table

The Hardware based MAC tables must be collected in order to be displayed

From release 3.9, display of mac-address table requires a command to sync (collect) the Hardware MAC tables first. In prior releases, resync was constantly going as background task.

From 4.1, we have small enhancement: the system will check the total MAC address at first. If it has less than 16K, then it will display the MAC without any resync CLI. System will resyncimplicitly. However, if it has more than 16K MAC address, then it will behavior the same as in 3.9 release.

The command to display mac-addresses tells you the last resync time (hold „old‟ the displayed mac table is), and tells you the command to resync…

show l2vpn forwarding bridge-domain mac-address loc 0/7/CPU0To Resynchronize MAC table from the Network Processors, use the command...

l2vpn resynchronize forwarding mac-address-table location

Mac Address Type Learned from/Filtered on LC learned ResyncAge Mapped to--------------------------------------------------------------------------------0000.5e00.0101 dynamic Te0/7/0/1.100 0/7/CPU0 0d 0h 34m 40s N/A

After issuing the resync command, an info message will indicate when sync is complete.

l2vpn resyn forwarding mac loc 0/7/CPU0

l2fib[189]: %L2-L2FIB-5-MAC_TABLE_RESYNC_COMPLETE : The resynchronization of

the MAC address table is complete 0/7/CPU0


Useful Show CLI – L2 Service

show run l2vpn

show l2vpn xconnect

show l2vpn xconnect group group1 xc-name vpws-1 detail

show l2vpn bridge-domain

show l2vpn bridge-domain bd-name vpls-1 detail

show l2vpn forwarding bridge-domain group1:132-vpls location 0/0/cpu0

show l2vpn forwarding bridge-domain group1:132-vpls detail location 0/0/cpu0

show l2vpn forwarding interface gig 0/0/0/15 detail location 0/0/cpu0

show controllers np struct 18 all loc 0/0/cpu0 NP structure 18 is for MAC address

show controllers np struct 18 detail all loc 0/0/cpu0


“8+1” CPU for Ultra-High multi-

dimensional control plane scale

ASR 9000 Scalable System Architecture

FIC

CPU BITS/DTI

Line Card

FIA

CPU

RSP

Control plane split among RSP and LC

CPU (same type of CPU as RSP)

L2 protocols, BFD, CFM, Netflow runs

on the LC CPU for high scale

Multi-dimensional scale

FIB 1.3M (v6 use two entries)

MAC address 512K

L2 interfaces 64K

Pseudowires 64K

Bridge-domain/VFI 8K

EFPs per BD 16K

L3 interfaces, VRF 8K

CFM MEP/MIP 32K/LC

Netflow 1M entries/LC, 100K pps/LC

BFD 1250/LC (with 150msechello timer), minimal hello

timer is 15msec

Scale examples


Scale Profiles

RP/0/RSP0/CPU0:asr1(admin-config)#hw-module pro scale ?

default Default scale profile

l3 L3 scale profile

l3xl L3 XL scale profile

RP/0/RSP0/CPU0:asr1(admin-config)#hw-module pro scale l3

Mon Apr 4 18:55:40.070 UTC

In order to activate this new memory resource profile, you must manually reboot the

line cards.

RP/0/RSP0/CPU0:asr1(admin-config)#hw-module pro scale l3xl

Mon Apr 4 18:55:52.654 UTC

In order to activate this new memory resource profile, you must manually reboot the

system.

Feature Default L3 L3XL

FIB

(v4 and v6 share the

same table, v6 use

two entries)

512K v4 1M v4 1.3M v4

256K v6 (128K

maximum global and

per-VRF)

512K v6 (128K

maximum global and

per-VRF)

650K v6 (256K

maximum global and

per-VRF)

MAC 512K 128K 32K

Bridge-domain/VFI 8K 8K 2K

Note, FIB scale is different than the RIB. FIB is hardware table in the forwarding ASIC. RIB is software table on the

RP. ASR9K can support more than 20M RIB

3 scale profiles


Access

Ethernet Node

Ethernet Node

DSL Node

PON Node

HA and Fast Convergence

Access

Distribution Node

Pre-Aggregation/aggregation node

IP/Service Edge

Business

Corporate

Business

Corporate

2G/3G RBS

Residential

STB

Residential

STB

Residential

STB

Business

Corporate

Cell Site Gateway

IP/MPLS

Residential Data/Voice

Business L3 VPN

Residential IPTV/VoD

Business L2 VPN (E-Line/E-LAN/E-Tree)

Mobile backhaul

L2 and L3 transport layer resiliency, transit link or transit node fail, L2 or L3 transport

protocol re-converge. Service layer is not aware

L2: MST, G.8032, REP(not supported), PVST(not supported)

L3: IGP/BGP, TE/FRR(link, node, path protection), IP/FRR (ISIS per-link and per-prefix

, OSPF per-link, OSPF per-prefix IP/FRR

Multicast: PIM, P2MP-TE/FRR

Link layer resiliency: transport and service layer is not aware, for link protection only Link Bundle (active-active, 1:1 backup, m:n backup)

L2/L3 AccessIP/MPLS (pre)aggregation

L2/L3 Link Bundle

System HA: IOS-XR Modular OS, Micro-kernel, RSP failover, SMU, ISSU(4.2), Process restart. All protocols are SSO, all protocol support NSF/GR, OSPF/ISIS/LDP/BGP support NSR. PIM support NSF and is NSR compliant

Resiliency infrastructure: distributed BFD, distributed EFD, HW based LoS detection, IPoDWDM FEC/EFC & proactive protection, Hierarchical Forwarding table structure and in-place modification, Fast programming FIB table

Service layer resiliency, service termination point (node or link) fail., service is down

and need to re-route

L2: PW redundancy, L3: BGP PIC, Multicast: MoFRR

Access dual home: REP-AG/MST-AG, Rapid-PVST/PVST-AG, MC-LAG, G.8032,

Flexible Link (not supported), HSRP/VRRP

State sync between primary and backup service node: ANCP, IGMP snooping,

DHCP/ARP/IGMP (in future), etc


FIAFIA

RSP1

Switch Fabric Bandwidth Access OverviewIntelligent Fabric and Internal System QoS

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

RSP0

1: Fabric Request

3: Fabric Grant

2: Arbitration

4: load-balanced transmission across fabric links

5: credit return

Ingress LC Egress LC

Arbiter

Arbiter

3 strict priority scheduling/queueing

Back pressure and virtual output queue

Multicast and Unicast separation

(separated queues and fabric plane)


End-to-End System QoS Queuing

Ingress (sub-)interface QoS Queues

VirtualOutputQueues

Egress FIAQueues

End-to-End priority (P1,P2, Best-effort) propagation Guarantee bandwidth, low latency for high priority traffic

at any congestion point3 strict priority level across all internal HW components

Configure withIngress MQC

4-layer hierarchyTwo strict high priority +

Normal priority

Egress (sub-)interface QoS Queues

Configure withEgress MQC

4-layer hierarchyTwo strict high priority +

Normal priority

Implicit ConfigurationTwo strict high priority +

Normal priority

Ingress side of LC Egress side of LC

NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHYFIA

CPUNP0PHY

NP2PHY

NP3PHY

NP1PHY

FIA

CPU

Switch Fabric1

23

4

12

34


Backpressure and VoQ Mechanism

Egress NP congestion backpressure to ingress FIA

Packet is en-queued in the dedicated VoQ

No impact of the packet going to different egress NP

No head-of-line-block issue

Ingress side of LC1 Egress side of LC2

NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHYFIA

CPUNP0PHY

NP2PHY

NP3PHY

NP1PHY

FIA

CPU 1

3

2

Backpressure: egress NP egress FIA fabric Arbiter ingress FIA VoQ

Switch Fabric

VoQ Scale: Each FIA has P1/P2/BE queue set for every NP and RSPs in the entire system

5Gbps

10Gbps

5Gbps

Packet going to different egress NP put into different VoQ set

Congestion on one NP won‟t block the packet going to different NP


CLI – System Queues Counters

RP/0/RSP0/CPU0:PE2-asr#show controllers fabric crossbar ? switch fabric

readreg Read register

serdes Display chico info

statistics Packet Statistics switch fabric drop counters

RP/0/RSP0/CPU0:PE2-asr#show controllers fabric fia ? fabric interface ASIC

bridge show command for bridge asic

drops Drops for fabric interface asic drop counters

errors Errors for fabric interface asic

link-status Link status between fabric/xbar and fabric interface asic

q-depth Ingress Queue Depth Counters

stats Packet statistics for fabric interface asic packet statistics

trace Show controllers fia trace data

RP/0/RSP0/CPU0:PE2-asr#show controllers fabric fia bridge ? bridge FPGA

ddr-status DDR staus between bridge and fia

flow-control Flow control information for bridge flow control counters

stats Packet statistics for bridge packet statistics

sync-status Bridge and NP sync status


Port QoS Overview

High scale

–Up to 3 Million queues per system (with E linecard)

–Up to 2 Million policers per system (with E linecard

Highly flexible: 4 layer hierarchy queuing/scheduling support

–Four layer scheduling hierarchy Port, Subscriber Group, Subscriber, Class

–Egress & Ingress, shaping and policing

Three strict priority scheduling with priority propagation

Flexible & granular classification, and marking

–Full Layer 2, Full Layer 3/4 IPv4, IPv6

NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHYFIA

CPUUser configure port level QoS via MQC

CLIs

QoS function is done by special Queuing

HW chip within the NP complex

Different memory options have different

scale

Q

Q

Q

Q


Agenda





‒ L3 Forwarding

‒ L2 Forwarding






Switch Fabric

NP0PHY

NP2PHY

NP3PHY

NP1PHY

FIA

CPU

B0

B1

show interface gig 0/0/0/35show|clear controllers gig 0/0/0/35 stats

show|clear control np counters np0 location 0/0/cpu0

clear control fabric fia location 0/0/cpu0show controllers fabric fia bridge stats location 0/0/cpu0

clear control fabric fia location 0/0/cpu0show controllers fabric fia stats location 0/0/cpu0

Gig0/0/0/35

Gig0/0/0/0

Ingress

Egress

Show interface gig 0/0/0/0show|clear controllers gig 0/0/0/0 stats

show|clear control np counters np3 location 0/0/cpu0

clear control fabric fia location 0/0/cpu0show controllers fabric fia bridge stats location 0/0/cpu0

clear control fabric fia location 0/0/cpu0show controllers fabric fia stats location 0/0/cpu0

Show controller fabric crossbar stats instance [0-1] location 0/x/cpu0

Data Plane CountersFocus on the PHY/NP counters,

NOT B/FIA/crossbar


Interface

“show interface Ten 0/1/0/3 “

Displays interface related statistics

“show controller Ten 0/1/0/3 stat”

Find out if there are any drops or errors at the physical layer

Input drop overrun = 0

Input drop abort = 0

Input drop invalid VLAN = 0

Input drop invalid DMAC = 0

Input drop invalid encap = 0

Input drop other = 0

Input error giant = 0

Input error runt = 0

Input error jabbers = 0

Input error fragments = 0

Input error CRC = 0

Input error collisions = 0

Input error symbol = 0

Input error other = 0

RP/0/RSP0/CPU0:A9K-TOP#sh int te 0/3/0/3 | i opsMon Mar 21 14:22:43.831 UTC

304292 packets input, 36389589 bytes, 47401 total input drops

4514 drops for unrecognized upper-level protocol107885 packets output, 10123636 bytes, 0 total output drops

RP/0/RSP0/CPU0:A9K-TOP#sh int te 0/3/0/3 | i err

Mon Mar 21 14:23:00.856 UTC0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 output errors, 0 underruns, 0 applique, 0 resets

Aggregate drops from mac/phy and np

Phy errors (controller related)


Which interface to which NP?

Use the command “show controller np ports all location 0/Y/CPU0” to find out which interface is connecting to which NP

RP/0/RSP1/CPU0:A9K-BOTTOM#sh controllers np ports all loc 0/7/CPU0

Fri Mar 4 12:09:41.132 EST

Node: 0/7/CPU0:

----------------------------------------------------------------

NP Bridge Fia Ports

-- ------ --- ---------------------------------------------------

0 0 0 TenGigE0/7/0/3

1 0 0 TenGigE0/7/0/2

2 1 0 TenGigE0/7/0/1

3 1 0 TenGigE0/7/0/0

In case you have an oversubscribed linecard, 2 TenGig interfaces can be connected to the same NP.

For 80G linecards such as the A9K-8T, you'll see 8 NP's listed.

For the 40 port 1GE, you'll see 10 1-Gig interfaces connecting to an NP, note that this is not oversubscription.


NP Counters (not all are drop or error related)

RP/0/RSP1/CPU0:A9K-BOTTOM#show controller np count np1 loc 0/7/CPU0Node: 0/7/CPU0:

----------------------------------------------------------------

Show global stats counters for NP1, revision v3

Read 30 non-zero NP counters:

Offset Counter FrameValue Rate (pps)

-------------------------------------------------------------------------------

22 PARSE_ENET_RECEIVE_CNT 51047 1

23 PARSE_FABRIC_RECEIVE_CNT 35826 0

30 MODIFY_ENET_TRANSMIT_CNT 36677 0

31 PARSE_INGRESS_DROP_CNT 1 0

34 RESOLVE_EGRESS_DROP_CNT 628 0

40 PARSE_INGRESS_PUNT_CNT 3015 0

41 PARSE_EGRESS_PUNT_CNT 222 0

Received from the ethernet framer/mac/phy

Received from fabric

Packets modified and sent to ethernet

Packets dropped in the PARSE stage

Packets received and determined to be punted to the CPU.

NP Drop counters on CCO: https://supportforums.cisco.com/docs/DOC-15552

Drops for unrecognized upper-level protocol error: https://supportforums.cisco.com/docs/DOC-15490

https://supportforums.cisco.com/docs/DOC-15552







Cisco ASR 9000 Summary

Most Scalable, Lowest-Power

Carrier Ethernet System in the

Industry

Best in Class L2 + L3, Carrier Ethernet, &

MSE Services

Optimized for Video Services over Converged

Wireline + Wireless NGNs

Nonstop Availability with IOS-XR Modular

OS


Thank you.

cisco asr 9000 architecture - andovercg.com · cisco asr 9000 architecture dennis cai, erbu...

Documents