ios xr platform hardware - clnv.s3.amazonaws.com · •introduction – • building a forwarding...

IOS XR Platform Hardware Architectures

LJ Wobker, Principal Engineer

BRKSPG-2404

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKSPG-2404


Teaser Slide…

• We’re going to talk about how routers are actually built. We’ll talk a lot about what things

go into the design choices, and why it’s really hard (actually, it’s impossible) to build things

that solve all the problems at the same time. What we’re really going to talk about are

engineering realities, and the tradeoffs that they impose on those of us who build, and

those who ultimately use, the systems. We’ll talk about the major dimensions that drive

hardware design, and the relative costs to make “one thing better” in a given system. We

will cover the high level architecture of several IOS XR platforms, but mostly in the context

of how and why they are different and what design choices went in to each one.

• Introduction –

• building a forwarding path

• Platform Design & Building Blocks

• IOS XR Platforms

• Hardware

• Virtualized

AgendaModern demands such as cloud computing, mobility,

and media delivery are driving greater bandwidth and

service requirements into provider networks. To

address this, Cisco offers a broad portfolio of systems

for provider networks. These systems all run the IOS

XR operating system, but have meaningful differences

in how they are designed from a system/hardware

standpoint. This presentation discusses system

architectures for the NCS-6000, ASR-9000, NCS-5500,

and other IOS XR platforms with a specific focus on the

high-level design decisions. A significant part of the talk

is around silicon selection (custom vs. merchant, for

example) and the mechanical and logical design of

linecards for these larger systems.


Should I be here?Today’s topics and not-topics.

• Hardware architectures

• System-level design

• Data planes

• Packet forwarding

• High-speed, complex, (expensive?) systems

• Routing protocols / design

• Network level designs

• Control plane / OS infrastructure

• Selling a specific product

Yes!

No! (or at least not much)

BRKSPG-2404 6


You can’t always get what you wantBut if you try sometimes well you just might find you get what you need

• Nothing is free

• Some things are closer to free

• The further from the middle,

the more things “cost”

BRKSPG-2404 7


The IOS XR Router Family

NCS 5000 NCS 5500 NCS 6000 ASR 9000

CRS

BRKSPG-2404 8


Optics

Power

Silicon

Cooling

Process PP

S

SE

RD

ES

NE

BS

Busbar

Chassis

Airflo

w

Backplane

Sig

nal In

teg

rity Pip

elin

e

Run to completion

LR4

QSFP28

CPAK

N:NFilters

StatsFeatures

Programmability

Die

siz

eInterposer

Cable Management

Fast Convergence

He

at S

inks

Aco

ust

ics

Materials

PCB

Junction temp

Liq

uid

co

olin

g

Co

nn

ecto

rs

Ro

uti

ng

Slo

t p

itch

MemoryFIB size

Buffering

Cap

acity

BandwidthO

ps/se

c

Se

rial

HB

M

HMC

GDDR5

DD

R3

TCAM

SRAM

On

-ch

ip

LLD

RA

M

ACL scale

Fabric

MPO

IPo

DW

DM

RouterHardware

BuildingBlocks

BRKSPG-2404 9


What’s needed to build a forwarding path?

1. Optical to electrical

2. Transport a signal from optics to NPU

3. Ingress forwarding operations

4. Transport a signal from ingress NPU to egress NPU (fabric)

5. Egress forwarding operations

6. NPU to optics

7. Electrical to optical

1/7

4

2/63/5

BRKSPG-2404 10


Logical view of forwarding path components

Fabric

Interface

ASIC

Optics

Optics

FIA

FIA

Optics

Optics

FIAOptics

nPower X1

nPower X1

nPower X1

nPower X1 FIAOptics

Tables

Packet

Buffers

TCAM TM

nPower X1

Fabric

Control

EthernetCPUDRAM

Ethernet

Switch

NPU

LASERS / RECEIVERS

FAST MEMORY

FAST MEMORY

TCAM MEMORY

SERDES

SERDES

SILICON

SILICON

BRKSPG-2404 11


Challenge: Scale routers faster than componentsNote: exponential scale

1x

64x

16x

4x

256x

1024x

4096x

Moore’s Law

?

?

?

?

BRKSPG-2404 12

1998 2000 2012 20162002 2004 201020082006 2014 2018


Technology Trends – POS/Ethernet StandardsFastest Interface Bandwidth

1x

64x

16x

4x

256x

1024x

4096x

OC-12

GE

100G

OC-48

40G

10G

400G

Moore’s LawInterface Speeds

BRKSPG-2404 13

1998 2000 2012 20162002 2004 201020082006 2014 2018


Technology Trends – POS/Ethernet StandardsFastest Interface & Router Bandwidth

1x

64x

16x

4x

256x

1024x

4096x

OC-12

GE

100G

40G

10G

400G

Moore’s LawInterface Speeds

58T

8G

8T

640G

2.4T

16T

Buffered Router Bandwidth

OC-4828G

24T

BRKSPG-2404 14

1998 2000 2012 20162002 2004 201020082006 2014 2018


Technology Trends - OpticsBandwidth/Volume

1x

64x

16x

4x

256x

1024x

4096x

GBIC 100G QSFP28100G CFP10G SFP+10G XFP100G CPAK40G QSFP+

Driven by physics, not Moore’s Law, costs shifting

SerDes dependency (NPU interface)

Cooling challenge – 30C lower max temp vs. ASICs

BRKSPG-2404 15

1998 2000 2012 20162002 2004 201020082006 2014 2018


Silicon & Modular Router PPS – Per NPU/LC*Mpps for Standalone and Modular/Buffered

1x

64x

16x

4x

256x

1024x

4096x

12575

164

700x6

280x5

CPU

3200

1200800

*represents a combination of memory, Silicon & SerDes

CRS-1

SoC – no buffer

Modular – buffered

NCS 6000

NCS 5500

12000

CRS-3

14x4

45x4

ASR 9000

ASR 9000

150x4

ASR 9000

BRKSPG-2404 16

1998 2000 2012 20162002 2004 201020082006 2014 2018

4800


Technology Trends – MemoryCommodity & Custom Bandwidth

1998 2000 2012 20162002 2004 201020082006 2014

1x

64x

16x

4x

256x

1024x

4096x

105

1.60.8

2528

100

GDDR5

DDR4

SDRDDR

DDR2

DDR3

HBM

325HMC

55

160

NCS 6000

NCS 6000

ASR 9000

2nd & 3rd gen

Scaling FIB above ~256K IPv4

usec vs. msec buffering

Operations / second critical for FIB

BRKSPG-2404 17


Technology Trends – SerDesHigh-performance Electrical Link (Speed in GHz)

1998 2000 2012 20162002 2004 201020082006 2014

1x

64x

16x

4x

256x

1024x

4096x

11.5

2.5

1.251.25

2515

5

Optics to NPU

NPU to fabric

NPU to TCAM

NPU to serial memory

BRKSPG-2404 18

2018

56


Technology Trends Compared

1998 2000 2012 20162002 2004 201020082006 2014

1x

64x

16x

4x

256x

1024x

4096x

InterfacesPPS

OpticsMemory

SerDes

BRKSPG-2404 19


Modular Buffered Router BandwidthSystems growing much faster than components – exponential scale

1998 2000 2012 20162002 2004 201020082006 2014

1x

64x

16x

4x

256x

1024x

4096x

InterfacesPPS

Memory

SerDes

*similar trends for conductive materials, connectors, fans, power supplies, CPUs, …

58T

8G

8T

640G

2.4T

16T

28G

24T

Optics

BRKSPG-2404 20

Platform Design


Why not one platform?Requirements have a butterfly effect

• An individual requirement may dramatically impact architecture

• Every component “ahead of the curve” adds complexity

• Key Drivers• Time / Cost

• Buffering

• FIB scale

• System Scale

• Features/Flexibility

BRKSPG-2404 22


Time –Terabit LCs in 2013 vs. 2016

• NCS 6000 1T – 200G NPU w/ full FIB, deep buffers, programmable• Significant investment in nPower X1 silicon development

• Custom memory for FIB and buffering – extremely high bandwidth and ops/sec

• $271M acquisition of Lightwire for small, low-power 100G optics

• ASR 9000 1.2T – similar speed, flexibility, and power – 3 years later• Commercial NPU, customized for Cisco, commodity memories

• Cisco fabric, backward compatible with earlier generations

• QSFP28 optics, full range of chassis sizes

• NCS 5500 3.6/2.4T – lighter silicon architecture• Commodity memory, partial bandwidth buffering, VoQ

• TCAM and non-TCAM options (density change)

• Reduced counters, QoS, queueing complexity

• 3.6T with very high scale only in 2018...BRKSPG-2404 23


Buffering

• On-chip limited to microseconds

• But… Off-chip requires • ASIC I/O pins – fewer interfaces

• Board space and power

• Commodity bandwidth is limited• Many devices may be needed

• Better suited for buffering than FIB (1 read/write per packet optimized for 128B read/write)

• Stalled – High-end graphics and networking moving to custom

• Custom memories• Fewer devices are needed to reach bandwidth, fewer pins

• Development and per-unit costs – Extremely expensive

NPU

Po

rts

Po

rts

Ports

Ports

FIB

Buffers

NPU

Po

rts

Po

rts

Ports

FIB

BRKSPG-2404 24


FIB Scale

• On-chip FIB limited to ~256K LPM entries• Usually an on-chip TCAM or lookup tree

• Can be scaled up to ~1M with well-known prefixes

• Combination of flat tables and LPM

• Larger FIB tables require external memory:

• ASIC I/O pins redirected from interfaces to memory• 4-8 reads for every packet – high operations per second• Board space (density) and power

NPU

Po

rts

FIB

Buffers

Po

rts

BRKSPG-2404 25


System ScaleFour common approaches

• Fixed – single NPU / Forwarding ASIC

• All resources dedicated to network interfaces

• May or may not have external memories

• Fixed – multiple NPUs

• Connected via mesh or fabric chips

• Modular – Expand with line cards

• Resources for fabric and usually external memories

• Multi-chassis

• Adds fiber connections to fabric cards (more power & board space)

• Increases software complexity

NPU

Fab

ric

Po

rts

FIB

Buffers

BRKSPG-2404 26


Features and Flexibility

• Packet Processing Engine (PPE)• C programmable

• Run to completion

• Anything is possible, no pure optimization

• Wide range of pipeline programmability• NCS 5000 < NCS 5500 < ASR 9000 < NCS6K

• Packet rate is strongly correlated to cost, power, and flexibility

• Counting stuff is much harder than most people realize

Netw

ork

In

terf

ace Ingress Egress

Port

TermParser

Link

Layer

VLAN

Trans

Tunnel

Service

Term

Fwd PMFFEC

ResFEC

Res

NCS 5500 PIPELINE

CRS PPES

BRKSPG-2404 27


Easy summary...

... and ...

A “cut corner” is the same thing as

“a tradeoff

you didn’t know you were making”

BRKSPG-2404 28

• “Trying to build X things to solve X^N problems”• is the same thing as

• A bunch of “engineering optimizations”

IOS XR Platforms


The IOS XR Router Family

NCS 5000 NCS 5500 NCS 6000 ASR 9000

CRS

BRKSPG-2404 30


• Highest router bandwidth capacity

• 128 Tbps first generation multi-chassis (2013)

• Only Tbps card on the market in 2013-2015

• With buffers and full FIB

• Dramatic improvement in power efficiency

• $6000/100G/month power savings over CRS-3 64x100

• Merchant fabric

• Highly programmable forwarding, massive stats

• Custom memories for FIB and buffering

• Custom CPAK optics

NCS 6000 – First Dense 100G

BRKSPG-2404 31


NCS 6000 Line Card Architecture

10X 100GE

• Slice architecture• Optics, NPU & FIA per slice

• Run to Completion NPU w/ PPEs

• 1 Generation (so far)• 1T line cards w/ 200G NPUs (40 nm)

• 400G NPUs in lab

• TCAM for ACL/QoS scale

• Deep buffers (50+ msec)

Fabric

Interface

ASIC

Optics

Optics

FIA

FIA

Optics

Optics

FIAOptics

nPower X1

nPower X1

nPower X1

nPower X1 FIAOptics

Tables

Packet

Buffers

T

C

A

M

T

M

nPower X1

CPUDRAM

PPEs

BRKSPG-2404 32


ASR 9000 Series

• Full range of chassis and interfaces• 40G, 200G, and 1T generations

• Highly programmable forwarding• Partnered for a customized NPU – cisco software

• Highest scale, feature, and QoS capabilities

• Cisco fabric

• Commodity memories for FIB and buffering

• Mostly commodity optics

BRKSPG-2404 33


ASR 9000 System Design

• Backplane or midplane

• Flexible switch fabric options• RP and fabric may be integrated into RSP

• Variable # of fabrics for increased capacity and redundancy

• First stage of fabric on line card

• Multiple airflow designs• Front-to-back – 9922, 9912, 9010

• Side-to-back – 9006

• Side-to-side – 9001, 9004

• Modular options for lower-speed and legacy interfaces

BRKSPG-2404 34


ASR 9000 Line Card Architecture

8X 100GE

CPUDRAM

FIANP-5cCPAK

CPAK

FIB Packet BuffersTCAM

FIANP-5cCPAK

CPAK

FIANP-5cCPAK

CPAK

FIANP-5cCPAK

CPAK

Fa

bric

• Slice architecture• Optics, NPU & FIA per slice

• Flexible pipeline NPU

• 1st stage of fabric on line card

• 3 Generations (so far)• 40G line cards w/ 15G NPUs (90 nm)

• 200-360G line cards w/ 60G NPUs (55 nm)

• 800G-1.2T line cards with 240G NPUs (28 nm)

• (at least) one more generation of NPU/LC

• TCAM for ACL/QoS scale

• Huge FIB (5M+) & buffers (200 msec)

BRKSPG-2404 35


ASR 9000 Modular Line Card Architecture

CPUDRAM

Fabric

FIANP-5c


FIANP-5c


Bay

0

Bay

1

2/4/8X 10GE 20X 10G

1/2X 40G20X 1G 1/2X100G

PH

Y

10G SFP+

10G SFP+

10G SFP+

10G SFP+

10G SFP+

10G SFP+

10G SFP+

10G SFP+

PH

Y

10G SFP+

10G SFP+

10G SFP+

10G SFP+

10G SFP+

10G SFP+

10G SFP+

10G SFP+

10G SFP+

10G SFP+

PH

Y

10G SFP+

10G SFP+

PH

Y

10G SFP+

10G SFP+

10G SFP+

10G SFP+

PH

Y

40G

QSFP+

40G

QSFP+

PH

Y

100G

CPAK

100G

CPAK

PH

Y

40G

QSFP+

PH

Y

100G

CPAK

PH

Y

1G SFP

1G SFP

1G SFP

1G SFP

1G SFP

1G SFP

1G SFP

1G SFP

1G SFP

1G SFP

MODULAR 3RD GENERATION LC

BRKSPG-2404 36


• 2x NP5c NPUs, “mini fabric”

• Full feature set from ASR9k

• direct inheritance of all functionality / config

• Dense hierarchical QoS

• Mixed set of interface speeds / optics

• MACSEC integrated into PHY

• full bandwidth deep packet buffers

• 2xQSFP28, 24xSFP+, 16xQSFP

ASR 9901 – ~450Gbps, 2RU full featured system

BRKSPG-2404 37

CPUDRAM

Fabric

FIA

Packet Buffers

NP

-5c

FIB

TCAM Packet Buffers

FIA

Packet Buffers

NP

-5c

FIB

TCAM Packet Buffers

QSFP(+,28)

40/100GE

SFP(+)

10GE / 1GE

x12

SFP

1GE

x8

QSFP(+,28)

40/100GE

SFP(+)

10GE / 1GE

x12

SFP

1GE

x8


• Dense 100GE with deep buffers

• 8 & 16-slot modular, 1 & 2 RU fixed

• Up to 57.6 Tbps in ½ rack in 2016

• 2.4T - 3.6T line cards

• Options for route and ACL scale via TCAM

• High scale 3.6T in 2018

• Highly integrated Silicon

• Single ASIC for forwarding & fabric interface

• Dramatic power reduction to 0.24 W/Gbps

• QSFP28 optics

• GDDR5 commodity buffers

NCS 5500 – 3rd generation 100G

BRKSPG-2404 38


• New chassis design for XR

• Orthogonal direct connect

• Horizontal line cards and vertical fabric

• Direct connection between line cards and fabric cards

• No midplane

• Distributed air intake between cards

• Fans cover fabric cards

• Provides cool air equally to all optics

• Optics require ~30C cooler operation than silicon

• Avoid preheating air from module to module

NCS 5500 System Design

REAR VIEW

AIR INLET

FAN

REMOVED

BRKSPG-2404 39


NCS 5500 Line Card Architecture

36X 100GE

• Slice architecture• Optics & Pipelined Forwarding ASIC per slice – integrated FIA

• Optional TCAM for FIB/ACL scale

• 1st Generation• 3.6T line cards FA at 600G

• 2.9T line cards FA at 720G + TCAM

• Deep VoQ buffers (50+ msec)

CPUDRAM

QSFP28

QSFP28

QSFP28

QSFP28

QSFP28

QSFP28

Forwarding

ASIC

Optics x 6 FA

Optics x 6 FA

Optics x 6 FA

Optics x 6 FA

Optics x 6 FA

Buffers

BRKSPG-2404 40


NCS 5501/5502 Architecture

NCS 5501 800G

• NCS 5501• 1RU – Single 800 Gbps Forwarding ASIC

• 4x 100G + 40x 10G

• NCS 5502• 2RU – 8 600Gbps Forwarding ASICs

• Integrated switch fabric

• 48x 100G

• Optional TCAM for scale

• Deep VoQ buffers (50+ msec)

SF

P+

SF

P+

SF

P+

SF

P+

SF

P+

SF

P+

Forwarding

ASIC

QS

FP

28

QS

FP

28

QS

FP

28

Bu

ffe

rs

40x10G 4x100G

TC

AM

CP

UD

RA

M

QS

FP

28

QS

FP

28

QS

FP

28

QS

FP

28

QS

FP

28

QS

FP

28

QS

FP

28

Forwarding

ASIC

QS

FP

x 6

FA

QS

FP

x 6

FA

QS

FP

x 6

FAB

uff

ers

TC

AM

48x100G

Fabric

CP

UD

RA

M

QS

FP

x 6

FA

QS

FP

x 6

FA

QS

FP

x 6

FA

QS

FP

x 6

FA

SwitchSwitch

NCS 5502 4.8T


• 1 RU Fixed System: 24x QSFP28 ports

• Base version only (no room for TCAMs or MACSEC...)

• 2x 900 Gbps Forwarding ASICs(1200:900 nominal oversubscription)

• No Fabric ASIC, Forwarding ASICs

are directly connected “back-to-back”

NCS55A1: 24 x 100GE

BRKSPG-2404 42


NCS 5000 PlatformsExtending IOS XR for satellite, ToR, and beyond

NCS 5011

NCS 5002

NCS 5001

• Maximizing Silicon capabilities• No external memories – small FIB, usec buffers

• All bandwidth to ports, no fabric

• Low cost and power

• Full IOS XR Routing

Forwarding ASIC32x100

Buffers

FIB

BRKSPG-2404 43


CRS-X

• Scaling CRS into 100G

• 4x 100G

• 40x 100G

• 2x 100G + 5x 40G

• 3x 100G + 1x 100G IPoDWDM

• Up to 51.2T via multi-chassis

• Fully compatible with CRS-1 & CRS-3

BRKSPG-2404 44


Virtualized IOS XR (“XRv 9000”)

• IOS XR on x86 hardware with Linux VM & containers

• Hosted & small PE with L2 & L3 VPNs

• Ideal route reflector• 64 bit

• 10+ M routes

• Router creation in seconds – rapid deployment

• Runs existing data plane forwarding code

• (recompiled) to an emulated NPU – maintains consistent behavior / features

BRKSPG-2404 45


Virtualized IOS XR – Performance & QoS

• Targeted for 5-50 Gbps forwarding

• Power per bit looks awful• except when it doesn’t really matter!

• 20+ Gbps with features per socket

• 3-layer Hierarchical QoS

• Scale horizontally

BRKSPG-2404 46


• The requirement space is pretty broad

• Optimizing for everything optimizes for nothing

• But we have to optimize for some things or no one buys it!

• And we can’t build an arbitrarily large suite of platforms

• So some of us are lucky enough to get paid to spend lots of

time sitting around arguing about: which memory technologies,

which silicon NPU architectures, which optical parameters,

which mechanical designs, and a bunch of other things to trade

off against each other...

Summary: Why not one platform?

BRKSPG-2404 47

I’m grateful for the opportunity to be here and to talk about something (at least I think) is interesting. It’s

important to me personally that attendees find these presentations helpful and/or valuable. Certainly not

everyone will enjoy every presentation -- but you guys spend a lot of money to come here and Cisco and

the speakers do a lot of work to put on the event and the presentations. If the material is excellent, we’ve

done our jobs. If there’s something we can do better, please let us know... I know you’re not supposed to on

the internet, but I really do read the comments. ;-)

Thanks for your time today.

Lawrence J Wobker, Cisco

February 2018

One serious thing before we go


Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKSPG-2404


• Please complete your Online Session Evaluations after each session

• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.

Complete Your Online Session Evaluation

http://ciscolive.com/Online

http://www.ciscolive.com/global/on-demand-library/


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Tech Circle

• Meet the Engineer 1:1 meetings

• Related sessions

BRKSPG-2404 52

Thank you

ios xr platform hardware - clnv.s3.amazonaws.com · •introduction – • building a forwarding...

Documents