creating the network at cisco live orlando: technical case study techadvantage webinar

Cisco TechAdvantage Webinars Creating the Network at Cisco Live Orlando: Technical Case Study

Patrick Warichet: Core Design and Switching Backbone Ryan D’Souza: Wireless Design Joe Clarke: Network Management Jason Davis: Network Management and Automation

We’ll get started a few minutes past the top of the hour.

Note: you may not hear any audio until we get started.

© 2013 Cisco and/or its affiliates. All rights reserved. PNLNMS-3000 Cisco Public

•  Submit questions in Q&A panel and send to “All Panelists” Avoid CHAT window for better access to panelists

•  Please complete the post-event survey

•  For WebEx audio, select COMMUNICATE > Join Audio Broadcast

•  Where can I get the presentation? Or send email to: [email protected]

•  For WebEx call back, click ALLOW phone button at the bottom of participants side panel

Housekeeping

•  Join us for upcoming TechAdvantage Webinars: www.cisco.com/go/techadvantage


Patrick Warichet Technical Marketing Engineer

[email protected]

Ryan D’Souza Network Consulting Engineer

[email protected] Twitter: @ryanstevedsouza

Introductions

Joe Clarke Distinguished Services Engineer

[email protected]

Jason Davis Distinguished Services Engineer

[email protected] Twitter: @SNMPguy


The CiscoLive! Network Our Team

4

CiscoLive! Network – Core / Data Center Design Orlando, June 23 - 27, 2013

Patrick Warichet, CCIE #14218

Technical Marketing Engineer – Network Operating Systems Technology Group


Backbone Architecture

6

Sunnyvale

Denver

AS 209 AS 209

AS 64726

63.231.216/21 2001:428:CE01::/48

Orlando

Bandwidth Provided by CenturyLink

IPv4 Address Range - 63.231.216.0/21 IPv6 Address Range – 2001:428:CE01::/48

Higher Weight for all Local addresses


BGP Route Validation with SoBGP

7

Colo Denver

Orlando Cisco Live 2013

Colo Sunnyvale

RPKI Validator Cache

Location: Switzerland

IPv4 and IPv6 RPKI Router-Server Protocol ASR1k

ASR1k

Cat6509 Quad Sup SSO

4 RIPE based servers were installed


SoBGP Prefix verification IPv4 and IPv6

8

8

JSV-ASR#show bgp ipv4 unicast summary BGP router identifier 66.77.8.142, local AS number 64726

Path RPKI states:5268 valid, 1742953 not found, 7516 invalid <SNIP> BGP activity 706942/244807 prefixes, 5078283/3255957 paths, scan interval 60 secs Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 63.231.216.9 4 64726 65062 65067 7695624 0 0 3d20h 3 66.77.8.129 4 209 702344 17230 7695607 0 0 1w3d 449040 66.77.8.130 4 209 706159 17211 7695607 0 0 1w3d 449039 66.77.8.150 4 64726 697436 750500 7695624 0 0 1w1d 3

JSV-ASR#show bgp ipv6 unicast summary BGP router identifier 66.77.8.142, local AS number 64726

Path RPKI states: 1370 valid, 24658 not found, 130 invalid <SNIP> BGP activity 707057/244951 prefixes, 5079057/3256907 paths, scan interval 60 secs Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 2001:428:CE00:1:66:77:8:129 4 209 108934 17232 115745 0 0 1w3d 13078 2001:428:CE00:1:66:77:8:130 4 209 109133 17229 115745 0 0 1w3d 13078 2001:428:CE01:0:63:231:216:9 4 64726 65155 98282 115746 0 0 3d20h 1


Internet Statistics

9

940 Mbps Traffic

Sun Mon Tue Wed

Link To Denver Colo

Link To Sunnyvale Colo

615 Mbps Traffic

Thu


IPv4 Traffic

10

220.2K PPS

74.1K PPS

Link To Denver Colo

Link To Sunnyvale Colo


IPv6 Traffic

11

12.7K PPS

Approximately 8 % of the traffic was IPv6


Multi-Hop FCOE

§ UCS release (2.1.1a) provides support for multi-hop FCOE capability –  “FCOE uplink port” supported on both FI 6100 and 6200 (native ports + extension

module ports for both models)

§  Nexus 7000 is a Director Class FCOE platform: supports FCOE with F1 (NX-OS 5.2), F2 (NX-OS 6.1.1) and F2E (NX-OS 6.1.2) series line cards: –  F1 with SUP1 or SUP2/SUP2E –  F2/F2E with SUP2 or SUP2E (SUP1 not supported in this case) –  ETH or FCOE ports on F1/F2/F2E (no FC port)

UCS-B with Director Class FCOE Nexus 7000

12


FCoE with N7004 and UCS-B – Problem Statement

13

Unified FCoE in VPC

IP traffic in VPC

FCoE traffic in a Port-Channel

Unified FCoE in VPC

IP traffic in VPC


Cisco Live 2012 San Diego Design Cisco Live 2013 Orlando Design

Design Can’t work! VPC requires Port-Channel interfaces in a Port-Channel that cannot be shared

N5K-1 N5K-2 N7K-1 N7K-2

Netapp-A Netapp-A Netapp-B Netapp-B


FCoE with N7000 and UCS-B – Design Solution

14

Unified FCoE in VPC

IP traffic in VPC


Separate Storage and IP Networking completely

Storage VDC Storage VDC

Downside: Not Unified Extra cabling

Create a storage VDC on Nexus 7000 Storage VDC = SAN VDC allocated to run as a virtual MDS FC switch

CiscoLive! Network – Switching Backbone Orlando, June 23 - 27, 2013


Network Backbone at Previous Events Routing, Switching and Services in the Cisco Live Network : design and topology has evolved through various events

16

2011 2012 2013

InteropNet, Las Vegas CiscoLive!, London CiscoLive!, Orlando

CiscoLive!, Las Vegas CiscoLive!, San Diego

InteropNet, New York

CiscoLive!, Mexico


Catalyst 6500 Equipment Inventory - Core

17

§  2 x 6509-E

§  4 x Supervisor 2T §  VS-SUP2T-10G with VS-F6K-PFC4XL

§  40 Gigabit Ethernet §  2 x WS-X6904-40G


§  1 Gigabit Ethernet Fiber §  2 x WS-X6848-SFP

17


Catalyst 6500 Equipment Inventory - Distribution

18

§  2 x 6513-E and 2 x 6504-E

§  4 x Supervisor 2T §  VS-SUP2T-10G with VS-F6K-PFC4



§  1 Gigabit Ethernet Fiber §  2 x WS-X6848-SFP

§  1 Gigabit Ethernet Copper §  2 x WS-X6848-GE-TX

18


Staging

19


Cisco Live Orlando 2013 Topology

21

OCCC – SmartCity Distribution

OCCC – Cisco Live Distribution

OCCC – World of Solutions

San Jose / Cisco CoLo (CenturyLink)

2 x UCS 6248UP

4510R+E (Peabody)

Peabody Hotel Orlando

2 x Nexus 7004

12 – 3750G Switches

Floors 1,2,3,4

55 - 3560CG Switches 125 – 3602E Access Points

35 - 3560-E Switches

Floors 1,2,3,4

6504-E SUP2T VSS

Wireless Control (10 x 5508)

18 – 3750-X Switches

Floors 1,2,3,4

12 – 3750-X Switches

Halls A - D

6513-E SUP2T

VSS

Ser

vice

P

rovi

der

Pea

body

MD

F

OC

CC

MD

F

Pea

body

IDF

OC

CC

IDF

OC

CC

IDF

OC

CC

IDF

Mee

ting

Roo

ms

Eve

nt

Cen

ter

Eve

nt

Cen

ter

Mee

ting

Roo

ms

2 x UCS 5108

110 - 3560CG Switches

150 – 3602E Access Points

75 - 3560CG Switches 177 – 3502P Access Points

- 3560CG Switches 130 - 3602E Access

Points

ASR 1006

Denver / Cisco CoLo (CenturyLink) ASR 1006

2 x ASA 5585-X NAT64

6513-E (SmartCity)

6509-E SUP2T QUAD SUP VSS CORE

DISTRIBUTION

4 x 40GE

ASA 5585-X Internet Firewall

2 x ASA 5585-X Data Center Firewall

2 x NetApp FAS3240

2 x NetApp DS2246


Core & Distribution - Features and Highlights

22 22

§  VSS Quad-Sup SSO and Multi-chassis Etherchannel (10 / 40 GE links)

§ Routing protocols - OSPF and BGP for IPv4 and IPv6

§ Network Management - SNMPv3,

Control Plane Policing, Syslog, etc.

§  Flexible NetFlow on Sup2T for IPv4 & IPv6 traffic

§ Multicast

22

1 Terabyte of Multicast traffic during the show Vlan361 is up, line protocol is up <SNIP> L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 741428621 pkt, 1009825781802 bytes L3 out Switched: ucast: 5 pkt, 590 bytes - mcast: 0 pkt, 0 bytes 741429788 packets input, 1006860549402 bytes, 0 no buffer


40 Gigabit Ethernet with a 160GE Backbone

23

§ WS-X6904-40G-2T / XL in the Network

Snapshot from ‘show version’ on the Core switch: 4 Virtual Ethernet interfaces 108 Gigabit Ethernet interfaces 116 Ten Gigabit Ethernet interfaces 4 Forty Gigabit Ethernet interfaces

§  Split-brain mode on Distribution switch (10GE/40GE mixed mode)

40GE ports to the Core, and 10GE to the Access on the same module hw-module switch 1 slot 1 operation-mode port-group 2 TenGigabitEthernet hw-module switch 2 slot 1 operation-mode port-group 2 TenGigabitEthernet

§ Multi-chassis Etherchannel between Core and Distribution VSS

23

CORE SWITCH 6509E

SUP2T-XL QUAD SUP VSS

DIST 6509E

SUP2T VSS 10GE / 40GE

DIST 6509E

SUP2T VSS 10GE

4x40GE 4x10GE


Core – Catalyst 6509-E Quad Supervisor VSS

24

VSS Quad SUP SSO

10GE

Si Si

LACP or PagP LACP

Monitoring Server

Access Switch or ToR or Blades

Traditional VSS

10GE

Si Si

LACP or PagP LACP

Monitoring Server

Access Switch or ToR or Blades

Simplified Network Design •  Spanning Tree and first-hop redundancy

protocols eliminated •  Single touchpoint manageability

Double Bandwidth Utilization •  With active-active multi-chassis

EtherChannel (LACP/ PagP) •  1+1 Supervisor redundancy for

dual-attached devices

Deterministic and Automated Recovery

•  Maximize network throughput with VSS quad Sup SSO

•  1:1 (active/standby) supervisor redundancy for single and dual attached devices

SSO Sync SSO Sync

SS

O S

ync

SS

O S

ync

NO Sup2T

Support 15.1(1)SY1 YES

Sup720


In Chassis Redundancy Domain


VSS Quad-Sup SSO Redundancy Domains

VSS Domain

In Chassis Active (ICA)

In Chassis Standby (ICS)


In Chassis Standby (ICS) VS Switch ID 1 VS Switch ID 2

Default Redundancy Domain

SSO VSS Active VSS Standby

SSO SSO




VSS Quad-Sup SSO Redundancy Domains

VSS Domain


In Chassis Standby (ICS) (Standby Hot Chassis)


In Chassis Standby (ICS)

(Standby Hot Chassis) VS Switch ID 1 VS Switch ID 2

Default Redundancy Domain

SSO VSS Active (Active)

VSS Standby Hot (Switchover Target)

SSO SSO


IPv4 and IPv6 Statistics

27

15.7K Unique IPv4 Mac Address

7.81K Unique IPv6 Mac Addresses

Sun Mon Tue Wed Thu


Goodies Catalyst 6500 Twitter Script

28

Tweet information from IOS using EEM and Twitter's API Follow it CiscoLive NOC @CiscoLive2013 Download the code from https://supportforums.cisco.com/docs/DOC-19363

CiscoLive! Network – Wireless Design Orlando, June 23 - 27, 2013

Ryan D’Souza

Network Consulting Engineer – Mobility and Wireless Practice, Advanced Services


Key Statistics

30

•  Convention Coverage Area: ~ 2.5 million sq.ft

•  Most Total Attendance ever : 20,000+ registered attendees

•  Highest Total unique associations: ~19,500

•  Largest simultaneous associations ever for Cisco Live: ~13,100 •  ~5500 concurrent associations during keynote

•  Highest ever recorded Aggregate Wi-Fi traffic: ~3.0 TB

•  Peak Wi-Fi throughput:~900 Mbps in, ~200 Mbps out

•  Client Distribution by Band: 43.7% 5 GHz (11n) 53.5% 2.4 GHz (11n)


CiscoLive! Wireless Network

•  10 x 5508 Controllers to manage a total of 673 Access Points across the venue

•  180 x 3502P Access Points with Air-ANT25137NP-R stadium antennas to cover Attendee Dining, Keynote and World of Solutions.

•  300 x 3602 Access Points with AIR-ANT2566P4W-R to cover the hallways and session rooms in OCCC


CiscoLive! Wireless Network •  4 x MSE 7.5 for Location Analytics, Ekahau RFID tags and show statistics

•  1 x CPI 1.4 to manage the 7.5 MSE

•  110 x 3602 APs to cover the hallways and session rooms in Peabody

•  83 x in house APs were used for some session room coverage in OCCC


Client Statistics

NOC Network Management Orlando, June 23 - 27, 2013

Joe Clarke, CCIE #5384

Distinguished Services Engineer – Global Technical Center


Built For Management

§  All devices managed with SNMPv3 authPriv (SHA / DES) §  Prime NAM used to traffic volume and utilization

§  Flexible NetFlow sent from the 6500 core and distribution switches to FreeBSD VM “exploder” which forwarded to other collectors (Prime Infrastructure, Prime NAM, StealthWatch, and Plixer [WoS])

§  Syslog sent from all devices to FreeBSD then forwarded to interested parties

Netflow Syslog


Cisco Prime LMS - Provisioning

37


Embedded Automation Using EEM To Set Port Descriptions

38

EEM dynamically sets descriptions on ports based on last connected CDP neighbor. Applet available from https://supportforums.cisco.com/docs/DOC-24529


Embedded Automation Converting Access Switches From Stage to Production

39

Switch from stage to production

VLAN

Ping production default GW, and verify reachability

If reachable, reconfigure switch for production…

…If not, then revert to stage.


Traffic Breakdown Courtesy of Cisco Prime Network Analysis Module

Video Streaming Wins!

NOC Network Management and Automation Orlando, June 23 - 27, 2013

Jason Davis

Distinguished Services Engineer – Advanced Services


Intelligent Automation Use at CiscoLive! NOC Cisco Service Portal – Cisco Process Orchestrator – Cisco Prime

42


Intelligent Automation to Monitor Network Device and Server Reachability

43


Prime Infrastructure and NCCM Monitoring for Faults and Config

44


Cisco Prime Network Registrar DNS / DHCP Statistics – Monday 3pm to Thursday Noon

45

Max Active DHCP Leases 33959 @ Wed 14:50 Total – 690M requests

Total – 122M DNS lookups

1 Regional 1 Authoritative 2 DHCP

2 cDNS 2 DNS Secondaries


How Reliable Was It? How Many Nines Are Enough?

What Will You Connect?


•  Thank you! •  Please complete the post-event survey •  Join us for upcoming webinars:

Register: www.cisco.com/go/techadvantage Follow us @GetYourBuildOn

creating the network at cisco live orlando: technical case study techadvantage webinar

Technology

cisco andor

cisco public6

cisco public7

cisco public88

cisco public9

cisco public10

cisco public11

cisco public12