real world fabricpath deployment at ibm data centers

© 2012 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialReal world FabricPath deployment at IBM Data Centers

Real world FabricPath Deployment at IBM Data Centers

1Cisco Confidential

Santiago FreitasCCIE#18776 (R&S / SP) Consulting Systems EngineerCisco-IBM Global Team [email protected]

Lasse LeegaardIT ArchitectAT&T [email protected]


What?

IBM has achieved tangible benefits by migrating its infrastructure from Catalyst 6500 to a Nexus 2000, 5000, 7000 Architecture.

IBM has adopted FabricPath on the Nexus 5K and 7K and MPLS L3VPN on the Nexus 7K.

The solution was extensively tested at Cisco ECATS.

FabricPath was a key differentiator when competing with Juniper.

We learned a lot from this deployment.

2


Session Objectives

At the end of the session, you should be able to:

Articulate to your customers the Business Benefits that IBM has achieved by migrating to a Nexus 2K/5K/7K Architecture.

Explain the reasons why they adopted FabricPath.

Understand the Tests performed to validate the solution before deployment.

Understand IBM’s future direction and how they plan to get there.

3


IBM Nordic Strategic Outsourcing

IBM SO provides outsourcing services that offer management of applications and other IT components in either an onsite or hosted arrangement.

Eight Data Centers located in Denmark, Sweden and Finland.

Serve around 200 customers‒ Some have dedicated infrastructures

‒ Over 100 served by a shared, multitenant

infrastructure

One of the company’s largest Integrated Market Teams (IMT) globally

4


Overview of the Network Infrastructure

5

SiSi SiSi SiSi SiSi

Up to 20 access switches

Service switchesFWSM/NAM/ACE

Server access block

Limit of 20 access switches is based on un-oversubscribed port density in the Core routers

Layer 2 trunks10G or Nx1G

SiSi

MPLS P routersMPLS layer

MPLS gateways

Customers

Telcos

InternetAS25384

Overall network structure

SiSi SiSi

Dedicated gateways hold 1 telcoShared gateways holds multiple telcosOne shared set hold direct customer connections on 1G and internet accessMPLS gateway: 6500, 7200 or 7300

MPLS route reflectors

VPLS PE routers

MPLS PE routers /Aggregation switches

SiSi

SiSi SiSi

SiSiSiSi

L2 trunk

MPLS LDP link

VLAN + IP

Access switch: 6500Service switch: 6500MPLS PE router / Aggregation switch: 6500VPLS PE router: 7600+ES20

MPLS P router: 6500 or 7600


Overview of the Network Infrastructure

6

~150 Cisco 6500/7600

~50 Cisco 7200/7300

~ 3000 VLANs and 290 virtual firewalls

~26000 Ethernet ports


One of the Access Blocks Reached EoLA portion of the shared infrastructure was approaching end of life

7

Cisco and AT&T performed EoL analysis.

Factual discussion: Vital to demonstrate the need for a full network refresh.

- 22 Access Switches- 4080 access ports

1G or 2x1G uplinks - 2 pair of FWSM in Service Switches


Hardware Refresh Options

8

Replace only of the parts that reached End of Life

Technology Refresh using Catalyst 6500

Technology Refresh using Nexus 2K/5K/7K

Does not solve the High Risks and Technology Limitations

Does not solve some of High Risks and Technology Limitations. Limited evolution - 10G uplinks and Resolves single point of failure issues

Solves all of High Risks and Technology Limitations

Hidden cost of line cards replacement within 3 years $2.440.430,00 (now) + $2.460.000,00 (2014) = $4.900.430,00

Total Rack Space – 390 RU

Total Power – 156 KW

Total cost - $5.378.635,00

Total Rack Space – 154 RU

Total Power – 60 KW

Total cost - $2.639.300,00


Business Benefits of the Nexus-based Solution

Significant OPEX savings when compared with the existing infrastructure:

‒ Reduced the power consumption by 61%

‒ Reduced the rack space used by network switches by 60%

‒ Reduced the number of managed devices in the network by 38.5% (from 26 to 16)

Easier way to scale, supports more access blocks on the same Core devices, therefore less expensive per customer port

Reduction in the time to onboard and configure the network for new customers

CAPEX savings – Next Generation DC based on Cisco Nexus and FabricPath was 46% cheaper than building similar architecture using Catalyst 6500

Why IBM chose to deploy Nexus and FabricPath

9


Juniper!!!

Like-for-like (EX8200/4500/4200 + MX routing + 6500/FWSM firewall)

Qfabric (Qfabric switching + MX routing + 6500/FWSM firewall)

No FCoE capable hardware

10G server density not impressive

FCoE development is beginning to catch up

However, Nexus has more/longer field exposure than Juniper kit in this area.

Organizational inertia and training would have to be overcome

Yes – we did consider doing it differently

10


What IBM actually deployed?

11

2x Nexus 7010

‒ M1/F1 combination

‒ MPLS L3 VPN PE

12x Nexus 5548UP

‒ Across 3 DCs

70x Nexus 2200

‒ 3360 access ports

2x 6500 Service chassis for FWSM modules

FabricPath

MPLS Backbone


L3L3

FabricPath Flexibility

Need more edge ports?

Need more bandwidth?

The Network Can Evolve With No Disruption

FabricPath FabricPathFabricPath

→ Add more leaf switches

→ Add more links and spines


Why IBM adopted FabricPath?

Better utilization of links

Increased Agility‒ New PODs and/or links for more capacity can be added non-disruptively

‒ Any VLAN anywhere

Simplicity of Configuration‒ Much simpler to implement and configure than vPC

Very fast convergence - sub-second in most cases

Need to route over the Fabric‒ Layer 3 over FabricPath

vPC and traditional STP topologies were considered

13


FabricPath enablement

install feature-set fabricpath

feature-set fabricpath

vlan 3865

mode fabricpath

spanning-tree mst configuration

name IBMMST02

revision 10

instance 1 vlan 1-2048

instance 2 vlan 204

interface Ethernet1/5

switchport mode fabricpath

Was that really it?

14

vpc domain 11

role priority 100

peer-keepalive destination 10.1.20.46 source 10.1.20.45

peer-gateway

auto-recovery

fabricpath switch-id 1000

fabricpath domain default spf-interval 50 50 50 lsp-gen-interval 50 50 50 root-priority 255 / 254 (N7K)

fabricpath switch-id 1


MPLS L3 VPN on Nexus 7000

Nexus 7010 as the MPLS L3VPN PE.

Customer VLANs mapped into VRF/VPN in the Aggregation Layer.

Remote Sites are 6500, 7600, 7300 and 7200, working well with the rest of the infrastructure.

Advantage over Juniper, extra layer required.

Works together with the rest of the infrastructure

15

Nexus 7010


Migration planHow to get from here to there (or from there to here depending on your point of view)

17

VPLS PE

MPLS P

FW/LB service +Access

MPLS PE +Aggregation

FabricPathVLANsVLANsVLANs

L3 L3


ECATS End of Test Report

36 Major Tests Areas

Detailed Results

DDTS/Bugs Found and workarounds

Technical Notes

Convergence Summary Table

HW and SW utilized

Lessons Learned

Configuration Files

18

See Additional Resources Slides for link to it

Cisco Enhanced Customer Aligned Testing Services - http://ecats


Migration planHow to get from here to there (or from there to here depending on your point of view)

19

VPLS PE

MPLS P

FW/LB service +Access

MPLS PE +Aggregation

FabricPathVLANsVLANsVLANs

L3 L3


ECATS End of Test Report

36 Major Tests Areas

Detailed Results

DDTS/Bugs Found and workarounds

Technical Notes

Convergence Summary Table

HW and SW utilized

Lessons Learned

Configuration Files

20

See Additional Resources Slides for link to it



ECATS testing experience

Vital on the success of this deployment.

Gives us experience before having used it

Test overlap with rollout

Reduction of risk of introducing new technology


21


Testing Topology

22

MP-BGP Peering

between PEs

Core1 Nexus 701010.53.234.16610.53.234.167

L2/L3 AggregationMPLS PE

BGP / ISIS /MP-BGP / LDP

Access1 Nexus

5548UP10.53.234.162

Nexus 2248

Access3 Nexus

5548UP10.53.234.164

Access4 Nexus

5548UP10.53.234.165

Nexus 2248Nexus 2248 Nexus 2248

Core2 Nexus 701010.53.234.16810.53.234.169


BGP / ISIS / MP-BGP / LDP

F1 ports F1 ports

CE1-285110.53.234.173

L3 BGP / OSPF connections

L3 - 10GMPLS/LDP

enabled linksISIS

M1 ports M1 ports

Core3 760010.53.234.170



Core4 760010.53.234.171



L3 - 10GMPLS/LDP

enabled linksISIS

Access5 650010.53.234.172

L2 Access Layer

L21G

FEX 100 Po20 FEX 100 Po20 FEX 100 Po20 FEX 100 Po20

1 2 1 2 1 2 1 2

2/13 2/13 2/13 2/132/14 2/14 2/14 2/14

1/1 1/11/2 1/21/1 1/2 1/1 1/21/3

1/4

1/3

1/4

1/3

1/4

1/3

1/4

Po 10

Po 10

Po 10

Po 10

100/1/47

100/1/48

100/1/48

100/1/47

IXIA4/1

IXIA 4/2

IXIA4/4

100/1/47

100/1/48

IXIA4/5

IXIA4/6

100/1/47

100/1/48

IXIA4/7

IXIA4/8

IXIA4/9

IXIA4/10

Description:Title:

Updated:

Filename:

Minimal-Testing-topology-FabricPathv19.vsdProposed topology for ECATS Nexus 7K/5K 7600 testingIBM Voyager13/03/2012

Author: mraines

9/1 9/2

9/47 9/48

3/1 4/1 3/1 4/13/2 3/2 4/2

1/1 1/12/1 2/1

CE2-282110.53.234.174

3/9 3/9

4/9 4/9

Note any IP addresses shown are for management purposes only.

9/1 9/2 9/19/2

8/18/1

9/3 9/3

9/4 9/4

F1 ports VPC+ peer link

Gi 0/1 Gi 0/1

100/1/1 100/1/1

100/1/46IXIA4/11

100/1/46IXIA4/13

IXIA4/14

IXIA4/15

Gi 0/1

100/1/2

CE3-285110.53.234.167

100/1/2

CE4-285110.53.234.169

Gi 0/1

BGP OSPFBGP OSPFL3 BGP / OSPF connections

100/1/10

100/1/10

100/1/10

100/1/10

LACP LACP

Switch 1 Switch 2

LACP

Switch 3

3/31

3/31

4/2

L2-10Gbs FabricPath Enabled Links

Access2 Nexus

5548UP10.53.234.163

IXIA4/3

IXIA4/16

10.53.234.146 10.53.234.147

10.53.234.148

FaE 0/0/0 FaE 0/1/8 FaE 0/1/8FaE 0/0/0FaE 0/3 FaE 0/3

FaE 0/3

Gi 0/1Gi 0/2

FaE 0/1FaE 0/2 FaE 0/1 FaE 0/2

Remote Site

‒ Agg/MPLS PEs (7600)

- L2/L3 Aggregation

- ISIS / MP-BGP / LDP

‒ Access Layer Cat6500 (Layer 2)

ISIS and MPLS in the core

Site Under Test

‒ Nexus 7010 as Agg/MPLS PE (L2/L3)

‒ vPC+ at the Core for Active/Active HSRP

‒ Nexus 5548UP/Nexus 2248 as Access

‒ FabricPath

‒ Servers attached with vPC+

‒ OSPF/BGP over FP


Testing Topology and Scale Numbers

Access Layer‒ Nexus 5548UP – NX-OS 5.1(3)N1(1a)

‒ FEX Nexus 2248

Core‒ Nexus 7010 – NX-OS 5.2(3a)

‒ 2x M1 8x 10GE (N7K-M108X2-12L)

‒ 2x F1 32x 1/10GE (N7K-F132XP-15)

Remote Site PE‒ 7609 with RSP-720 – IOS 15.1(1)S

Hardware and Software Versions and Scale Numbers

23

For YourReference

300 VLANs

300 SVIs and 300 HSRP

200 VRFs / MPLS L3 VPN

3000 MAC addresses injected

IMIX Ethernet Traffic

‒ 4Gbps within Nexus Access Block (East – West)

‒ 800Mbps towards remote site (North-South)

‒ A full mix of bi-directional traffic paths (Inter-VLAN, Intra-VLAN, Inter-VRF)


Convergence Times

Layer 3 Link Failure on Core towards Remote site – 64 ms / 30 ms on Recovery

M1 Line Card Failure on Core - 950 ms (North-South) / 75 ms on Recovery

Fabric Path Link Failures (multiple tests) – 117 ms / 241 ms on Recovery

F1 Line Card failure on Core - 1380 ms / 319 ms on Recovery

Core Node Failure (power off N7010) - 2584 ms / 2703 ms on Recovery

Access Node Failure - 316.52 ms for vPC+ attached servers / 181 ms on Recovery

Failover Test Result Convergence Summary

24

Sub-second on FabricPath link failures


Dynamic Routing Protocol and FabricPathYou can run OSPF and BGP over FabricPath, you can’t over vPC

25

The OSPF CE routers CE-3 and CE-4 were configured with “ip ospf priority 0” interface configuration so they don’t participate in DR/BDR election process

FULL OSPF neighborships are formed with both Core1 and Core2

Traffic still forwarded even when crossing peer-link

FabricPath doesn’t have same limitations as vPC

CE3-2851-RK18#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface

10.10.101.5 1 FULL/BDR 00:00:36 10.10.101.5 GigabitEthernet0/1

10.10.101.7 1 FULL/DR 00:00:33 10.10.101.7 GigabitEthernet0/1

10.10.101.8 0 2WAY/DROTHER 00:00:30 10.10.101.8 GigabitEthernet0/1

CE3-2851-RK18#


Technical Lessons Learned

No Show Stopper DDTS

‒ One cosmetic, one catastrophic but with an easy workaround (already fixed) and one Unreproducible.

Several Technical Lessons Learned on the areas of:

‒ Peer-Link Failure and vPC+ attached devices

‒ MAC Learning with vPC+ domain

‒ Multidestination tree and vPC+

‒ MAC Learning on N7K with M1/F1 for L2 Traffic

It would be a session on its own…

26

Details on the hidden slides and on Additional Resources page


Further developmentsWhere do we see the rest of the infrastructure go?

27

28

SAN A SAN BSAN A SAN BSAN A SAN BSAN A SAN BSAN A SAN B

Up to 20Access Switches

Services Switches(FWSM/ACE/NAM)









MPLS PE/Agg Switches


MPLS P Routers

VPLSPE

VPLSPE

Layer 3 / MPLS

Layer 2IPv4

Layer 2IPv4

Layer 2IPv4

Layer 2IPv4

Layer 2IPv4

Evolution Plan

Evolution Plan

29

SAN A SAN BSAN A SAN B













MPLS P Routers

VPLSPE

VPLSPE

Layer 2IPv4

Layer 2IPv4

Layer 2IPv4

Layer 2IPv4

Layer 2IPv4

SAN A SAN B SAN A SAN BSAN A SAN B

IPv4/IPv6 IPv4/IPv6

Layer 3 / MPLS

Layer 2IPv4/ IPv6

SAN A SAN B

1 2

3 4

1 2

3 4

StorageFC/FCoE/NAS

StorageFC/FCoE/NAS

Dynamic Infrastructure2^12 = 4096 VLANs…2^24 = 16777216 Segment IDs

ManagementOrchestrationProvisioning Automation


Key Takeaways

The Key Takeaways of this presentation were:

IBM has achieved OPEX and CAPEX savings by migrating to a Nexus 2K/5K/7K Architecture in their Data Centers.

IBM has adopted FabricPath and is very happy with its Flexibility, Easy to Implement and Use and Convergence Time.

FabricPath was extensively tested and validated at Cisco ECATS.

FabricPath and MPLS on N7K were differentiators against Juniper.

You can reuse the lessons learned and additional resources available from this deployment to position FabricPath to your customers.

30


Additional Resources You can find the following additional information on the

link below

‒ Customer Requirements and Business Case for Catalyst 6500 -> Nexus and FabricPath

‒ Joint Technical Plan of Record (test requirements)

‒ Detailed Test Plan

‒ Complete end of Test Report (including detailed test results and configurations)

‒ Lessons Learned Presentation

‒ INTERNAL Case Study of IBM Nordic Adoption of Nexus and FabricPath

‒ EXTERNAL version of the Case Study

31

http://bock-bock.cisco.com/wiki/User:Safreita:FabricPath_Testing

real world fabricpath deployment at ibm data centers

Documents