Whataburger Network Stabilization Single Point Of Failure Optimization

Objective

• To improve overall performance of the Whataburger network - Three areas are being addressed:

– Sprint Network

– Network Connectivity at the Units

– Internal Network Design at ADC

History

• Over the last 7 months…– Two major business impacts due to Fortigate firmware

– Six major Sprint MPLS outages

– Two major local fiber cuts (one was a single fiber cut, the other was both main network connections to ADC)

– Outage with Cisco core equipment at ADC

Whataburger Network Design Overview

Corpus Christi, TX(Corporate HQ)

Irving, TX(East Division)

Austin, TX(Data Center)

OC-3

OC-3

OC-3

SPRINT GLOBAL MPLS VPN

(Fully MeshedAny to Any connectivity ))

OC-3

San Antonio,TX(West Division)

OC-3

7206VXR 7206VXR

OC-3

7206VXR

7206VXR

SBC/AT&T Frame

Approx. 100 restaurants

Approx. 450+ restaurants

CDMA Data LinkWWAN

Sprint Network

• Whataburger Network Single Points of Failure– Austin Data Center

• Two local circuits from Sprint’s Point Of Presence (POP) to ADC follow common terrestrial path and enter ADC at single entry point

– Corpus Christi Data Center• Two local circuits from the CCDC and Sprint’s Point Of

Presence (POP) follow common terrestrial path

– Remote Offices (Irving & San Antonio) have only one network connection

Sprint Network

• Austin Data Center– Two local circuits from Sprint’s Point Of Presence (POP)

to ADC follow common terrestrial path and enter ADC at single entry point

• Plan – One of the two Sprint local circuits will be re-engineered using a

separate path to the ADC

– The new local circuit will enter the Austin Data Center (Data Foundry) at a physically separate location of facility

– ADC network connections will be terminated at two geographically separate Sprint MPLS switch sites, insuring that a catastrophic switch failure at one site would not result in a total network outage

Sprint Network

• Corpus Christi Data Center– Two network connections between CCDC and Sprint

follow common terrestrial path• Plan

– CCDC network connections will be terminated at two geographically separate Sprint MPLS switch sites, insuring that a catastrophic switch failure at one site would not result in a total network outage

• Issues– There is only one local circuit carrier in the city of Corpus Christi

(AT&T)

– Single local circuit terrestrial route and single entry point into headquarters could be cost prohibitive

Sprint Network

• Remote Offices (Irving & San Antonio) have only one network connection– Plan

• If the business feels that redundancy is required at both locations, a 2nd circuit can be installed

• Additional network connection will be terminated at geographically separate Sprint MPLS switch site, insuring that a catastrophic switch failure at the site would not result in a total site outage

Network Connectivity at Units

• Fortigates– Configured to point to only one router at ADC

• Plan – With new network design for Restaurant Technology, Cisco 1841

routers will replace Fortigates and be “dual-addressed” to both routers at ADC

• Network– Single point of failure with one network connection

• Plan – A second line is being installed at each Unit to improve

performance and add redundancy. The new DSL circuit will be the primary network connection and the wireless network will be the backup

Internal Network Design at ADC

10.200 .200.1/30

G0/310.200.200 .2

G0/3

G0/110.200.201.248/24

G0/110.200.201.247/24

G0/2G0/2

2/11

2/12

2/13

2/14

2/15

2/16

2/11

2/12

2/13

2/14

2/15

2/16

1/48 1/47 1/481/47

1

2

3

4

5

6

FANSTATUS

Power Supply 1 Power Supply 2

Catalyst 6500 SERIES

WB-ADC-6506-1

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

RUNINSTA

LL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

RUNINSTALL

WS-SUP720

SUPERVISOR 720 WI TH INTEGRATED SWITCH FABRI C

SYSTE

M

STATUS

ACTIVE

PWR

MG

MT

EJECT

DI SK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

LINK

LINK

LINK

RESET

STATUS

WS-X6748-GE-TX47

48

37

3 8

35

36

25

26

2 3

24

13

14

11

12

1

2

4 8 P O R T

12119 107 85 63 41 2 242321 2219 2017 1815 1613 14 363533 3431 3229 3027 2825 26 484745 4643 4441 4239 4037 38

10/100/1000GE MOD

FABRIC ENABLEDRJ45

STATUS

WS-X6748-GE-TX47

48

37

3 8

35

36

25

26

2 3

24

13

14

11

12

1

2

4 8 P O R T

12119 107 85 63 41 2 242321 2219 2017 1815 1613 14 363533 3431 3229 3027 2825 26 484745 4643 4441 4239 4037 38

10/100/1000GE MOD

FABRIC ENABLEDRJ45

STATUS

WS-X6516A-GBI C

LINK

1 2LIN

KLIN

K

3 4LIN

KLIN

K

5 6LIN

KLIN

K

7 8LIN

KLIN

K

9 10LIN

KLIN

K

11 12LIN

KLIN

K

13 14LIN

KLIN

K

15 16LIN

K

3

4

1

2

5

6

7

8

9

10

11

12

13

14

15

16

S TATUS PHONE

WS-X6548-GE-TX 4748

373 8

3536

2526

2 324

1314

1112

12

4 8 P O R T

12119 107 85 63 41 2 242321 2219 2017 1815 1613 14 363533 3431 3229 3027 2825 26 484745 4643 4441 4239 4037 38

10/100/1000 BASE-TETHERNET

SWITCHING MODULE

STATU

S

F IREWAL L SERVICES MODULE

DO NOT REMOVE CARD WHI LESTATUS L ED IS GREEN OR

DISK CORRUPTION MAY OCCUR

WS-SVC -FWM-1

SHUT DOWN

1

2

3

4

5

6

FANSTATUS

Power Supply 1 Power Supply 2

Catalyst 6500 SERIES

WB-ADC-6506-2

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

RUNINSTALL

200-240 V23 A

60/50 Hz

INPUTOK

FANOK

OUTPUTFAIL

RUNINSTALL

WS-SUP720

SUPERVI SOR 720 WITH INTEGRATED SWITCH FABRI C

SYSTEM

STATUS

ACTIVE

PWR

MG

MT EJECT

DI SK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

LINK

LINK

LINK

RESET

STATUS

WS-X 6748-GE-TX47

48

37

3 8

35

36

25

26

2 3

24

13

1 4

11

12

1

2

4 8 P O R T

12119 107 85 63 41 2 242321 2219 2017 1815 1613 14 363533 3431 3229 3027 2825 26 484745 4643 4441 4239 4037 38

10/100/1000GE MOD

FABRIC ENABLEDRJ45

STATUS

WS-X 6748-GE-TX47

48

37

3 8

35

36

25

26

2 3

24

13

1 4

11

12

1

2

4 8 P O R T

12119 107 85 63 41 2 242321 2219 2017 1815 1613 14 363533 3431 3229 3027 2825 26 48

4745 4643 4441 4239 4037 38

10/100/1000GE MOD

FABRIC ENABLEDRJ45

STATUS

WS-X6516A-GBI C

LINK

1 2LIN

KLIN

K

3 4LIN

KLIN

K

5 6LIN

KLIN

K

7 8LIN

KLIN

K

9 10LIN

KLIN

K

11 12LIN

KLIN

K

13 14LIN

KLIN

K

15 16LIN

K

3

4

1

2

5

6

7

8

9

10

11

12

13

14

15

16

STATUS PHONE

WS-X 6548-GE-TX 4748

373 8

3536

2526

2 324

131 4

1112

12

4 8 P O R T

12119 107 85 63 41 2 242321 2219 2017 1815 1613 14 363533 3431 3229 3027 2825 26 48

4745 4643 4441 4239 4037 38

10/100/1000 BASE- TET HERNET

SWITCHING MODULE

STATUS

F IREWAL L SERVICES MODULE

DO NOT REMOVE CARD WHI LEST ATUS LED IS GREEN OR

DISK CORRUPTION MAY OCCUR

WS-SVC-FWM-1

SHUT DOWN

0

7206

2

4

6

1

5

3

PACKET OVER SONET/S DH

ENAB

LED

RX C

ARRI

ER

RX

PACK

ETS

BN

155-

MM

RX TX

CONSOLE

LIN K

TX GBICRX

EN

GIGABIT ETH ERN ET 0/1

RJ 45

LINK

TX GB ICRX

EN

GIGAB IT ETHER NET 0/2

R J45

LINK

TX GBICR X

EN

GIGA BIT ETH ERN ET 0/3

RJ45 COMPAC T FLASH

C PUR ESET

NETWOR K PROCESSING EN GINE – G1 SLOTA CTIVE

POWERON

A UXNPE-G1

0

7206

2

4

6

1

5

3

P ACKET OVER SONET/SDH

ENAB

LED

RX C

ARRI

ER

RX

PACK

ETS

BN

155-M

M

RX TX

CONSOLE

LIN K

TX GBICRX

EN

GIGAB IT ETH ERN ET 0/1

RJ45

LINK

TX GB ICR X

EN

GIGA BIT ETHER NET 0/2

R J45

LINK

TX GBICRX

EN

GIGAB IT ETH ERN ET 0/3

RJ45 COMPA CT FLA SH

C PUR ESET

N ETWORK PROC ESSING ENGINE – G1 SLOTA CTIVE

POWERON

AU XNPE -G1

WB-ADC-7206-1 WB-ADC-7206-2

10.200.200.253/2510.200.200.254/2

5

Switch X-Connects

Servers

Sup 720

Servers

• Internal network design review to be conducted to ensure 100% redundancy (to address Cisco Core architecture)– Plan

• Cisco Engineering is currently performing analysis and will develop recommendations, if required

Next Steps

• Initiate Network Diversity Order Request- For CCDC - Email request to Sprint account team

- For ADC - Sign diversity request document (PDF)

• Submit Project Work Plan• Develop WBS• Plan/Schedule ADC Site Walk Thru• Start Project Execution• Monitor Project Milestones/Deliverables• Report Project Status/Progress

Detailed Network Configuration

Current / Proposed Design

San Antonio & Irving OC-3 Connections

Plano MAN1262.003

MAN 3203

MAN node19232

IrvingPL 542692

IrvingPOP (4485)

Atlanta 2140 IP Node

SPRINT GLOBAL MPLS

VPN

MAN1040.006

SONETRing 60

MAN node17131

San AntonioPOP (5133 )

SONETRing 27.5

HoustonIP Node 2290SL-PE10-HOU

Austin POP4440

SPRINT GLOBAL MPLS

VPN

San AntonioPL 542700

San Antonio & Irving OC-3 connections

Ft. Worth 2040IP Node

SL-PE30-FW

San AntonioPOP (4540 ) Ft. Worth

2040

San Antonio/Irving Details / Current Design

• San Antonio, TX (West Division)

- Fiber cable run thru conduit terminating into OC-3 line card on Cisco router

- OC-3 connection terminating in Houston, TX via MAN ring 1040.006 & SONET rings 60/27.5 with single point of failure termination in Ft. Worth, TX

• Irving, TX (East Division)

- Fiber cable run thru conduit terminating into OC-3 line card on Cisco router

- OC-3 connection terminating in Ft. Worth, TX via MAN rings 1262.003 & 3203 with single point of failure termination in Ft. Worth, TX

San Antonio/Irving Details/Proposed Design

• San Antonio, TX (West Division)

- Fiber cable run thru separate conduit (possibly at opposite building entrance) terminating into OC-3 line card on separate Cisco router

- Additional network connection will be terminated at geographically separate Sprint MPLS switch site, insuring that a catastrophic switch failure at the site would not result in a total site outage

• Irving, TX (East Division)

- Fiber cable run thru separate conduit (possibly at opposite building entrance) terminating into OC-3 line card on separate Cisco router

- Additional network connection will be terminated at geographically separate Sprint MPLS switch site, insuring that a catastrophic switch failure at the site would not result in a total site outage

Austin Data Center OC-3 Connections

Austin MAN1090.002

SONETRing 60.7

MAN node17030

Austin Data center OC3

PL 537791Austin

POP (4889 )

Atlanta 2140 IP Node

SPRINT GLOBAL MPLS

VPN

Austin MAN1090.002

SONETRing 60.7

MAN node17030

AustinPOP (4889)

SONETRing 11.17

Atlanta 2140 IP NodeFt. Worth 2040

IP Node SPRINT GLOBAL MPLS

VPN

Austin Data center OC3

PL 538183

Austin OC-3 connections

Ft. Worth 2040IP Node

Known Common points : MAN node/ring, Austin POP, SONET Ring from Austin to

Ft. Worth

Austin Data Center Details / Current Design

• Austin Data Center, Austin, TX- Fiber cable runs thru same conduit terminating into OC-3 line

card on two separate Cisco routers

- First OC-3 connection terminating in Ft. Worth, TX via MAN ring 1090.002 & SONET ring 60.7 with single point of failure termination in Ft. Worth, TX

- Second OC-3 connection terminating in Atlanta, GA via MAN ring 1090.002 & SONET rings 60.7/11.17 with single point of failure termination in Ft. Worth, TX

Austin Data Center Details/Proposed Design

• Austin Data Center, Austin, TX<True Redundancy to operate in parallel with current

implementation>

- Aerial fiber cable run terminating into OC-3 line card on first Cisco router thru different OC-3 carrier (Time Warner) terminating in Ft. Worth, TX via MAN ring 1090.002 & SONET ring 60.7 with no single point of failure

<Time Warner has existing aerial setup that WB could utilize at no additional cost>

Corpus Christi HQ OC-3 Connections

SONETRing 60.4

OC-3 local access(AT&T)

WBHQ Copus OC3PL 540029

Corpus ChristiPOP (4480)

SONETRing 60.4

Corpus ChristiPOP 4480

AustinPOP (4440)

SONETRing 27.4

HoustonIP Node 2290

PE 11

Ft. Worth 2040IP Node

WBHQ Corpus OC3PL 537702

Corpus OC-3 connections

SPRINT GLOBAL MPLS

VPN

Corpus ChristiPOP (4480)

HoustonIP Node 2290

PE 11SPRINT

GLOBAL MPLS VPN

Known Common points : Corpus POP, SONET ring

(different paths ), Houston IP Node router

Corpus Christi HQ Details / Current Design

• Corpus Christi HQ, Corpus Christi, TX- Fiber cable runs thru same conduit terminating into OC-3 line

card on two separate Cisco routers

- First OC-3 connection terminating in Houston, TX via SONET ring 60.4 with single point of failure termination in Houston, TX

- Second OC-3 connection terminating in Houston, TX via SONET rings 60.4/27.4 with single point of failure termination in Houston, TX

Corpus Christi HQ Details / Proposed Design

• Corpus Christi HQ, Corpus Christi, TX<There is only one local carrier (SBC/AT&T) for the Corpus

region>

- Eliminate SONET ring path 27.4 in second OC-3 path terminating in Houston, TX, thereby eliminating single point of failure in Houston, TX

- Since there is only the option of one local carrier for the Corpus region, having the OC-3 path termination points in Houston and

in Ft. Worth will optimize network diversity

-Sprint charges a one time network diversity fee of $600.00 and a recurring fee of $300.00 per month