>> bellsouth.net > engineering & architecture floridamix next generation exchange...

>> BellSouth.net > Engineering & Architecture

FloridaMIXFloridaMIX

Next Generation Exchange

February 2001

February 2001 2

>> BellSouth.net > Engineering & ArchitectureAgenda

Introduction/Background

Operations

“Classic” Options• Optical/TDM Transport Core

• ATM Transport

Evolution

Facilities/Survivability

Participants

February 2001 3

>> BellSouth.net > Engineering & ArchitectureDesign Team

• BellSouth.netSusan Campbell

Mike Duckett, Sr. Director, Eng. & Architecture

Robbie Harrell

Christian Kuhtz

Troy Meuninck

Deron Ringen

Tim Seaver

(+ various clueful vendor people & peering geeks)

February 2001 4


Company Background

• $27.6 Billion Total Revenue; $3.54 Billion Data Services Revenues (2000)

• 956,000 Internet Customers

• 3.4 Million Fiber Route Miles Deployed

• Customers:

– 34.7 Million US

– 6.9 Million Latin America

February 2001 5

>> BellSouth.net > Engineering & ArchitectureWhat is it?

• FloridaMIX is an exchange facility providing infrastructure in the greater Miami area to connect organizations with each other, regardless of their geography.

• Minimum DS-3 port speed, up to OC-48 (at launch; OC-192 mid 2001).

• Ethernet (100base, GE) introduced 3Q 2001.• SONET & SDH, conversion late 2001.• built from the ground up to be resilient, highly

available, reliable, diverse, ..• Hosted by BellSouth.

February 2001 6

>> BellSouth.net > Engineering & ArchitectureWhy?

• Infrastructure to support multimedia content distribution will be critical to economic development of the tri-county area, Florida and the Southeast

• Improved access to International Markets• Enable economic private peering

infrastructure for large ISPs• Provide smaller ISPs and private businesses

local access to the Internet backbones

February 2001 7

>> BellSouth.net > Engineering & ArchitectureDesign Criteria

• Guaranteed Peering Quality

• Flexible, rapidly provisionable, any to any connectivity.

• Independence– Location– Carrier

• High degree of survivability

February 2001 8

>> BellSouth.net > Engineering & ArchitectureR&D facilities

• BellSouth.net IP Engineering & Architecture Lab @ Perimeter (Atlanta, GA)– Dedicated, substantial replica of components (optical

transport core, TDM/IP/MPLS fabrics, provisioning engines) for the FloridaMIX. To be used for Design, Testing, Regression Testing, Troubleshooting, Design Evolution.

– Immediate access to 40+ IP networking & systems engineering people.

• Additional testing efforts at various vendor labs around the nation.

February 2001 9


Connection Models

• Two general models are available:

– Colocate at FloridaMIX Data Center

– Attach remotely to one or more FlordiaMIX POPs.

February 2001 10

>> BellSouth.net > Engineering & ArchitectureCore Layout

Co-location Data Center

POP

POP

POP

POP

OC-48 DWDM fabric

February 2001 11

>> BellSouth.net > Engineering & ArchitectureCoverage

February 2001 12


Example Applications

• Traditional Public & Private Peering

• Content Distribution Injection Point

• Backup Data Centers for South America.

• Generic Broadband, Rapid Provisioning Infrastructure (Bandwidth on Demand)

• Bandwidth Brokerage

• …

February 2001 13

>> BellSouth.net > Engineering & ArchitectureOperations

• BellSouth will operate and manage the FlordiaMIX backbone and equipment through dedicated organization. FloridaMIX NOC run by optical & IP engineers.– 3 engineering seats (24x7) in Atlanta– 3 engineering seats (24x7) in Miami

• Merit will manage the public peering RS (MOU in progress).

• Communication is a top priority.

February 2001 14

>> BellSouth.net > Engineering & ArchitecturePipes

• Core and access fiber infrastructure for FlordiaMIX provided by– FPL Fibernet – BellSouth Telecommunications

• Any optical loop will accepted, as long as conduit space is available. Electrical interfaces on a case-by-case basis.

• Intent is to be connected to every cable landing site. Negotiations in progress.

February 2001 15

>> BellSouth.net > Engineering & ArchitectureAccess

• Conduit access must be negotiated with landlord (not affiliated with BellSouth). As long as landlord provides conduit space, we will connect.

• Contractual guarantees for “any loop” will be available – intent: “no carrier jail”.

February 2001 16


Transport Highlights

• Robust optical transport service offerings ranging from an unprotected service to full route redundancy with SONET/SDH like recovery and restoration (< 50ms per link including detection and restoration)

• Service connections support SONET and SDH framing

• Service connection capacities range from STS-1 to STS-48 in STS-1 increments

• Support for optical VPNs

February 2001 17

>> BellSouth.net > Engineering & ArchitectureFiber Layout

FPL FiberNetBST

POP POP

POP POP

1 dark fiber pair each

February 2001 18

>> BellSouth.net > Engineering & ArchitectureFiber Layout

• 6 λ per inter-POP pair @ turn-up

• installed capacity for 24 λ per POP (equipment capable of 80 λ)

• no concern about fiber grade w/ OC-48, OC-192’s depend on actual fiber condition.

• Can mix and match OC-48 & OC-192 (no interruption once OC-192 is ready).

February 2001 19


Optical POP layout

SycamoreSN16000

SycamoreSN8000

SycamoreSN8000

Edge Core

DWDM

February 2001 20


Equipment Choices

• Why Sycamore?

“If you believe software is the critical gate, you have to go with a software centric solution. Neccessary for on-demand enhancements, and SONET/SDH is crucial in an international environment.”

February 2001 21

>> BellSouth.net > Engineering & ArchitectureTDM Transport

February 2001 22

>> BellSouth.net > Engineering & ArchitectureATM Transport

February 2001 23


• The ATM transport provides a fully redundant OC12 backbone.

• Public peering over ATM at DS3,OC3 and OC12 port speeds utilizing ATM UBR class of service.

• Private peering and transport over ATM at DS3, OC3 and OC12 port speeds utilizing ATM CBR, VBR and UBR class of service.

February 2001 24


• Minimum guaranteed bandwidth over the public ATM infrastructure.

• Rapid provisioning to meet increasing customer requirements.

• No overbooking or over-subscription allowed in order to maximize throughput of IP based traffic.

• CMN capabilities are in the works.

February 2001 25


Architectural Requirements

• Any-to-Any connectivity

• Closed User Group

• Guaranteed Bandwidth

• Multiple Transmission Interfaces Types

• Intelligent Congestion Management

• Transparent transport of tagged frames

• Link Aggregation

• Jumbo Frame Support

• Loop Detection

• Multicast Traffic

• Manageability of Service

• SLA Support

• Unified Control Plane

February 2001 26

>> BellSouth.net > Engineering & ArchitecturePhase II

FL-MIX Co-LocationFacilities

CUG 1

Ethernet Switch

CUG 2

Ethernet Trunks

FL-MIXSONet / DWDM

Core

CustomerCustomer

Customer

Customer

Customer

Ethernet Switch

Ethernet Switch

Ethernet Switch

CUG 2

SONet, ATM, FRTransport

Customer

CUG 3

Dark Fiber

Customer

Customer

CustomerCUG 1

3rd Party Co-LactionFacilities

Customer

3rd PartyEthernet

MANService

CUG 3

EthernetTrunk

Flordia-MIX Phase IIEthernet over TDM Transport

February 2001 27

>> BellSouth.net > Engineering & ArchitecturePhase II

FL-MIX Co-LocationFacilities

CUG 1

CUG 2

Label Switched Pathes

FL-MIXSONet / DWDM

Core

CustomerCustomer

Customer

Customer

Customer

Label Switch Router

CUG 2

SONet, ATM, FRTransport

CustomerCUG 3

Dark Fiber

Customer

Customer

CustomerCUG 1

3rd Party Co-LactionFacilities

Customer

3rd PartyEthernet

MANService

CUG 3

Flordia-MIX Phase IIEthernet over MPLS Transport

Label Switch Router

LSP

Label Switch Router

LSP

LSP

Label Switch Router /Ethernet Switch

Ethernet

February 2001 28

>> BellSouth.net > Engineering & ArchitectureFacilities

• 4 POPs

– 10,000 sq.ft each

• Colo Data Center in West Miami

– 70,000 sq.ft. raised floor, 100,000 sq.ft. total

– Additionally Hosting Centers are obvious alternates (3rd party)

February 2001 29

>> BellSouth.net > Engineering & ArchitectureSurvivability

• no site is in category 1 evac zone (subject to storm surge)

• 3 POPs category 5, 1 POP category 3• Own generator, battery plant per site. 6 hrs

minimum battery coverage. (Colo has UPS)• “Miami has at times severe weather conditions,

you have to have the resources and committment to fly in resouces (people, equipment, fuel) to guarantee high availability and resiliency.”

February 2001 30

>> BellSouth.net > Engineering & ArchitectureBuild-Out Plan

Three Phases:

Phase 1:

Phase 2:

Phase 3:

Private Peering via Optical, TDM, ATM

Public Peering via ATM

Private & Public Peering via Ethernet.

Private & Public Peering via MPLS.

Fully Automated Provisioning

February 2001 31

>> BellSouth.net > Engineering & ArchitectureCurrent Timeline

1st Quarter 2001– Deploy phase one (Target: very late CQ1)

• Optical Transport

• ATM private and public services

2nd Quarter 2001– Assess and define Ethernet and transport

agnostic transport mechanisms.(MPLS, GMPLS, MPLS “Circuit Cross Connect” , RFC2547(bis) layer 3 VPN’s, etc)

February 2001 32

>> BellSouth.net > Engineering & ArchitectureParticipants

• The following have signed LOI’s w/ publicity release statements:

Qwest, Diveo, Intermedia, FPL Fibernet, UUNET (a WorldCom Company), Exodus, PSINet, Nova

Southeastern University, Computer Office Solutions, Hyperbyte, Cyberlinyx, Advanced

Data Center, Inc., Worldwide Internet Services, Synergyx, Worldwide Telnet and Florida

Atlantic University

February 2001 33


Thank You!<[email protected]>

February 2001 34


February 2001 35


– DS3-OC12 support– Scalable non-blocking switch fabric– Support for automatic protection switching (APS)

for redundancy and protection– Support of both SONET and SDH framing– Intelligent cell discard (visibility into IP frames)– Per VC queuing– Congestion control– Flow control– Ease of provisioning

February 2001 36

>> BellSouth.net > Engineering & ArchitectureATM NAP

Peer1

Peer2

Peer3IX

ATM

OC-3 Pipe

DS-3 Pipe

DS-3 Pipe

Transport PipeATM UBR VC

February 2001 37


Peer1

Peer2

Peer3IX

ATM

25 Mbps offered bandwidth

25 Mbps offered bandwidth

Congestion Point

February 2001 38


IP Packet

Cell IP Chunk

Cell IP Chunk

Cell IP Chunk

Cell IP Chunk

… packet shredding occurs @ Peer1 & Peer2

February 2001 39


IP Packet

Cell IP Chunk

Cell IP Chunk

Cell IP Chunk

Cell IP Chunk

… congestion cell discard occurs @ ATM IX switch

February 2001 40


IP Packet

Cell IP Chunk

Cell IP Chunk

Cell IP Chunk

… re-assembly fails @ Peer3

February 2001 41


• Conclusion (in a nutshell):– Must have packet aware discard mechanism to

guarantee decent goodput. (otherwise, for example, <5% cell loss can translate into 70+% AAL5’ed IP frame loss/corrupted frames).

– Testing shows ABR w/ VSVD in the core allows for very nice IP traffic guarantees over ATM.

– Long term migrate to a packet based infrastructure. => Strong focus on using Ethernet & MPLS w/ strong R&D effort.

February 2001 42


Congestion Management

• Frame based congestion management– Primarily at the provider edge of the FlordiaMIX.

– Pushed out to “customer” edge in the case of hosting centers/carrier hotels for multiple providers sharing a transport pipe.

• Exact configurations are presently under study

February 2001 43

>> BellSouth.net > Engineering & ArchitectureEthernet CUG

• Ethernet switches at the FloridaMIX can join 802.1q VLANs to create closed user groups (CUG) for peering purposes. – Multicast

– experimental peering services such as IPv6

>> bellsouth.net > engineering & architecture floridamix next generation exchange...

Documents