Cable Metro Packet Optical Transport

An Overview and Vision for L0-L7 Convergence

A Technical Paper prepared for the Society of Cable Telecommunications Engineers

By

Andrew Smith

Distinguished Engineer & Chief Architect for Cable MSO Networks

Juniper Networks

1194 N Mathilda Ave

Sunnyvale, CA 94089

+1 408 745 2000

asmith@juniper.net

 

1) Introduction

Contemporary cable metro networks have undergone a tremendous transformation in the last

decade. From the early days of IP (Internet Protocol) data as simple Internet access, to modern

networks supporting a multitude of residential and commercial data services, the cable industry has

captured tremendous benefit from packet infrastructures. A high rate of disruptive innovation has enabled

new applications and revenue streams for cable operators, all based upon the high-speed delivery of

packetized, IP data.

This innovation has developed predominately at the packet and application layers. Underneath

this innovation lies an optical transport layer (the “Optical Domain”) which has continued to operate in

isolation of the routing and application layers (the “Packet Domain”). This separation of domains leads to

suboptimal placement of technology and capital, resulting in inefficiencies and persistent operational

challenges.

This paper describes a proposed architecture for a packet oriented, application driven transport

system that enables the next generation of all-packet, all-IP dynamic multimedia residential and

commercial services in modern cable regional networks. In the regional / metro area, due to the relatively

shorter distances involved, unique cable industry use cases, and all-IP based services, we have license

to rethink the status quo network architecture through the adoption of new technology enablers and

disruptive innovations.

Specifically:

- Next generation of optical integration into the packet domain

- Auto-discovery, and automation, of optical attributes

- Ease of packet deployment and transport LSP (Label-Switched Path) establishment

- Multi-layer optimization, including application layer feedback

 

- Use of modern SDN (Software Defined Networking) derived tools to automate, manage and scale

the infrastructure

Such an evolution is based on the Packet Domain subsuming the Optical Domain so as to enable

a degree of flexibility and programmability and to meet the next generation of SDN enabled services.

Traditional metro network architectures have meant overinvestment in optical transport, or packet

forwarding, or some combination thereof. Through graceful and software driven use of optical bandwidth,

cable operators can reliably transport data across the network while avoiding, to a degree, the very latest

optical technology.

In modern Internet and packet networks operators are accustomed to the many benefits of

statistical multiplexing (stat-mux) of data. Operators routinely “overprovision” the networks. The

fundamental architecture of any-to-any, packet based, connectionless networks permits reliable service

delivery. Stat-mux gain is increases with increasing discrete flows on the network and increasing

variation of each flow. Extending packet stat-mux to the optical layer and controlling the network with

modern software results in a totally optimized packet network .

In Figure 1, a circuit switched network would require a circuit the size of P1 (at a minimum) and

another circuit of size P2 (at a minimum). The granularity is very coarse, as an ODU0 (Optical Data Unit)

is 1.25 Gb/s. In packet switched networks, the required bandwidth in the core is actually less than P1 +

P2.

 

Figure 1: In OTN or circuit switched networks, circuits must be provisioned to peak capacity between each router. In packet transport systems, this is not necessary. A “PE” is an edge router in a service provider

network.

2) Optical Integration

This paper proposes total integration of the optical transponder and multiplexor into the router

(Packet Domain). In traditional architectures, the transponder and multiplexor reside in an external

element within the Optical Domain. This separation of optical and packet networks imposes discrete

Peak%value%P1%

Bit-rate%

/me%

Peak%value%P2%

Bit-rate%

/me%

PE1%to%PE2%traffic%

PE3%to%PE4%traffic%

OTN%circuit%size%

OTN%circuit%size%

Wasted%Bandwidth%

Wasted%Bandwidth%

PE1% PE2%

PE3% PE4%

Core%

B/W%?%

 

control and management planes, which require separate engineering efforts and unique disciplines in

network planning.

With the development of modern 100G coherent technology, the transponder can be gracefully

integrated into the Packet Domain, creating an optimal relationship between optical and packet

forwarding capacities, a reduction of operational expenditure, and in some cases, an improvement in

optical performance. No longer is there a gap between the packet forwarding capacity in the packet

domain and the supportable bandwidth in the optical domain. The relationship between the two becomes

a tight bond, with significant optimizations to the operator.

Future optical components are expected to continue their trajectory of miniaturization and power

reduction, bandwidth improvements, and industry efforts such as OpenFEC will permit multi-vendor

interoperability. OpenFEC is an effort to create open, publicly documented Forward Error Correction

standards, with one intent being to create vendor standardization and interoperability between optical

systems. These developments will enable continued phases of core evolution, and will enable the

densities and power envelopes required for the packet domain to subsume the optical domain. This

integration of the multiplexing and demultiplexing of lambdas, or individual wavelengths, into the packet

domain enables an opportunity to rethink how a transport domain supports IP services.

Previous methods to integrate the optical transponder into the packet domain required the use of

a packet-to-optical control plane integration scheme. A suite of protocols known as GMPLS (Generalized

Multi-Protocol Label Switching) was leveraged to permit the router to signal a wavelength through the

optical domain and various “split management” techniques, such as Virtual Transponder, to permit the

optical domain to manage the router’s transponder as if it were still an element in the optical domain.

While these methods of packet optical integration represented a significant optimization of packet

resources and operational savings, it imposed the cost of a complex control and management layer.

 

These contemporary solutions represent only an incremental evolution towards transforming the network

core, and this paper discards it’s use.

Modern data networks in cable systems are dominated by router-to-router IP services and data

center interconnection. As shown in Figure 2, this results in two disparate “systems” – optical and packet

– that have independent optimization points. This architecture proposes to merge those layers into a

collapsed packet-transport network fabric, resulting in an aligned, congruent and flexible system.

Figure 2: Traditional networks have separate transport and packet layers, resulting in inefficiency. Merging packet and optical results in an optimal architecture.

3) Auto Discovery and Automation

Mul$%layer*Meshed*Network*

Collapsed*Packet%Transport*Network*Fabric*

 

The packet domain is accustomed to a model where network elements discover one another

dynamically, exchange capabilities, and subsequently build the required databases and forwarding state

to enable the switching of services. Once the optical transponder and multiplexing and demultiplexing

functions are integrated into the packet layer, new opportunities for discovering total multi-layer

connectivity become available. In this evolved architecture, the integrated transport system elements

will dynamically exchange optical characteristics through ‘Optical Peer Discovery’ procedures, automating

the wavelength and channel setup by simply defining how much packet bandwidth is required between

packet nodes.

Autodiscovery of adjacent packet elements will happen through the traditional Optical Supervisory

Channel (OSC), which has long been a part of contemporary DWDM (Dense Wave Division Multiplexing)

optical systems. Elements will exchange wavelength allocation policies and spacing requirements over

the OSC. Existing protocols, such as LLDP (Link Layer Discovery Protocol) and Netconf, can be

leveraged to discover, manage, configure and operate elements along the path. Through the automatic

discovery of optical infrastructure, such as amplifiers, and the coordinated exchange of wavelength

requirements between routing elements, a “plug and play” functionality can be brought to the optical

domain that has long been the standard operating model of the packet domain. As shown in Figure 3,

much how routing protocols discover adjacent nodes and exchange properties, this architecture proposes

a similar mechanism at the optical layer.

 

Figure 3: Packet Optical devices discover their neighbors, exchange capabilities, and establish service. Routing protocols do this today in the packet domain.

4) Simplified Packet Configuration and Centralized Path Optimization & Management

In traditional packet domain architectures, the network operator must choose specific

configuration items for the routing infrastructure, with a mix of IGP (OSPF, IS-IS), MPLS (LDP, RSVP)

BGP (multiple address families) and related protocols. Originally intended to permit maximum flexibility to

the operator, the sheer number of permutations these choices enable can create an environment of

uncertainly and perceived complexity of what is otherwise a very elegant and graceful packet

architecture.

As a product of the specific use case of this architecture, and by leveraging advances in Software

Defined Networking technologies, the configuration of the MPLS infrastructure to support packet transport

is radically simplified. Simple, and automated configuration of the IGP and the label distribution protocol

Peer$Discovery$

Property$Exchange$

1Tb$Transport7group$

800G$Transport7group$

 

is all that is required, actual path computation and LSP establishment is delegated to the Path

Computation Engine (PCE).

A PCE is a critical element of this evolved architecture. The PCE provides the central function of

LSP provisioning and resource allocation within the transport domain. By leveraging the Stateful PCE

extensions of the Path Computation Element Protocol (PCEP), a PCE can compute constrained paths,

provide reliable state synchronization and optimize both existing and new LSP’s.

Leveraging the power of MPLS Traffic Engineering (MPLS TE) in packet optical transport

networks enables an elastic, load-adaptable and highly available architecture. MPLS TE permits the

network to respond dynamically to changing traffic conditions, and to adapt the network to the demands

of the modern cable regional core network. The analytics produced by the Path Computation Element is

the origin of the advanced logical topology, instantiated by MPLS TE, and delivered over a packet plus

optical domain. Figure 4 shows the work flow of various protocols and software processes. The concept

is to retrieve data from the network and applications, compute an optimized path for all demands on the

network, the install that path onto the network. Discovery of network information can come from PECP,

from BGP Linkstate (BGP-LS) or other external sources, instansiation of the topology to the network can

be deployed via PCEP, BGP, OpenFlow, I2RS or other data plane control protocols.

 

Figure 4: The power of the Path Computation Element, Software Defined Networking and MPLS Traffic Engineering creates an analytically driven optimization system for packet optical transport networks.

5) Multi-layer optimization, with application feedback

The Path Computation Engine, by virtue of its role in optimizing services on the network, has a

multi-layered view of the transport infrastructure:

- The IGP (ISIS or OSPF) and the optical infrastructure (Layer Zero) are congruent, meaning there is

no ambiguity as to the physical path that traffic will take;

- MPLS LSP’s, pushed into the network via PCEP, are congruent with specific service demands and

requirements of the network, so there is no ambiguity as to the logical path traffic will take.

The contemporary use case of PCE technology is to reconcile the demands, constraints and

requirements of layers zero through three, and produce an elegant logical packet forwarding topology

SOFTWARE-DRIVEN POLICY

Topology Discovery Path Computation Path Installation

PCEP - LSP discovery IGP-TE, BGP-LS - TED discovery jVision – Streaming Analytics

PCEP – Control/Create*traffic*engineered*LSP*

Netconf/YANG May include: BGP, DMI, OpenFlow, I2RS

ANALYZE OPTIMIZE VIRTUALIZE

Routing Netconf/YANG PCEP Junos CSPF Algorithms

 

based on MPLS LSP’s to accommodate services as required. The root metric for allocating and

assigning LSP’s is bandwidth, either inferred from the network or assigned as a reservation.

In cable networks, however, many times bandwidth is not a reliable indicator of successful service

delivery. In an era of unicast, adaptive http video application transport, an interface operating at 100%

load could actually be delivering an acceptable Quality of Experience (QoE) to the end user. Video

players, by their nature, will adapt and may (or may not) choose content of a lower bitrate, but this may

be within tolerance of that particular video asset’s QoE. Conversely, an interface operating at less than

100% load may actually be transporting video that falls beneath a required QoE, due to congestion

elsewhere in the network path. Quality of Experience is a factor in cable networks, and one that most

Path Computation Engines are not aware of.

This evolved architecture proposes an application feedback mechanism, such that a statistically

relevant sample of video players are able to feed back, in real time, Quality of Experience metrics to an

analytics engine associated with the path computation engine. The analysis performed would be to

reconcile the MPLS FEC (group of IP addresses) associated with video players that are reporting

suboptimal QoE, and make a recommendation to the PCE that it is desirable to shift traffic associated

with that FEC to a different path in the network.

Through this feedback mechanism, sophisticated analytics, multi-layer optimization and logical

MPLS topology manipulation, a true convergence of cable metro services can be achieved. Figure 5

shows a high level overview of this proposed architecture.

 

Figure 5: Multiple sources can contribute to the PCE computation engine, this paper proposes video Quality of Experience as a source of intelligence.

6) SDN as a key technology enabler

The emphasis of this paper has been on the benefits of tight integration of the Optical Domain

into the Packet Domain. The benefits of packet infrastructure assuming the role of the optical

infrastructure are numerous. However credit must also be given to developments from the Software

Defined Networking (SDN) movement, as without modern software this architecture would be limited or

perhaps not even possible. Specifically:

- PCE, or Path Computation Engine, performing a stateful optimization of LSP allocation on the

network. LSP’s replace lambdas in terms of service enablement.

- Netconf, and proposed Yang models, as the language enabling two very different layers, with

historically different motivations and technologies, to communicate and interoperate

- LLDP and related extensible discovery protocols, that can discover and indicate elements in a

neutral, open manner

7) Conclusion

!!

!!

Agent&RPD&

Kernel&&&&&&&&&&&&&&&&&&&&&&&&&&&

PFE& PFE& PFE&

Sensors!Programmable!objects!to!tap!into!any!state!of!the!network!node!

Capabili:es!can!be!enhanced!over!:me!

Standard!IP!based!interface!with!data!store!

Decoupling!Telemetry!data!is!outside!JUNOS!

Extendible!client!interface!

Different!analy:c!views!can!be!provided!independent!of!JUNOS!

PCE!Controller!

S S S

Analy2cs&

 

The vision expressed in this paper would not be possible but for the confluence of continued

innovation in optical packaging, packet forwarding technology, software engineering, SDN and dynamic

service requirements of cable operators. Taking advantage of these disruptive forces permits a re-

evaluation of metro network architecture with wide ranging benefits to cable operators.

Contributors

Colby Barth of Juniper Networks contributed to this paper.

 

Abbreviations & Acronyms

Coherent Optics Optical interfaces which take advantage of advanced modulation,

polarization and DSP technologies. Coherent optical interfaces

require fewer amplifiers, and suffer from less chromatic dispersion

than previous systems, easing deployment and management.

OpenFlow A protocol for configuring the data plane of a switch

DWDM Dense Wave Division Multiplexing. A method for transmitting

multiple data streams on a single strand of fiber optic cable.

FEC Forwarding Equivalence Class

I2RS Interface to the Routing System (draft-ietf-i2rs-architecture)

IGP Interior Gateway Protocol

IP Internet Protocol (RFC 791)

IS-IS Intermediate System to Intermediate System. (RFC 1195)

Lambda An individual wavelength in a DWDM system

LDP Label Distribution Protocol (RFC 5036)

LLDP Link Layer Discovery Protocol (IEEE 802.1AB)

LSP Label-Switched Path, a forwarding definition in MPLS networks

NETCONF A standardized method for programmatically configuring network

Elements (RFC 6241)

 

MPLS Multi Protocol Label Switching (RFC 3031)

ODU Optical Data Unit

OpenFEC Open Forward Error Correction. An industry effort to standardize

FEC’s across vendors. http://openfec.org/

Optical Domain The traditional optical transport layer of the network

OSPF Open Shortest Path (RFC 2328)

Packet Domain The traditional packet switching and routing layers of the network

PCE Path Computation Engine

PCEP Path Computation Engine Protocol (RFC 5440)

PE Provider Edge. An edge router in a service provider network.

QoE Quality of Experience

RSVP Resource Reservation Protocol (RFC 3209)

SDN Software Defined Networking. A large and evolving term,

generally related to technology which leverages advanced

software to manage, provision, or optimize modern data networks.

YANG A proposed language for configuring network devices via

NETCONF (RFC 6020)