Peering & Internet eXchange Points

AfNOG 2001 Meeting12th May, 2001.

Miklin HotelAccra – Ghana

Abstract

Internet Service Provider (ISP) Peering has emerged as one of the most important and effective ways of improving efficiency of service and operation. Peering is the interconnection mutual business arrangement between at least two Service Providers whereby each directly exchanges traffic to and from each other's clients. Peering relationships are sought primarily because peering reduces cost and reliance on purchased Internet bandwidth and/or transit. Secondly, peering lowers inter-AS traffic latency. By avoiding a transit provider hop in between ISPs traffic between peering ISPs has lower latency. So how is peering done and what are the issues and structures of peering?

In peering arrangements, especially when two ISPs of similar size want to interconnect it becomes a problem. The setting up of Internet Exchange Paints (IXP), which is the neutral interconnection point of traffic exchange between peering ISPs has been a way to solving peering problems.

In this paper, focus is made on issues regarding peering and Internet Exchanges, structures, roles and the decision making process which further breaks down into three distinct major processes as follows: Identification (Traffic Engineering Data Collection Analysis), Contact and Qualification (Initial Peering Negotiations) and Implementation Discussion (Peering Methodology). The first phase identifies 'the who‘and 'the why', while the second phase focuses on 'the how'.

Background to ISP Peering

The global telephone system and the Internet share a major common attribute of value to all users of their respective services - that of comprehensive connectivity

Preservation of such connectivity became a problem due to the growth of the Internet.

Transition and Financial consideration necessary to guarantee the long term viability of the medium were also problems

Peering is a one way to solving these problems The Interconnection points for traffic exchange between peering ISPs on the

Internet is the Internet eXchange Point Exchange Points are high-speed (not less than 100mb) switch network, with

current physical configuration of today being a mix of FDDI/ATM switches

General Peering Models

Takes place within one of three general models First is that of bilateral settlements The second model is that of the Sender Keep All (SKA) The third is that of transit fees Within the Internet to date only the second and third models can be observed

within the deployed infrastructure SKA arrangement is only stable where the parties involved perceive equal benefit

from the interconnection The third model (transit fees) is commonly seen as a component of the

relationship between a transit provider and a local access provider, where the local provider is charged for transit services within a larger service agreement.

There is also that of multilateral settlement, which has been developed at neutral exchanges.

Issues & Structures

Within the Internet the common basis of the settlement appears to be one of flat fees, without reference to exchanged data volumes

Significant issues include the status of an ISP - a peer to, or a client of another ISP Another is that of imbalance in traffic exchange There are three generic peering structure options:

direct point-to-point interconnection, multilateral exchange points and mutual bilateral exchange points

Issues & Structures

Direct point-to-point peering Each bilateral peer arrangement is implemented through a physical

interconnection Multilateral exchange points

Seen within the CIX exchange architecture, and within the European EBONE structure

Architecture is the enforcement of a single multilateral peering structure Stable when all parties are in a position of approximate equal positioning with

respect to the mutual traffic exchange Mutual bilateral exchange points

Offers the greatest degree of flexibility Have been used for some years within the United States

Decision Making Process

The decision making process involves three main phases These phases identifies ‘the why’, ‘the who’ and ‘the how’ of Peering and

Exchange Points Setup

The Three Major Decision Making Process Identification of Potential Peer: Traffic Engineering Data Collection and

Analysis Contact & Qualification: Initial Peering Negotiation Implementation Discussions: Peering Methodology

Phase One:

Identification of Potential PeerTraffic Engineering Data Collection and Analysis

An interview conducted by W.B. Norton with some ISPs highlighted a common concern: traffic destined for a competitors customer located across the street may need to traverse a couple of transit providers across great distances (with high latency) before interconnecting. The worst example was that traffic between the United Arab Emirates and Saudi Arabia must traverse an exchange point in Washington DC. But through direct interconnections (through direct circuits or regional exchange points) ISP customers realize better performance

Reduces Transit Costs. Internet transit costs dominates the decision making process Peering reduces transit costs

Lower Latency Direct interconnection with peers lowers latency in traffic between peering

ISP's High Speed, low latency access to sites for end-users

Scaling Bandwidth Continued success of the Internet depends on increasing bandwidth Since packet loss and latency slows traffic consumption, there is benefit from

a low latency, low packet loss Internet Enables POP Activity

Provides a hardened and scalable environment in which to operate POPs

Motivations: Why do you need to peer?

ISP A

ISP B

Transit ISP

Transit Cost ($$)

Transit Cost ($$)

Peering provides routes only to each others customers

Transit vs. Peering Interconnection

Peering between ISP A and ISP B

Transit from ISP A and ISP B to Upstream or Transit ISP

Variety of Criteria are used to identify Potential Peers, such may include the following:

Quantities of traffic  Business Credibility Interconnection Policies

The Criteria: Whom do you peer with?

Yes

Yes

Yes

Yes

Yes

Part of broadBusiness

Relationship?

DominantTraffic Flow?

Large new Customer impact?

Traversing Expensive transit

cost?

Will PeeringHave a positive

effect on mynetwork?

Proceed to Phase 2Contact Peer

A Simplified Peering Selection Decision Tree

Migration Path from Transit to Peering

Interviews conducted with some tier 2 ISPs highlighted an emerging peering transition strategy:

Access the Internet via transit from a global provider Pursue peering arrangements on public switches at exchange points to

reduce load on transit links and improve performance Migrate high traffic peering interconnections to private interconnections (via

fiber or direct circuits) Ultimately migrate traffic away from transit purchase and negotiate (free or

for-fee) peering with former transit provider

Interview was conducted by W. B. Norton (Equinix Inc.) in the U.S.

Phase Two:

Contact & QualificationInitial Peering Negotiation

ISPs should have a person or group in the organization specifically tasked with traffic engineering issues

The first step in a peering arrangement is for one the parties to initiate the contact with the potential peer partner

Finding the right person to speak with is a difficult intensive task Second step, though optional is the negotiation and signing of mutual non-

disclosures The third step is to evaluate requirements and capability Each party then decides whether to stay or to walk away until criteria and

requirements are met

Principles for Peering Arrangements

However, some principles are needed to be considered during preparations for peering arrangements. These include:

  providing interconnection agreement under non-discriminatory and

transparent terms and conditions; providing interconnection agreement at non-discriminatory cost-oriented

rates and comparable quality to that which it provides itself; negotiate interconnection in good faith; and ensure that interconnection agreements, or a reference offer, are made

public.

Interconnection is a necessary preliminary to traffic exchange

Contact & Qualification Decision Tree

Proceed to Phase 3Implementation

of Discussion

Peering @ orpersonal contact

Exchange Point Contact

list

tech-c oradmin-c in DNS/ASNRegistry

Operations Forum

Trade Shows

Sales Force

Larger business

transactions

InitialContact

Do both parties findmotivation to continue

peering discussion?

Close Discussion

Yes No

Sign NDA,See Policies

Share trafficdata, BLPA

Phase Three:

Implementation DiscussionsPeering Methodology

Since peering is of mutual benefit, both parties next explore the interconnection method(s) that most effectively exchange traffic to and from each other's customers

The primary goal is to establish mutual point(s) of interconnection, and secondarily detail optimal traffic exchange behaviours

ISPs face three options for interconnections: Direct Circuit Interconnection, Exchange-Based Interconnection or Some global combination thereof

The preferred methodology to adopt depends on the number of peers participating and interconnecting at the Exchange Point

And also on the bandwidth required for the Exchange Interconnections ISPs that expect to interconnect at high or rapidly increasing bandwidth within the

region, or expect interconnections with more than five parties in the region often prefer the exchange-based solution where we have the multilateral exchange points structure

Those that do not anticipate a large number of regional interconnects prefer direct-circuit (the direct point to point structure) and typically decide to split the costs of interconnection with the peer by region

Peering Arrangement Goals

get peering set up as soon as possible, minimize the cost of the interconnection and their transit costs maximize the benefits of a systematic approach to peering execute the regional operations plan as strategy dictates (may be architecture /

network development group goal), and fulfill obligations of larger business agreement.

Direct-Circuit Interconnection

Interconnection takes place in one of two forms: Peering and Transit In either case, it requires provision of bandwidth between parties Direct Point-to-Point interconnection model requires the lease of point-to-point

circuits between parties that scale linearly with the number of ISPs The circuits traverse miles of fiber underground, subject to outages due to

construction (and the rapid proliferation of backhoes) As the bandwidth requirements grow, ISPs need to upgrade circuits to many

different places; they are unable to take advantage of traffic aggregation over a very high bandwidth circuit

Cost of this interconnection strategy does not scale either

Direct-Circuit Interconnection

SSS

GGG

UUU

A A A

S

G

U

A

CCCC

Point-to-point circuits for ISP

interconnection in an exchange

Direct-Circuit Interconnection

Point-to-point circuits for ISP interconnection in an exchange (a simpler diagram)

ISP B

ISP A

ISP C

R2

R1

R3

Exchange-Based Interconnection

An alternative to the direct circuit interconnection model is for ISPs to purchase instead a much larger circuit into the Neutral Internet Business Exchange to take advantage of the relatively inexpensive and the rich connectivity mesh within the exchange

Efficient to scale a single high bandwidth, point-to-point circuit than many lower speed point-to-point circuits

The greater the dependence on the interconnection, the more hardened and scalable that interconnection environment should be

In a Neutral Internet Business Exchange Model, ISPs have access to an additional source of revenue: transit sales to content providers and ISPs

Exchange-Based Interconnection

ISP A R1

ISP C R3

ISP E R5 ISP FR6

ISP DR4

ISP BR2

R0ExchangePoint

Exchange PointHigh-Speed Switch

Each ISP has its own transit uplink

Peering Implementationand Operation

Exchange Environment Selection Criteria

Deployment Issues

ISP CurrentPresences

TelecomAccessIssues

CostIssues

BusinessIssues

Operations Issues

CredibilityIssues

Exchange Population

Existing Vs.Emerging

Exchange Environment Selection Criteria

Telecommunication Cost Issues

Have to do with getting telecommunications services into the exchange How fast can circuits be bought into the interconnection environment? How many carriers compete for my business for circuits back to my local Point of

Presence (POP)? For facilities-based ISPs, what is the cost of trenching into the exchange (how far

away and what obstacles present themselves)? Are there nearby fiber providers that lease strands? These answers will answer the most important question to ISPs:

How fast can my peer and I get connectivity into the exchange?

The answers to these questions strongly impact the desirability of the exchange environment

Deployment Issues

Have to do with getting equipment into the exchange How do I get my equipment into the exchange (assuming it supports collocation)?

Do I ship equipment in or do I have to bring it with me as I fly in? Will someone act as remote hands and eyes to get the equipment into the

racks or do I do the installation myself? What are the costs associated with deployment (travel, staff time, etc.)? Does the exchange have sufficient space, power?

The answers to these questions impact the deployment schedule for the ISP engineers and the costs of the interconnection method

ISP Current Presences

Based on the observations that the most inexpensive and expedient peering arrangements are the ones made between ISPs that are already located in the same exchange

The assumption here is that there is sufficient capacity to interconnect. Cross-connects or switching fabrics can easily establish peering within hours or at most days

Operations Issues

Focus should at this stage shift to operations activities allowed within the Exchange

Does the exchange allow private network interconnections? Are there requirements to connect to a central switch? How secure is the facility? Is there sufficient power, capacity at the switch, space for additional racks, real

time staff support? Is it easy to upgrade my presence over time?

Upgrading in this context means the ability to increase the speed of circuits into the exchange, the ability to purchase dark fiber, the ability to increase the number of racks and cross connects in the exchange, the ease of increasing the speed of interconnection. ISPs will prefer bandwidth-rich, ISP-friendly exchanges to those with restrictions over future operations.

Business Issues

Perhaps the most far-reaching business issues are strategic, as an ISP, ask these questions:

Do we want to support this exchange operator Do the interests of Exchange partners enhance or conflict with ours?

“Bandwidth, strategic partner alliances, and corporate ties often override the technical justification.”

Will using this exchange support a competitor (contribute to their net income, their credibility, their positioning)?

Does the exchange have requirements (require use of their carrier or ISP services) that limit the market for services within the exchange?

Cost Issues

Crosses all other criteria What is the cost of using this exchange? What are the rack fees, cross connect fees, port fees, and installation fees? What is the future operating fees going to be? What are the motivations and parameters surrounding these fees?

Exchange Population Issues

These issues go to the other (side) benefits to using this exchange, Are there other ISPs there that are peering candidates?

Are there transit sales possible at the exchange? In the context of the credibility issue discussed above, who will likely be at the

exchange in the future, and when will the cost of participation equal the value of the interconnection (also known as the Critical Mass Point)?

Existing vs. New Exchanges

There are many existing exchanges and also emerging ones Which one would benefit you as an ISP intending to go into peering with others? Preferences for an New Exchange

Chronic traffic congestion can influence the decision to plan to peer in an existing exchange or wait until a better exchange opens

Long-term benefits (scalability) may lead to preferring a next generation exchange. However, all else considered equal, ISPs generally prefer an existing exchange to an emerging one

How important are these Issues?

These issues where not listed in any particular order

An interview conducted by W.B. Norton with some ISPs in the U.S. reveals that they shared varied weightings of importance of each of these issues. To some the most important issues where business related, and others weighted more highly the flexibility of ongoing operations. Each ISP will place higher or lower importance on different issues and not surprisingly select operations environment based on their specific criteria. It was noted that they all seem to agree upon the general issues within which their criteria operate.

Summary

Looked through a rough description of the decision process ISPs follow to obtain peering relationships and establish peering

It focuses on the elements of the decision that lead to the selection of a specific exchange environment for peering

It can be concluded that Peering has some important advantages, with the issue of reduced cost and improvement in performance (lower latency) as the primary benefits 

The goals of peering ISPs include: to get peering set up in the soonest possible time; to minimize the cost of the interconnection and their transit costs; to maximize the benefits of a systematic approach to peering; to execute the regional operations plan as strategy dictates (may be

architecture/network development group goal) and finally, to fulfill obligations of larger business agreement.

It should also be of critical note to potential peers that the need to make proper decision before joining an Internet eXchange is very critical

One major challenge facing Peering Coordinators is the identification of potential peers and initiating discussions