1

IXP Best Practices

Tay Chee Yong MyNOG 3

28 November 2013

2

IXP Essentials

•  Layer 2 Ethernet network consisting of one or more switches •  Members connects to the network with an assigned IP

address •  Only BGP is allowed –  Bi-lateral (BGP between members) –  Multi-lateral (BGP with route servers)

3

IXP Essentials

•  Announce own origin and customer routes •  Exchange traffic with all other members to improve traffic

gravity and performance –  Members save cost on Internet transit –  Better user experience (reduced latency)

•  One port with many peers –  Allows exchange of routes/traffic among all IXP members

4

IXP Benefits

•  Keep the local traffic local! –  ISP within the country/region peer with each other – Doesn’t need to take a long route out and return –  Improved latency and efficiency

•  Save money! – Traffic stays local means save transit bandwidth = save money

•  Improve network performance – Better RTT between end points – Direct traffic forwarding instead of sub-optimal routing

5

Be responsible!

•  IXP operator responsible to ensure infrastructure is stable and secure –  Choice of hardware/software –  Stability of route server daemon –  Security measures –  Competent operational staffs

•  Usual BGP best practices still apply to all members

•  IXP best practices and etiquettes to be adhered

6

Leaking of IX prefix to Internet

•  Announce IXP prefix outside of AS boundary is not a good idea

•  Providing free transit for IXP prefix

•  Vulnerable to DDOS attacks

• Common reason : redistribute connected to bgp

•  Prefix list/route maps to deny IXP prefix announcement

7

Routing control discipline

•  Same set of routes should be announced over both transit links and IX port

•  Consistent routing policy over different IXP

•  Members announcing more specific routes, may result in transit over the IXP

•  No Static/Default route!

8

Unwanted protocols towards IXP

•  Interior routing protocols : OSPF, IS-IS, EIGRP, RIP - Generates unwanted broadcast/multicast traffic

•  Layer 2 protocols : - STP, VTP, Proxy Arp

• Network discovery : - CDP, LLDP, EDP

9

Proxy ARP

• Members acting as a arp relay, potentially very dangerous

•  Leading to hijacking of packets destined to other members

• Usual culprits are of Cisco equipment •  IOS : enabled by default •  IOS-XR : disabled by default •  JUNOS : disabled by default

#sh arp 219 202.yyy.yyy.yyy 0012.7fxx.xxxx Dynamic 0 15/20 225 202.yyy.yyy.yyy 0012.7fxx.xxxx Dynamic 0 15/20 242 202.yyy.yyy.yyy 0012.7fxx.xxxx Dynamic 0 15/20 316 202.yyy.yyy.yyy 0012.7fxx.xxxx Dynamic 0 15/20

10

Proxy ARP

•  Tools to detect members with proxy arp enabled

•  Violation logs to be sent to NMS monitoring

•  Enhance internal monitoring & operational process

•  Follow up , Follow up

11

Looping back an Ethernet Port…

•  Loopback towards on an IXP port is never a good idea

• Result : broadcast storm towards all other members

• Cripple the IXP, and disrupting traffic

12

Peering with route servers

•  Facilitate implementation of peering arrangement •  Allow new members to join the community easily • Generally have 2 route servers for redundancy

•  Single routing daemon •  Dual routing daemon

•  Reduced the number of peering sessions •  Just peer with 2 to get all routes from all members

•  Ability to manipulate routing policy via bgp communities

13

Port Security

•  MAC address filtering

•  Only permit specific IP ethertypes •  IPv4, ARP, IPv6 •  Drop everything else

•  Enforce one-mac-address-per-port rule •  No additional devices are permitted •  Prevent noise from any intermediate L2 devices (eg. STP)

•  Inform your IXP if you are doing any migration or change of device •  Mac address change

14

Prefix Filtering

•  Applied on route servers •  Per neighbor prefix filtering •  Pros

•  Prevent unintentional route hijack or route leak by members •  Treat IXP as a normal upstream provider to update prefix list

•  Cons •  Accidental of route denial – reduction in traffic •  Solutions : Route update using IRR where possible •  Challenge : Route objects should be updated regularly

15

Configuration Automation

•  Fat fingers and human nature at times cause issues in IXP - Applying incorrect switch configuration - Forgot to apply port security - Typo error -  etc

•  Reduce errors during provisioning of switch or route servers

•  Increase IXP productivity and efficiency

•  Standardize configuration across IXP platform

16

Transparent AS

•  AS-PATH Transparency : Route servers do not insert its own AS number in the AS-PATH updates to members

•  In route servers, well-known BGP attributes (AS-Path, MED, next-hop, communities) are not modified before redistributing to other members.

•  Peering sessions appears to be directly between members, but the RS is mediating the session.

•  Common problem seen with Cisco routers due to default behavior •  IOS : no bgp enforce-first-as •  IOS XR : bgp enforce-first-as disable

17

Transparent AS

•  Non route server setup

AS10 AS20 AS100

Prefix AS-PATH 20.20.0.0/16 100 20

10.10.0.0/16 20.20.0.0/16

Prefix AS-PATH 10.10.0.0/16 100 10

18

Transparent AS

•  With route server setup

IXP A AS 100

AS10 AS20

10.10.0.0/16 20.20.0.0/16

Prefix AS-PATH 20.20.0.0/16 20

Prefix AS-PATH 10.10.0.0/16 10

19

Storm Control

•  Broadcast storm into an IXP a major challenge for the operator – beyond their control

•  IXP hardware to have better storm control capability or features to counter

•  Various hardware vendors has employed certain level of storm control detection and mitigation feature

Vendor Mechanism/Capability

Cisco Nexus •  Interface level (Threshold : Interface bandwidth)

Brocade MLX •  Interface level ACL/rate-limit •  Global Level / VPLS Level (Threshold : # of packets)

Extreme •  Interface level ACL/rate-limit •  Global/CPU level (Threshold : # of packets)

20

Summary of Best Practices

Members •  Disable unwanted traffic

towards IXP •  Do not loop towards IXP •  Do not leak IXP prefix to

Internet •  Peering with route servers •  Consistent route

announcement

Operator •  Port Security •  Prefix Filtering •  Configuration Automation •  Transparent AS •  Storm Control

21

Reference

•  AMS-IX •  https://www.ams-ix.net/technical/specifications-descriptions/

config-guide

•  Euro-IX •  https://www.euro-ix.net/ixp-bcp

22

chee-yong.tay@ap.equinix.com