1 isp design
TRANSCRIPT
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 2
ISP Network Design
PoP Topologies and Design
Backbone Design
ISP Systems Design
Addressing
Routing Protocols
Security
Out of Band Management
Operational Considerations
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 4
PoP Topologies
Core routers – high speed trunk connections
Distribution routers and Access routers – high portdensity
Border routers – connections to other providers
Service routers – hosting and servers
Some functions might be handled by a single router
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 5
PoP Design
Modular Design
Aggregation Services separated according toconnection speedcustomer servicecontention ratiosecurity considerations
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 6
Modular PoP Design
Backbone linkto another PoP
Backbone linkto another PoP
Leased line customeraggregation layer
for leased line circuit deliveryChannelised circuits
NetworkOperations
Centre
Consumer
DIal Access
NetworkCore
Consumer cable, xDSL and
wireless Access
for MetroE circuit deliveryGigE fibre trunks
MetroE customeraggregation layer
ISP Services(DNS, Mail, News,
FTP, WWW)
Hosted Services &Datacentre
Other ISPsWeb Cache
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 7
Modular Routing Protocol DesignSmaller ISPs
Modular IGP implementationIGP “area” per PoPCore routers in backbone area (Area 0/L2)Aggregation/summarisation where possible into the core
Modular iBGP implementationBGP route reflector cluster per moduleCore routers are the route-reflectorsRemaining routers are clients & peer with route-reflectors only
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 8
Modular Routing Protocol DesignLarger ISPs
Modular IGP implementationIGP “area” per module (but avoid overloading core routers)Core routers in backbone area (Area 0/L2)Aggregation/summarisation where possible into the core
Modular iBGP implementationBGP route reflector cluster per moduleDedicated route-reflectors adjacent to core routersClients peer with route-reflectors only
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 10
PoP Modules
Low Speed customer connectionsPSTN/ISDN dialupLow bandwidth needsLow revenue, large numbers
Leased line customer connectionsE1/T1 speed rangeDelivery over channelised mediaMedium bandwidth needsMedium revenue, medium numbers
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 11
PoP Modules
Broad Band customer connectionsxDSL, Cable and WirelessHigh bandwidth needsLow revenue, large numbers
MetroE & Highband customer connectionsTrunk onto GigE or 10GigE of 10Mbps and higherChannelised OC3/12 delivery of E3/T3 and higherHigh bandwidth needsHigh revenue, low numbers
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 12
PoP Modules
PoP CoreTwo dedicated routersHigh Speed interconnectBackbone Links ONLYDo not touch them!
Border NetworkDedicated border router to other ISPsThe ISP’s “front” doorTransparent web caching?Two in backbone is minimum guarantee for redundancy
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 13
PoP Modules
ISP ServicesDNS (cache, secondary)News (still relevant?)Mail (POP3, Relay, Anti-virus/anti-spam)WWW (server, proxy, cache)
Hosted Services/DataCentresVirtual Web, WWW (server, proxy, cache)Information/Content ServicesElectronic Commerce
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 14
PoP Modules
Network Operations CentreConsider primary and backup locationsNetwork monitoringStatistics and log gatheringDirect but secure access
Out of Band Management NetworkThe ISP Network “Safety Belt”
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 15
Low Speed Access Module
To Core Routers
Primary Rate T1/E1
PSTN lines tomodem bank
PSTN lines tobuilt-in modems
AS5400
2811
2800/3800
TACACS+/Radiusproxy, DNS resolver,
Content
Web Cache
Access NetworkGateway Routers
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 16
Medium Speed Access Module
To Core Routers
Channelised T1/E1
64K and nx64K circuits
Mixture of channelisedT1/E1, 56/64K and
nx64K circuits
3800/7206/7600
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 17
High Speed Access Module
To Core Routers
Metro Ethernet
Channelised T3/E3
Channelised OC3/OC12
7200/7600/ASR1000/ASR9000
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 18
Broad Band Access Module
To Core Routers
Telephone Network
The cable system
6400
SSG, DHCP, TACACS+or Radius Servers/Proxies,
DNS resolver, Content
Web Cache
Access NetworkGateway Routers
uBR7246
61xx
IP, ATM
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 19
ISP Services Module
DNScache
DNSsecondary POP3 Mail
Relay NEWS
To core routers
WWWcache
Service NetworkGateway Routers
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 20
Hosted Services Module
Customer 7Customer 3Customer 4
Customer 5Customer 6
To core routers
Hosted NetworkGateway Routers
Customer 2Customer 1
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 21
Border Module
To core routers
NetworkBorder Routers
To local IXP -NB - no default route +
local AS routing table only
ISP1 ISP2
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 22
NOC Module
Primary DNS
To core routers
Hosted NetworkGateway Routers
SYSLOGserver
TACACS+server
Network Operations Centre Staff
Out of Band
Management Network
2811/32async
NetFlowAnalyser
Firewall
Billing, Database and Accounting
Systems
Corporate LAN
Critical ServicesModule
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 23
Out of Band Network
Out of Band
Management Network
2811/32asyncTo the NOC
Out of Band Ethernet
NetFlow
Collector
NetFlow
enabled
routers
Routerconsoles
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 25
Backbone Design
Routed Backbone
Switched BackboneVirtually obsolete
Point-to-point circuitsnx64K, T1/E1, T3/E3, OC3, OC12, GigE, OC48, 10GigE, OC192
ATM/Frame Relay service from telcoT3, OC3, OC12,… deliveryEasily upgradeable bandwidth (CIR)Almost vanished in availability now
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 26
Distributed Network Design
PoP design “standardised”operational scalability and simplicity
ISP essential services distributed around backbone
NOC and “backup” NOC
Redundant backbone links
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 27
Distributed Network Design
POP One
POP Two
POP Three
Customerconnections
Customerconnections
Customerconnections
Externalconnections
Externalconnections Operations Centre
BackupOperations Centre
ISP Services
ISP Services
ISP Services
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 28
Backbone Links
ATM/Frame RelayVirtually disappeared due to overhead, extra equipment, andshared with other customers of the telcoMPLS has replaced ATM & FR as the telco favourite
Leased Line/CircuitMost popular with backbone providersIP over Optics and Metro Ethernet very common in many partsof the world
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 29
Long Distance Backbone Links
Tend to cost more
Plan for the future (at least two years ahead) but stay inbudget
Unplanned “emergency” upgrades can be disruptive withoutredundancy
Allow sufficient capacity on alternative paths for failuresituations
Sufficient can be 20% to 50%Some businesses choose 0% – meaning they have no sparecapacity at all!!
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 30
Long Distance Links
POP One
POP Two
POP Three
Long distance link
Alternative/Backup Path
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 31
Metropolitan Area Backbone Links
Tend to be cheaperCircuit concentrationChoose from multiple suppliers
Think bigMore redundancyLess impact of upgradesLess impact of failures
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 32
Metropolitan Area Backbone Links
POP One
POP Two
POP Three
Metropolitan Links
Metropolitan Links
Traditional Point to Point Links
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 33
ISP Services
DNS, Mail, NewsDesign and Placement
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 34
ISP Services
Most ISP services such as DNS, Mail, etc are easilydeliverable on low budget hardware platforms
Single Rack Unit in height (1RU)Dual processor is “default” nowRAM is very cheap (may as well use 2Gbytes or more)Hard drives are very cheap (SCSI more reliable)Unix like operating systems (FreeBSD, Debian, Ubuntu,CentOS) are very common
In addition to commercial operating systems such as Solaris,RedHat Enterprise Linux &c
Minimal overhead, minimal OS install, plus the service required
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 35
ISP Services:DNS
Domain Name SystemProvides name and address resolutionServers need to be differentiated, properly located and specified
Primary nameserverSecondary nameserverCaching nameserver – resolver
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 36
ISP Services:DNS
Primary nameserverHolds ISP zone files
Forward zone (list of name to address mappings) for allISP’s and any customer zonesReverse zone (list of address to name mappings) for allISP’s address space
Hardware & OS: easily satisfied by simple specificationLocated in secure part of net, e.g. NOC LANUsually run as “hidden master” – secondary nameservers arethe official listed nameservers
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 37
ISP Services:DNS
Secondary nameserverHolds copies of ISP zone filesAt least two are required, more is betterHardware & OS: easily satisfied by simple specificationStrongly recommended to be geographically separate fromeach other and the primary DNS
At different PoPsOn a different continent e.g. via services offered by ISC,PCH and othersAt another ISP
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 38
$ dig apnic.net ns;; ANSWER SECTION:apnic.net. 10800 NS ns1.apnic.net.apnic.net. 10800 NS ns3.apnic.net.apnic.net. 10800 NS ns4.apnic.net.apnic.net. 10800 NS ns5.apnic.com.apnic.net. 10800 NS cumin.apnic.net.apnic.net. 10800 NS ns-sec.ripe.net.apnic.net. 10800 NS tinnie.arin.net.apnic.net. 10800 NS tinnie.apnic.net.
;; ADDITIONAL SECTION:ns1.apnic.net. 3600 A 202.12.29.25ns3.apnic.net. 3600 A 202.12.28.131ns4.apnic.net. 3600 A 202.12.31.140ns5.apnic.com. 10800 A 203.119.43.200cumin.apnic.net. 3600 A 202.12.29.59tinnie.apnic.net. 3600 A 202.12.29.60ns-sec.ripe.net. 113685 A 193.0.0.196tinnie.arin.net. 10800 A 199.212.0.53
TokyoHong KongWashington
Brisbane
ISP Services:Secondary DNS Example
Brisbane
AmsterdamWashington
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 39
ISP Services:Secondary DNS Example
apnic.net zonePrimary DNS in Brisbane (ns1.apnic.net)Secondary DNS run all over the world by APNIC:
BrisbaneHong KongTokyoWashington
Zone secondaried byRIPE NCC in AmsterdamARIN in Washington
Geographical and service provider redundancy – this is theperfect example!
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 40
ISP Services:DNS
Caching nameserverThis is the resolver – it is the DNS cacheYour customers use this as resolver, NEVER your primary orsecondary DNSProvides very fast lookupsDoes NOT secondary any zonesOne, or preferably two per PoP (redundancy)Hardware & OS: easily satisfied by simple specification
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 41
ISP Services:Caching Nameserver
To Core Routers
DIAL network
Web Cache
DNS Cache DNS Cache
Radius proxy
Switch redundancyRouter redundancyDNS Cache redundancy
DIAL users automatically given the IP addresses of DNS cacheswhen they dial in
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 42
GeekAlert
ISP Services:Anycasting the Caching Nameserver
One trick of the tradeassign two unique IP addresses to be used for the two DNS resolver systemsuse these two IP addresses in every PoProute the two /32s across your backboneeven if the two resolver systems in the local PoP aredown, the IGP will ensure that the next nearest resolverswill be reachableKnown as IP Anycast
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 43
ISP Services:DNS
Efficient and resilient designPrimary DNS – keep it secureSecondary DNS – geographical and provider redundancy
Don’t ever put them on the same LAN, switched or otherwiseDon’t put them in the same PoP
Caching DNS – one or two per PoPReduces DNS traffic across backboneMore efficient, spreads the load
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 44
ISP Services:DNS
SoftwareMake sure that the BIND distribution on the Unix system is up todate
The vendor’s distribution is rarely currentPay attention to bug reports, security issuesReboot the DNS cache on a regular (e.g. monthly) basis
Clears out the cacheReleases any lost RAMAccepted good practice by system administrators
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 45
ISP Services:DNS
ImplementationPut all your hosts, point-to-point links and loopbacks into theDNS
Under your ISP’s domain nameUse sensible/meaningful names
Put all your hosts, point-to-point links and loopbacks into theREVERSE DNS also
Don’t forget about in-addr.arpa and ip6.arpa – many ISPs doSome systems demand forward/reverse DNS mappingbefore allowing access
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 46
ISP Services:Mail
Must have at least two mail hosts (MX records) for all supporteddomains
Geographical separation helps
Dedicated POP3 serverConsumers/mobile users get mail from here
SMTP gateway dedicated to that functionConsumers/mobile users send mail via here
Mail relay open to CUSTOMERS only!Don’t let outside world use your mail relay
Block port 25 outbound for all customersInsist that outbound e-mail goes through SMTP relaySMTP relay does virus (ClamAV) and spam (Spamassassin) filtering
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 47
ISP Services:Mail Configuration
smtp.isp.net
Customer mail relay
Incoming mailfrom customer
mail.isp.net
ISP Mail Gateway
Incoming mailfrom Internet
pop3.isp.net
Customer POP3/IMAP server
Mail pulled bycustomer client
Mail out tothe Internet
SpamAssassinClamAV
SpamAssassinClamAV
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 48
$ dig cisco.com mx
;; ANSWER SECTION:cisco.com. 86400 MX 10 sj-inbound-a.cisco.com.cisco.com. 86400 MX 10 sj-inbound-b.cisco.com.cisco.com. 86400 MX 10 sj-inbound-c.cisco.com.cisco.com. 86400 MX 10 sj-inbound-d.cisco.com.cisco.com. 86400 MX 10 sj-inbound-e.cisco.com.cisco.com. 86400 MX 10 sj-inbound-f.cisco.com.cisco.com. 86400 MX 15 rtp-mx-01.cisco.com.cisco.com. 86400 MX 20 ams-inbound-a.cisco.com.cisco.com. 86400 MX 25 syd-inbound-a.cisco.com.
ISP Services:Mail Example
cisco.com mail (MX records)primary MX are 6 systems in San JoseThree backup MXes in RTP, Amsterdam and Sydneybackup MX only used if primary unavailable
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 49
IMPORTANT: Do NOT allow non-customers to use your mail system as a relay
ISP Services:Mail
SoftwareMake sure that the MAIL and POP3 distributions on the Unixsystem are up to date
The vendor distributions are rarely currentPay attention to bug reports, security issues, unsolicited junkmail complaints
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 50
ISP Services:News
News servers provide a Usenet news feed to customers
Distributed design requiredIncoming newsfeed to one large serverDistributed to feed servers in each PoPFeed servers provide news feed to customersOutgoing news goes to another serverSeparate reading news systemSeparate posting news system
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 51
ISP Services:News System Placement
POP One
POP Two
POP Three
Customerconnections
Customerconnections
Customerconnections
Externalconnections
Externalconnections News Collector
News Feeder
News Feeder
News Feeder
News Distributor
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 52
ISP Services:News System Placement
POP One
POP Two
POP Three
Customerconnections
Customerconnections
Customerconnections
Externalconnections
Externalconnections News Collector
News Feeder
News Feeder
News Feeder
News Distributor
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 53
IMPORTANT: Do NOT allow non-customers to use your news system for posting messages
ISP Services:News
SoftwareMake sure that the Internet News distribution on the Unixsystem is up to date
The vendor distributions are rarely currentPay attention to bug reports, security issues, unsolicited junkposting complaints
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 55
Where to get IP addresses and AS numbers
Your upstream ISP
AfricaAfriNIC – http://www.afrinic.net
Asia and the PacificAPNIC – http://www.apnic.net
North AmericaARIN – http://www.arin.net
Latin America and the CaribbeanLACNIC – http://www.lacnic.net
Europe and Middle EastRIPE NCC – http://www.ripe.net/info/ncc
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 57
Getting IP address space
Take part of upstream ISP’s PA spaceor
Become a member of your Regional Internet Registryand get your own allocation
Require a plan for a year aheadGeneral policies are outlined in RFC2050, more specific detailsare on the individual RIR website
There is still plenty of IPv4 address spaceRegistries require high quality documentationWhen applying for IPv4 addresses, get an IPv6 allocation too!
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 58
What about RFC1918 addressing?
RFC1918 defines IP addresses reserved for privateInternets
Not to be used on Internet backboneshttp://www.ietf.org/rfc/rfc1918.txt
Commonly used within end-user networksNAT used to translate from private internal to public externaladdressingAllows the end-user network to migrate ISPs without a majorinternal renumbering exercise
Most ISPs filter RFC1918 addressing at their networkedge
http://www.cymru.com/Documents/bogon-list.html
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 59
What about RFC1918 addressing?
List of well known problems with this approach for an SP backbone:Breaks Path MTU DiscoveryPotential conflicts with usage of private addressing inside customernetworksSecurity through obscurity does not provide securityTroubleshooting outside the local network becomes very hard
Router interface addresses are only locally visibleInternet becomes invisible from the router
Troubleshooting of connectivity issues on an Internet scale becomesimpossible
Traceroutes and pings provide no informationNo distinction between “network invisible” and “network broken”
Increases operational complexity of the network infrastructure and routingconfiguration
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 60
Private versus Globally Routable IPAddressing
Infrastructure Security: not improved by using privateaddressing
Still can be attacked from inside, or from customers, or byreflection techniques from the outside
Troubleshooting: made an order of magnitude harderNo Internet view from routersOther ISPs cannot distinguish between down and broken
Performance: PMTUD breakage
Summary:ALWAYS use globally routable IP addressing for ISPInfrastructure
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 61
Addressing Plans – ISP Infrastructure
Address block for router loop-back interfaces
Address block for infrastructurePer PoP or whole backboneSummarise between sites if it makes senseAllocate according to genuine requirements, not historic classfulboundaries
Similar allocation policies should be used for IPv6 aswell
ISPs just get a substantially larger block (relatively) soassignments within the backbone are easier to make
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 62
Addressing Plans – Customer
Customers are assigned address space according toneed
Should not be reserved or assigned on a per PoP basisISP iBGP carries customer netsAggregation not required and usually not desirable
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 63
223.10.0.0/21
Customer assignments Infrastructure Loopbacks
/24223.10.6.255223.10.0.1
223.10.0.0/20
Original assignments New Assignments
/24/24223.10.0.1
223.10.5.255 223.10.15.255
Addressing Plans – ISP Infrastructure
Phase One
Phase Two
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 64
Addressing PlansPlanning
Registries will usually allocate the next block to becontiguous with the first allocation
Minimum allocation could be /21Very likely that subsequent allocation will make this up to a /20So plan accordingly
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 65
Addressing Plans (contd)
Document infrastructure allocationEases operation, debugging and management
Document customer allocationContained in iBGPEases operation, debugging and managementSubmit network object to RIR Database
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 67
Routing Protocols
IGP – Interior Gateway Protocolcarries infrastructure addresses, point-to-point linksexamples are OSPF, ISIS, EIGRP...
EGP – Exterior Gateway Protocolcarries customer prefixes and Internet routescurrent EGP is BGP version 4
No connection between IGP and EGP
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 68
Why Do We Need an IGP?
ISP backbone scalingHierarchyModular infrastructure constructionLimiting scope of failureHealing of infrastructure faults using dynamic routing with fastconvergence
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 69
Why Do We Need an EGP?
Scaling to large networkHierarchyLimit scope of failure
PolicyControl reachability to prefixesMerge separate organizationsConnect multiple IGPs
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 70
Interior versus Exterior RoutingProtocols
InteriorAutomatic neighbour discoveryGenerally trust your IGProutersPrefixes go to all IGP routersBinds routers in one AStogether
ExteriorSpecifically configured peersConnecting with outsidenetworksSet administrative boundariesBinds AS’s together
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 71
Interior versus Exterior RoutingProtocols
InteriorCarries ISP infrastructureaddresses onlyISPs aim to keep the IGPsmall for efficiency andscalability
ExteriorCarries customer prefixesCarries Internet prefixesEGPs are independent of ISPnetwork topology
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 72
BGP4
Hierarchy of Routing Protocols
BGP4and OSPF/ISIS
Other ISPs
CustomersIXP
Static/BGP4
BGP4
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 73
Routing Protocols:Choosing an IGP
Review the “Introduction to Link State Protocols”presentation
i.e. – OSPF and ISIS have very similar properties
ISP usually chooses between OSPF and ISISChoose which is appropriate for your operators’ experienceIn IOS, both OSPF and ISIS have sufficient “nerd knobs” totweak the IGP’s behaviourOSPF runs on IPISIS runs on infrastructure, alongside IP
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 74
Routing Protocols:IGP Recommendations
Keep the IGP routing table as small as possibleIf you can count the routers and the point to point links in thebackbone, that total is the number of IGP entries you shouldsee
IGP details:Should only have router loopbacks, backbone WAN point-to-point link addresses, and network addresses of any LANshaving an IGP running on themStrongly recommended to use inter-router authenticationUse inter-area summarisation if possible
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 75
Routing Protocols:More IGP recommendations
To fine tune IGP table size more, consider:Using “ip unnumbered” on customer point-to-point links – savescarrying that /30 in IGP
(If customer point-to-point /30 is required for monitoring purposes,then put this in iBGP)
Use contiguous addresses for backbone WAN links in eacharea – then summarise into backbone areaDon’t summarise router loopback addresses – as iBGP needsthose (for next-hop)Use iBGP for carrying anything which does not contribute to theIGP Routing process
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 76
Routing Protocols:iBGP Recommendations
iBGP should carry everything which doesn’t contributeto the IGP routing process
Internet routing tableCustomer assigned addressesCustomer point-to-point linksDial network pools, passive LANs, etc
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 77
Routing Protocols:More iBGP Recommendations
Scalable iBGP features:Use neighbour authenticationUse peer-groups to speed update process and for configurationefficiencyUse communities for ease of filteringUse route-reflector hierarchy
Route reflector pair per PoP (overlaid clusters)
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 79
Security
ISP Infrastructure security
ISP Network security
Security is not optional!
ISPs need to:Protect themselvesHelp protect their customers from the InternetProtect the Internet from their customers
The following slides are general recommendationsDo more research on security before deploying any network
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 80
ISP Infrastructure Security
Router securityUsernames, passwords, vty filters, TACACS+Disable telnet on vtys, only use SSHvty filters should only allow NOC access, no external accessSee IOS Essentials for the recommended practices for ISPs
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 81
ISP Infrastructure Security
ISP server securityUsernames, passwords, TCP wrappers, IPTABLESProtect all servers using routers with strong filters applied
Hosted services securityProtect network from hosted servers using routers with strongfiltersProtect hosted servers from Internet using routers with strongfilters
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 82
DNScache
DNSsecondary POP3 Mail
Relay NEWS
To core routers
Service NetworkGateway Routers
Access-list examples:
Allow tcp/established to all serversICMPDNS 2ary: udp/53 and tcp/53POP3: tcp/110Mail Relay: tcp/25 and ISP address
range onlyNews: tcp/119 and ISP
address range onlyDNS Cache: udp/53Web server: tcp/80
Other necessary filters:
All servers: SSH (tcp/22) from NOC LAN only
Webserver
ISP Infrastructure SecurityISP Server Protection
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 83
Access-list examples:InboundAllow tcp/established to all serversICMPWeb server: tcp/80SSH for customer accessAny other ports for services
sold to customers
OutboundICMPAllow DNS udp/53 and
tcp/53Block all access to ISP
address rangeServer5Server1 Server2 Server3 Server4
To core routers
Service NetworkGateway Routers
Server6
ISP Infrastructure SecurityHosted Server Protection
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 84
ISP Infrastructure Security
Premises securityLocks – electronic/card key preferredSecure access – 24x7 security arrangementsEnvironment control – good aircon
Staff responsibilityPassword policy, strangers, temp staffEmployee exit procedures
RFC2196(Site Security Handbook)
RFC3871(Operational Security Requirements for Large ISP IP NetworkInfrastructure )
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 85
ISP Network Security
Denial of Service Attackseg: “smurfing”see http://www.denialinfo.com
Effective filteringNetwork borders – see Cisco ISP EssentialsCustomer connections – unicast RPF on ALL of themNetwork operation centreISP corporate network – behind firewall
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 86
ISP Network SecuritySecure external access
How to provide staff access from outsideSet up ssh gateway (Unix system with ssh daemon and nothingelse configured)Provide ssh client on all staff laptopsssh available on Unix and Windowsssh is Secure Shell – encrypted link
How not to provide access from outsidetelnet, rsh, rlogin – these are all insecureOpen host – insecure, can be compromised
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 87
Ingress & Egress Route Filtering
Your customers should not besending any IP packets out to the
Internet with a source addressother then the address you have
allocated to them!
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 89
Out of Band Management
Not optional! Allows access to network equipment in times of failure
Ensures quality of service to customersMinimises downtimeMinimises repair timeEases diagnostics and debugging
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 90
Out of Band Management
OoB Example – Access server:modem attached to allow NOC dial inconsole ports of all network equipment connected to serial portsLAN and/or WAN link connects to network core, or via separatemanagement link to NOC
Full remote control access under all circumstances
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 91
Out of Band Network
Ethernetto the NOC
Router, switchand ISP server
consoles
(Optional) Out of bandWAN link to other PoPs
Modem – accessto PSTN for out of
band dialin
Equipment RackEquipment Rack
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 92
Out of Band Management
OoB Example – Statistics gathering:Routers are NetFlow and syslog enabledManagement data is congestion/failure sensitiveEnsures management data integrity in case of failure
Full remote information under all circumstances
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 94
Test Laboratory
Designed to look like a typical PoPOperated like a typical PoP
Used to trial new services or new software underrealistic conditions
Allows discovery and fixing of potential problems beforethey are introduced to the network
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 95
Test Laboratory
Some ISPs dedicate equipment to the lab
Other ISPs “purchase ahead” so that today’s labequipment becomes tomorrow’s PoP equipment
Other ISPs use lab equipment for “hot spares” in theevent of hardware failure
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 96
Test Laboratory
Can’t afford a test lab?Set aside one spare router and server to trial new servicesNever ever try out new hardware, software or services on thelive network
Every major ISP in the US and Europe has a test labIt’s a serious consideration
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 98
Operational Considerations
Why design the world’s best networkwhen you have not thought about whatoperational good practices should be
implemented?
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 99
Operational ConsiderationsMaintenance
Never work on the live network, no matter how trivialthe modification may seem
Establish maintenance periods which your customers are awareof
e.g. Tuesday 4-7am, Thursday 4-7am
Never do maintenance on a FridayUnless you want to work all weekend cleaning up
Never do maintenance on a MondayUnless you want to work all weekend preparing
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 100
Operational ConsiderationsSupport
Differentiate between customer support and theNetwork Operations Centre
Customer support fixes customer problemsNOC deals with and fixes backbone and Internet relatedproblems
Network Engineering team is last resortThey design the next generation network, improve the routingdesign, implement new services, etcThey do not and should not be doing support!
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 101
Operational ConsiderationsNOC Communications
NOC should know contact details for equivalent NOCsin upstream providers and peers
Or consider joining the INOC-DBA systemVoice over IP phone system using SIPRuns over the Internetwww.pch.net/inoc-dba for more information
© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 103
ISP Design Summary
KEEP IT SIMPLE & STUPID ! (KISS)
Simple is elegant is scalable
Use Redundancy, Security, and Technology to makelife easier for yourself
Above all, ensure quality of service for your customers