nanog -1- orbit1000 technology discussion opnix smart routing technology overview ”there is more...
TRANSCRIPT
![Page 1: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/1.jpg)
Orbit1000 Technology DiscussionNANOG -1-
Opnix Smart Routing Technology Overview
”There is more then one way to skin a cat…”
Opnix Smart Routing Technology Overview
”There is more then one way to skin a cat…”
Aaron D. BrittOpnix, Inc.
![Page 2: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/2.jpg)
Orbit1000 Technology DiscussionNANOG -2-
Orbit1000 Technology Discussion Overview
• Orbit1000 CPE Overview• Probing Method in More detail• Orbit1000 CORE Overview• Things to Come…• Lets Review - Q & A
![Page 3: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/3.jpg)
Orbit1000 Technology DiscussionNANOG -3-
Orbit1000 CPE High Level Architecture Subscriber
AS 100
Carrier BAS 200
Carrier CAS 300
20.20.20.130.30.30.1
10.10.10.2
20.20.20.230.30.30.2
IP Block Advertised - (24.10.0.0/16)
OSPFArea 0
A
BC
EBGP
EBGPEBGP
IBGPIBGP
Orbit 1000
OrbitAS 64701
24.10.4.1
LAN 24.10.1.1
Opnix CORE
ENCRYPTED
![Page 4: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/4.jpg)
Orbit1000 Technology DiscussionNANOG -4-
Functions of the Orbit1000 CPE• Probe stuff• Receive BGP Feed and Set Routes• Communicate with the CORE
– Send Raw Probe Data– Receive Optimized Routes
Orbit1000 CPE
Discovery ProbesSet BGP Routes
QA Probes
InternetCustomerRouter(s)
CORE
EN
CR
YP
TE
D
![Page 5: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/5.jpg)
Orbit1000 Technology DiscussionNANOG -5-
How we become one with the Packet • UDP Probes – Proactive Philosophy using patented ActiveScan
– Tried ICMP - routers drop ICMP despite what RFC says– We tried TCP – set off IDS systems all over the place– We tried the force - but none of us had enough
metaclorians.– We now use a UDP probe, though proprietary in nature,
very similar to that of a typical traceroute.– We found that during testing, routing policy set using UDP
Probe data is within 2% of the routing policy set using TCP probe data, but it doesn’t set off IDS systems!
![Page 6: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/6.jpg)
Orbit1000 Technology DiscussionNANOG -6-
Probing Mechanism• Where do we probe?
– Prefix List based on prefixes important to each Customer• Top 500 Trafficked Sites/ News Groups etc…• Route Feed from Customer Routers• Traffic Flow Data (Netflow, Span Port <sniff sniff>)• Logs (Web, DNS etc…)• Capable of probing 110,000+ routes, but it doesn’t make
sense to (most of the time)– discovery.ignore and discovery.include lists.– ’Prefix + 1’ methodology, unless a more specific ip address is
specified in the configuration.• We probe multiple prefixes over multiple upstreams in parallel,
configurable amount – how much bandwidth do you want to spend on Probes?
![Page 7: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/7.jpg)
Orbit1000 Technology DiscussionNANOG -7-
Metrics Gathered • OpScore (Algorithm based on the probe data weighted, and
calculated based on customer defined settings)– Latency– Unreliability
• Link Unreliability• Probe Closure• Packet Loss• Routing Loops
– Bad Hops – Layer 3 Hops– Carrier Preference
• Lowest score wins
Prefix 216.183.192.0/19 Over Carrier "B" Prefix 216.183.192.0/19 Over Carrier "C"
Actual Weight Result Actual Weight Result25 25% 6.25 75 25% 18.75
Actual Weight Result Actual Weight Result15 10.00% 1.5 20 10.00% 2
Actual Weight Result Actual Weight Result1 10.00% 0.1 0 10.00% 0
Actual Weight Result Actual Weight Result50 25.00% 12.5 25 25.00% 6.25
Actual Weight Result Actual Weight Result125 30.00% 37.5 50 30.00% 15
OpScore 57.85 OpScore 42.00
Layer 3 Hops (range 2 to 30)
Bad Hops (range 1 to 5)
Unreliability (Range 1 - 100)
Latency (5 to 300 ms)
Carrier Preferenc (Range 100 - 1) Carrier Preference (Range 100 - 1)
Layer 3 Hops (range 2 to 30)
Bad Hops (range 1 to 5)
Unreliability (Range 1 - 100)
Latency (5 to 300 ms)
![Page 8: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/8.jpg)
Orbit1000 Technology DiscussionNANOG -8-
QA Process (Testing the Active Link)• UDP Based (Just like our Discovery Probes)• We QA everything!• We send the QA probe to a TTL based on where we think the
endpoint is based on our discovery data.• We check the latency and unreliability against the probe data
we used to set the route. • How many QA routes do we send, and how fast?
– The QA Limit is configurable like Carrier Limit in the Client Config – which means you control how many routes we can QA in parallel.
• QA happens much faster then Discovery.
![Page 9: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/9.jpg)
Orbit1000 Technology DiscussionNANOG -9-
Orbit1000 CORE• 5 Pieces
– Balancer (Communicates w/CPE)– Optimizer (Crunches Numbers)– View (Keeps Latest and Greatest Views per CPE)– SQL dB (Stores Stuff)– Customer Portal (Looks stuff up)
View
Customer Portal
SQL dB
Balancer OptimizerCPE
Portal
CORE
![Page 10: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/10.jpg)
Orbit1000 Technology DiscussionNANOG -10-
Data Access• Portal
– Access to Data, raw and graphical (Current and Historical)– All metrics and weights represented– Access to each CPE Client Config– RouteVision (Visualize over Multiple Paths)– Aggregate Summarizations
• SQL dB– Raw Data
• Transactional Data (Real Time)• Warehoused Data (Portal)• Archival Data
![Page 11: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/11.jpg)
Orbit1000 Technology DiscussionNANOG -11-
Fault Tolerance Stuff…• If it goes up in smoke, the Customer router reverts back to standard
BGP.• Discovery Probes halt if the CPE loses the CORE connection, if keep-
alives fail within a period of time, product removes routes and “sleeps” until communication with the CORE is reestablished.
• Fault Tolerant reasoning behind storing CPE config on central dB• Heartbeat / fail over process between CPE’s• SNMP traps, early warning system (RAM, Hard Disk, CPU etc..)• Always working on additional MIB support
![Page 12: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/12.jpg)
Orbit1000 Technology DiscussionNANOG -12-
Things to Come…• Probes to support Jumbo Frames (Adjustable Frame Size)• Dedicated Jitter Metrics• Black- hole and Routing Loop Discovery/reports via Website• TCP Slow Start Algorithm emulation• TCP and/or UDP probes (Pick your poison)• TCP Sniffing for Active Links (Monitor Actual Data – Replace QA)• Multicast Support• IPV6 Support• Additional MIB support• NEBS Compliant (just kidding)
![Page 13: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/13.jpg)
Orbit1000 Technology DiscussionNANOG -13-
Contact Information
If you have any questions or would like to comment and/or critique this method of ‘Cat Skinning’ (I would love for some hecklers to drop me a line, with-out peer review no progress is possible) here is my contact info…
http://[email protected]
Case Studies available today…• Tier 1 ISP• Fortune 5 Enterprise• Fortune 100 Financial Institution• Internet2/Abilene Deployment
![Page 14: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/14.jpg)
Orbit1000 Technology DiscussionNANOG -14-
Layer 3 Hops vs latency (30 day Summary)
ttl: 3 latency: 0.020716ttl: 4 latency: 0.024832ttl: 5 latency: 0.033791ttl: 6 latency: 0.045662ttl: 7 latency: 0.055674ttl: 8 latency: 0.079405ttl: 9 latency: 0.109979ttl: 10 latency: 0.131937ttl: 11 latency: 0.141727ttl: 12 latency: 0.142373ttl: 13 latency: 0.143105ttl: 14 latency: 0.151558ttl: 15 latency: 0.177103ttl: 16 latency: 0.196629ttl: 17 latency: 0.216883ttl: 18 latency: 0.231439ttl: 19 latency: 0.244841ttl: 20 latency: 0.263682ttl: 21 latency: 0.268043
0
0.05
0.1
0.15
0.2
0.25
0.3
1 3 5 7 9 11 13 15 17 19
Series1
![Page 15: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/15.jpg)
Orbit1000 Technology DiscussionNANOG -15-
Prefixes are how many hops away?
ttl: 3 # prefixes: 2047ttl: 4 # prefixes: 473ttl: 5 # prefixes: 660ttl: 6 # prefixes: 1621ttl: 7 # prefixes: 2726ttl: 8 # prefixes: 3601ttl: 9 # prefixes: 4340ttl: 10 # prefixes: 5527ttl: 11 # prefixes: 7831ttl: 12 # prefixes: 8761ttl: 13 # prefixes: 9111ttl: 14 # prefixes: 13756ttl: 15 # prefixes: 9506ttl: 16 # prefixes: 7743ttl: 17 # prefixes: 7174ttl: 18 # prefixes: 4679ttl: 19 # prefixes: 4321ttl: 20 # prefixes: 2881ttl: 21 # prefixes: 1339
0
2000
4000
6000
8000
10000
12000
14000
16000
1 3 5 7 9 11 13 15 17 19
Series1
![Page 16: NANOG -1- Orbit1000 Technology Discussion Opnix Smart Routing Technology Overview ”There is more then one way to skin a cat…” Opnix Smart Routing Technology](https://reader033.vdocuments.us/reader033/viewer/2022051216/56649e725503460f94b715d2/html5/thumbnails/16.jpg)
Orbit1000 Technology DiscussionNANOG -16-
Other Questions to ask…
• Is there a direct correlation between Hops and Latency? Hop count seems anecdotal, yet the numbers are quite convincing…
• How accurate does UDP measurements compare with TCP measurements when talking about Latency, Packet Loss and Throughput?
• How much does Asymmetrical routing, play a part in the world of Sub optimal routing?
• With Netflow stats, on average it seems that Routers only forward packets to 10% or so of the Global Rib, yet our routing Tables are tenfold +. Seems we can do something here, I just don’t know what, yet…