when bgp meets big-data · what can we do with large-scale collection of historical event...
TRANSCRIPT
![Page 1: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/1.jpg)
When BGP meets Big-Data
![Page 2: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/2.jpg)
2 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
• Millions of BGP events occurring every day
• How do we extract ‘signal’ from ‘noise’? • Can we apply techniques from other domains in
this pursuit?
The Internet is very much ‘alive’
• 15 Routers Monitored
• 410 active peers (both IPv4 and IPv6)
• ~120,000,000 Prefixes Advertised
• ~950,000 events per day from a single transit peer
• ~202,000,000 changes per day
• ~6,000,000,000 changes per month
![Page 3: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/3.jpg)
3 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
• If we know the questions we want to ask, how do we ask them?
• Enhance traditional dampening and suppression with analytics
The Internet is very much ‘alive’
![Page 4: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/4.jpg)
4 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
Five Monitoring Points in BGP Peer-B MPBGP
Static
Post rib
ISIS OSPF Adj-RIB-In (Pre-Policy)
EPE
Distribute/BGP-LS
Peer-A MPBGP
Adj-RIB-In (Post-Policy) Post rib
rib Adj-RIB-Out (Pre-Policy)
Adj-RIB-Out (Post-Policy)
Peer-C MPBGP
Local-RIB rib
Selection/Use
![Page 5: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/5.jpg)
5 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
Five Monitoring Points in BGP Peer-B MPBGP
Static
Post rib
ISIS OSPF Adj-RIB-In (Pre-Policy)
EPE
Distribute/BGP-LS
Peer-A MPBGP
Adj-RIB-In (Post-Policy) Post rib
rib Adj-RIB-Out (Pre-Policy)
Adj-RIB-Out (Post-Policy)
Peer-C MPBGP
Local-RIB rib
Selection/Use
RFC7854 BMP
draft-evens-grow-bmp-local-rib draft-evens-grow-bmp-adj-rib-out
![Page 6: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/6.jpg)
6 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
SNAS Architecture
TCP Listener
Connection Thread
Ingress Buffer
Parser (BMP & BGP)
Produce
RAW (Native BGP)
Textual (JSON/CSV)
Kafka
collector
Router BMP Feed
database
Consumer
SQL Transformation
DB Connection
MariaDB API
Web UI
![Page 7: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/7.jpg)
7 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
SNAS Architecture
TCP Listener
Connection Thread
Ingress Buffer
Parser (BMP & BGP)
Produce
RAW (Native BGP)
Textual (JSON/CSV)
Kafka
collector
Router BMP Feed
database
Consumer
SQL Transformation
DB Connection
MariaDB API
Web UI
App
![Page 8: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/8.jpg)
8 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
SNAS Architecture
TCP Listener
Connection Thread
Ingress Buffer
Parser (BMP & BGP)
Produce
RAW (Native BGP)
Textual (JSON/CSV)
Kafka
collector
Router BMP Feed
database
Consumer
SQL Transformation
DB Connection
MariaDB API
Web UI
App
![Page 9: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/9.jpg)
9 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
E2E architecture • Encoding app required to
perform ‘avro’ encoding of BMP data
• BGP App runs as Spark batch job, running periodically
• Can be converted to a Spark ‘streaming’ application for near-real-time processing
Encoding app
SNAS (collector)
![Page 10: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/10.jpg)
10 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
SNAS.io gives us the ability to record the dynamics of the Internet PNDA platform enables - • ‘Raw’ event recording capability, with horizontal scaling (HDFS) • Run analysis over very large data-sets with parallelism • Ask questions of the aggregate data about the Internet • Ask specific question
• Per-prefix • Per-AS • Per AS-Path
What does this give us?
![Page 11: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/11.jpg)
11 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
Top-N analysis
![Page 12: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/12.jpg)
12 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
Path stability
![Page 13: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/13.jpg)
13 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
AS Connectivity - FLAG
![Page 14: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/14.jpg)
14 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
AS Connectivity – Deutsche Telekom
![Page 15: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/15.jpg)
15 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
Prefix to Path history
Path Len = 9
Path Len = 5
![Page 16: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/16.jpg)
16 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
AS Path variance – 6939 to 8386 Shortest path – 3 hops Longest path – 28 hops Longest unique AS path – 5 Unique paths - 9 Largest prepend count – 17x Prepend variation – [7-17] Path with most updates – via AS1273 Data recorded in a 24hr period
![Page 17: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/17.jpg)
17 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
AS Path variance – 6939 to 8386 Shortest path – 4 hops Longest path – 29 hops Longest unique AS path – 6 Unique paths - 9 Largest prepend count – 17 Prepend variation – [7-17] Path with most updates – via AS1273 Data recorded in a 24hr period
![Page 18: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/18.jpg)
18 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
Security – Short prefix / long prefix detection
Default Route and Long prefix injection detected
![Page 19: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/19.jpg)
19 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
Security – Unallocated prefixes
Observed over a 12 hour period
![Page 20: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/20.jpg)
20 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
Security – Prefix drill-down
![Page 21: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/21.jpg)
21 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
Security – drill-down
![Page 22: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/22.jpg)
22 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
• AS 12345 originates 100.100.0.0/18 • Hijacker originates 100.100.63.0/24 • Basically a needle in a large haystack, does anyone notice? • What does RPKI show? • Do the origin ASNs match? • Does the less specific share the same transit set or similar as_paths? • Does RIR have the same organization name or contacts for both
origins? • Anything out of the norm for the new originating ASN?
More specific prefix detection
![Page 23: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/23.jpg)
23 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
What can we do with large-scale collection of historical event information? • Event impact analysis –
• Stability • Security • Misconfiguration • Forensics
• Application of ML/DL to data-set • Pattern-detection and network ‘weather forecasting’
Potential
![Page 24: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/24.jpg)
24 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
PNDA.io – the platform
![Page 25: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/25.jpg)
25 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
What is PNDA? PNDA brings together a number of open source technologies to provide a simple, scalable open big data analytics Platform for Network Data Analytics Linux Foundation Collaborative Project based on the Apache ecosystem
![Page 26: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/26.jpg)
26 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
• Linux Foundation project
• Selected by MEF for Analytics function within Lifecycle Service Orchestration framework
• In service trials with two Service Providers
• One platform supporting a range of use-cases including • Network security – Apache Spot • 6CN • Virtualization infrastructure monitoring and analysis • Smart Cities • Smart Transportion use-cases
Where is PNDA today?
![Page 27: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/27.jpg)
27 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
• Horizontally scalable platform for analytics and data processing applications
• Support for near-real-time stream processing and in-depth batch analysis on massive datasets
• Decouples data collection and aggregation from data analysis
• Consuming applications can be either platform apps developed for PNDA or client apps integrated with PNDA
• Client apps can use one of several structured query interfaces or consume streams directly.
• Leverages best current practise in big data analytics
PNDA
PNDA Plugins
ODL
Logstash
SNAS
pmacct
IOS XR Telemetry
Real -time
Data D
istribution File Store
Platform Services: Installation, Mgmt, Security, Data Privacy
App Packaging and Mgmt
Stream
Batch
Processing
SQL Query
OLAP Cube
Search/ Lucene
NoSQL Time Series
Data Exploration
Metric Visualisation
Event Visualisation PNDA
Mnged App
PNDA Mnged App
Unmnged App
Unmnged App
Query Visualisation and Exploration
PNDA Applications
PNDA Producer API
PNDA Consumer API
Bulk ingest
Custom
![Page 28: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/28.jpg)
28 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
• Simple, scalable open data platform
• Provides a common set of services for developing analytics applications
• Accelerates the process of developing big data analytics applications whilst significantly reducing the TCO
• PNDA provides a platform for convergence of network data analytics
PNDA
PNDA Plugins
ODL
Logstash
SNAS
pmacct
IOS XR Telemetry
Real -time
Data D
istribution File Store
Platform Services: Installation, Mgmt, Security, Data Privacy
App Packaging and Mgmt
Stream
Batch
Processing
SQL Query
OLAP Cube
Search/ Lucene
NoSQL Time Series
Data Exploration
Metric Visualisation
Event Visualisation PNDA
Mnged App
PNDA Mnged App
Unmnged App
Unmnged App
Query Visualisation and Exploration
PNDA Applications
PNDA Producer API
PNDA Consumer API
Bulk ingest
Custom
![Page 29: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/29.jpg)
29 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
Why PNDA? Innovation in the big data space is extremely rapid, but combining multiple technologies into an end-to-end solution can be extremely complex and time-consuming PNDA removes this complexity and allows you to focus on developing the analytics applications, not on developing the pipeline – significantly reducing the effort required and time-to-value
![Page 30: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/30.jpg)
30 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
PNDA Software Components
![Page 31: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/31.jpg)
31 © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved. © 2017 PNDA a Linux Foundation Collaborative Project. All rights reserved.
• www.pnda.io • https://github.com/pndaproject • www.snas.io
Where can I learn more?
![Page 32: When BGP meets Big-Data · What can we do with large-scale collection of historical event information? ... data analytics PNDA PNDA Plugins ODL Logstash SNAS pmacct IOS XR Telemetry](https://reader033.vdocuments.us/reader033/viewer/2022050611/5fb26b0985be0d3b5e7a09b9/html5/thumbnails/32.jpg)