pnda the open platform for network data analytics...1hwzrun gdwd lv ehfrplqj d elj gdwd sureohp«...
TRANSCRIPT
PNDAThe open platform for network data analytics
Network data is becoming abig data problem …
3-fold increase in total IP
Traffic
3-fold increase in total IP
Traffic
>60% increase in devices and connections
>60% increase in devices and connections
Telemetry data streamed in
near real-time
Telemetry data streamed in
near real-time
Computer science is driving bifurcation of the OSS stack…
OSSOSS
BSSBSS
Network and Services
Network and Services
CustomerCustomerOrderOrder
Order MgmtOrder Mgmt
Provisioning & ActivationProvisioning & Activation
ServiceService DataData
Monitoring and analysisMonitoring
and analysis
Billing and ReportingBilling and Reporting
Bills and ReportsBills and Reports
Orchestration is responsible for service provisioning and pushes state to the infrastructure
The “C” in FCAPS
Analytics is responsible for collecting data from the infrastructure, monitoring and analysisThe “F_APS” in FCAPS
AnalyticsAnalyticsOrchestrationOrchestration
• Tight coupling of data aggregation/store/analysis
• Multiple analytics pipelines implemented from open source components
• Common design patterns ~75% of effort wasted / duplicated
• Siloes limit the potential of big data analytics and lead to industry divergence
Today’s siloed analytics pipelines
TelemetryTelemetry
MetricsMetrics
Data sources
HDFS
Data store
Spark Streaming
Spark Streaming
MapRMapR
Data analysis
Hbase
StormStorm
KafkaKafka
StreamsetsStreamsets
Data aggregation
KafkaKafka
ImpalaImpala
QueryQuery
Outputs
Dashboard & Reporting
Dashboard & ReportingNiFiNiFi
LogsLogs
What is PNDA?PNDA brings together a number of open source technologies to provide a simple, scalable open big data analytics Platform for Network Data Analytics
Linux Foundation Collaborative Project based on the Apache ecosystem
• Scalable data platform
• Provides a common set of services for developing analytics applications
• 100% open source
• Open APIs
• Application packaging and management
• Multi-vendor, multi-data
• Infra and distribution agnostic
PNDA
PNDAPlugins
ODLODL
LogstashLogstash
OpenBPMOpenBPM
pmacctpmacct
XR TelemetryXR Telemetry
Real-timeReal-time
Data D
istributio
nD
ata Distribu
tion
FileStoreFile
Store
Platform Services: Installation, Mgmt, Security, Data Privacy
Platform Services: Installation, Mgmt, Security, Data Privacy
App Packaging and Mgmt
App Packaging and Mgmt
StreamStream
BatchBatch
Processing
SQL QuerySQL
Query
OLAP CubeOLAP Cube
Search/LuceneSearch/Lucene
NoSQLNoSQL TimeSeriesTime
Series
DataExploration
DataExploration
Metric Visualisation
Metric Visualisation
Event Visualisation
Event Visualisation PNDA
Mnged AppPNDA
Mnged App
PNDA Mnged App
PNDA Mnged App
UnmngedApp
UnmngedApp
UnmngedApp
UnmngedApp
Query Visualisationand Exploration
PNDA Applications
PNDAProducer API
PNDAConsumer API
• PNDA decouples data aggregation from data analysis
• Horizontally scalable platform for analytics and data processing applications
• Support for near-real-time stream processing and in-depth batch analysis on massive datasets
• Consuming applications can be either platform apps developed for PNDA or client apps integrated with PNDA
• Client apps can use one of several structured query interfaces or consume streams directly.
• Leverages best practise in big data analytics
PNDA
PNDAPlugins
ODLODL
LogstashLogstash
OpenBPMOpenBPM
pmacctpmacct
XR TelemetryXR Telemetry
Real-timeReal-time
Data D
istributio
nD
ata Distribu
tion
FileStoreFile
Store
Platform Services: Installation, Mgmt, Security, Data Privacy
Platform Services: Installation, Mgmt, Security, Data Privacy
App Packaging and Mgmt
App Packaging and Mgmt
StreamStream
BatchBatch
Processing
SQL QuerySQL
Query
OLAP CubeOLAP Cube
Search/LuceneSearch/Lucene
NoSQLNoSQL TimeSeriesTime
Series
DataExploration
DataExploration
Metric Visualisation
Metric Visualisation
Event Visualisation
Event Visualisation PNDA
Mnged AppPNDA
Mnged App
PNDA Mnged App
PNDA Mnged App
UnmngedApp
UnmngedApp
UnmngedApp
UnmngedApp
Query Visualisationand Exploration
PNDA Applications
PNDAProducer API
PNDAConsumer API
There are a bewildering number of big data technologies out there, so how do you decide what to use?
Combining multiple technologies into an end-to-end solution can be extremely complex and time-consuming
PNDA solves these problems –streamlining the process and minimsing the TCO of developing data processing applications
Why PNDA?
PNDA Software Components
• The PNDA console provides a dashboard across all components in a cluster
• Inbuilt platform test agents verify the operation of all components
• Active platform testing verifies the end-to-end data pipeline
PNDA Console
• Platform for data aggregation, distribution, processing and storage
• Automated installation, creation, and configuration• Openstack, AWS and baremetal
• Ubunto and RHEL
• Typical install ~1hr
• Online and offline install; modular install
• Open producer and consumer APIs• Avro platform schema
• Plugins for Logstash, pmacct, OpenBMP, OpenDaylight, Cisco XR-telemetry, bulk ingest …
• Data distribution – Apache Kafka
• Data store:• Automated data partitioning and storage
(HDFS)
• OpenTSDB – time series analysis
• Hbase - NoSQL
• Support for batch and stream processing:• Apache Spark and Spark Streaming
• Jupyter notebook server for app prototyping and data exploration
• Impala-based SQL query support
• Grafana for time series visualisation
• PNDA application packaging
• PNDA management and dashboard
PNDA 3.5 Capabilities
• Ingested data should be encapsulated in PNDA Avro schema and published on a pre-defined Kafka topic or set of topics
Publishing Data to PNDA
PNDA PluginsData Type Data Aggregator Data Aggregator Reference PNDA Producer Plugin Reference
BGP (inc. BGP LS) OpenBMP http://www.openbmp.org/#!index.md#Using_Kafka_for_Collector_Integration
http://pnda.io/pnda-guide/producer/openbmp.html
BGP PMACCT (BGP listener) http://www.pmacct.net/ http://pnda.io/pnda-guide/producer/pmacct.html
Bulk Ingest PNDA Bulk Ingest Tool http://pnda.io/pnda-guide/bulkingest/
ISIS PMACCT (ISIS listener) http://www.pmacct.net/ http://pnda.io/pnda-guide/producer/pmacct.html
Cisco XR streaming telemetry Pipeline https://github.com/cisco/bigmuddy-network-telemetry-collector
CollectD (CollectD supports multiple plugins as listed here https://collectd.org/wiki/index.php/Table_of_Plugins)
Logstash https://www.elastic.co/guide/en/logstash/current/plugins-codecs-collectd.html
http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/
IoT sensor via HTTP Node-RED https://nodered.org
Logstash (Logstash supports multiple plugins as listed here https://www.elastic.co/guide/en/logstash/current/input-plugins.html)
Logstash http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/
NETCONF Notifications ODL http://www.opendaylight.org/ http://pnda.io/pnda-guide/producer/opendl.html
Netflow / IPFIX Logstash https://www.elastic.co/guide/en/logstash/current/plugins-codecs-netflow.html
http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/
Netflow / IPFIX / sFlow pmacct http://www.pmacct.net/ http://pnda.io/pnda-guide/producer/pmacct.html
Openstack Work in progress
sFlow Logstash https://github.com/ashangit/logstash-codec-sflow
http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/
SNMP Metrics and Traps ODL https://wiki.opendaylight.org/view/SNMP_Plugin:Getting_Started
http://pnda.io/pnda-guide/producer/opendl.html
SNMP Traps Logstash https://www.elastic.co/guide/en/logstash/current/plugins-inputs-snmptrap.html
http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/
Syslog Logstash https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html
http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/
Syslog (RFC3164 or RFC5424 - needed for newer IOS/IOS XR/ NX OS etc.)
Logstash https://gist.github.com/donaldh/89b7304981f96497c94fe4d98bb03d71
http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/
• Smaller, simpler subset of PNDA designed for development, demonstration and education
• Can run it on your laptop
• Consistent technologies, including:• PNDA data-ingest (Kakfa/AVRO)
• Data-exploration tools: Jupyter, OpenTSDB and Grafana
• Apache Spark and Hbase
• Doesn’t include HDFS and other Hadoop infrastructure for distributed processing.
Red PNDA
PNDA Flavors span the Data Science Lifecycle
pico
standard Red
PNDA Applied to NFVIn
fras
truc
tur
eIn
fras
truc
tur
e
OSS Analytics
OSS Analytics
Data AggregatorsData Aggregators
Open Data Platform (PNDA)Open Data Platform (PNDA)
OSS Analytics ApplicationsOSS Analytics Applications
OSSFunctions
OSSFunctions
BSSFunctions
BSSFunctions
Ops and BusIntelligenceOps and BusIntelligence
AlertsAlerts
Metrics
Metrics
Telemetry
Telemetry
LogsLogs
Data Sources
InventoryInventory
Orc
hest
ratio
nO
rche
stra
tion
NFVONFVO
VNFMVNFM
VIMVIM
NFVINFVI
VNFVNF
Data CenterCoreUser
StateState D
ata
Dat
a
Access Aggregation
Loosely coupledsystems
ContextContext
NetworkControlNetworkControl
Orchestration service models
provide necessary context for analytics
Output from analytics can close the control loop to realize reactive networking
Enabling LSO to realise the vision of reactive networking MEF and PNDA Collaboration
• Analytics Based Service assurance
• ML-based Security Analytics with Apache SPOT on PNDA
• Path Anomaly detection in PNDA using in-band OAM
• Openstack Analytics with PNDA and Calipso
• Smart Transport – Connected Car Cloud Analytics with Machine Learning using PNDA
• BGP analytics with SNAS.io and PNDA.io
• ETSI NFV and Big Data Analytics with PNDA
• PNDA and Paris IOT Smart Cities Pilot
• Cable Plant Anomaly Detection with PNDA
Increasing List of PNDA Use Cases
Convergence of network data analytics
OperationalIntelligence
PlanningIntelligence
SecurityIntelligence
• Bare metal install
• HDP / Apache Ambari
• Apache Flink
• OPNFV
• ONAP (Release 2)
What’s coming?