![Page 1: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/1.jpg)
Fabian Fischer | Data Analysis and Visualization Group | University of Konstanz
NStreamAware: Real-Time Visual Analytics forData Streams to Enhance Situational AwarenessFabian Fischer and Daniel A. Keim
Symposium on Visualization for Cyber Security (VizSec 2014)10th November 2014, Paris, France
![Page 2: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/2.jpg)
2Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness
Motivation: Heterogeneous Data Streams
• Network Alerts (e.g., OSSEC)
• Syslog Messages
• NetFlow Data
Analyzing Data Streams =Crucial for security in your network!
Monitoring & Exploration
Crucial for situational awareness (SA)!
REAL-TIME
VISUAL
ANALYTICS
![Page 3: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/3.jpg)
DATA CHALLENGE
How to make streamanalysis scalable?
![Page 4: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/4.jpg)
4Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness
NStreamAware: Infrastructure
Web
Ap
plic
atio
n(N
Vis
Aw
are)
REST REST Service
(VACS-REST)
Dat
a St
ream
s(f
rom
vari
ou
sso
urc
es)
MongoDB ElasticSearch
Scalable
SPARK Service(VACS-Spark)
Distributed Streaming Analytics
Apache Spark™ is a fast and general engine for large-scale data processing which can run on a distributed computer cluster.
![Page 5: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/5.jpg)
5Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness
Integrated Perspectives
• Real-Time Data Stream Monitoring
• Real-Time Sliding Slices (NVisAware)
• Visual Feature Selection
• Summarized Sliding Slices
• Event Timeline & Insights
• Search & Exploration
![Page 6: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/6.jpg)
6Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness
Real-Time Data Stream Monitoring
![Page 7: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/7.jpg)
Demo
![Page 8: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/8.jpg)
![Page 9: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/9.jpg)
SITUATIONAL AWARENESS CHALLENGE
How to reduce thecognitive load?
![Page 10: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/10.jpg)
10Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness
NVisAware: Analytics
Web
Ap
plic
atio
n(N
Vis
Aw
are)
REST REST Service
(VACS-REST)
Dat
a St
ream
s(f
rom
vari
ou
sso
urc
es)
MongoDB ElasticSearch
SPARK Service(VACS-Spark)
Distributed Streaming Analytics
Visual Analytics Approach:Calculate and visualize sliding slices.
(based on sliding windows)
• Calculate Sliding Slice Summaryfor each sliding window.
• Push slicet to web application.
![Page 11: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/11.jpg)
11Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness
slicet
Real-Time Sliding Slice• Interactive Widgets
– Treemaps
– Counters
– Node-link diagrams
• Interactions– Star/Annotate slice
– Remove slice
– Retrieve data
• Color Encoding– Background for similarity
– Importance of alerts
![Page 12: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/12.jpg)
Demo
![Page 13: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/13.jpg)
![Page 14: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/14.jpg)
![Page 15: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/15.jpg)
![Page 16: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/16.jpg)
![Page 17: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/17.jpg)
EXPLORATION CHALLENGE
How to exploremany sliding slices?
![Page 18: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/18.jpg)
18Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness
Visual Feature Selection Visual Analytics Approach:Aggregate / Summarize
according interest function(visually steered by the expert)
![Page 19: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/19.jpg)
19Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness
Example: Using Visual Analytics for Interactive Summarization
… …
![Page 20: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/20.jpg)
Demo
![Page 21: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/21.jpg)
![Page 22: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/22.jpg)
22Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness
Application to Real-Time Social Media Analysis (VAST Challenge 2014 MC3)
• Real-Time Monitoring Task:Discover major events in the streamto support an ongoing police operation.
• Available Data Stream:Real-time feeds of microblogsand emergency calls.
• Successful participation:“Award for Outstanding ComprehensiveMini-Challenge 3 Submission”
![Page 23: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/23.jpg)
23Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness
Further Challenges and Future Work
• Challenge: Parameter adjustmentfor sliding slices and clustering.
• Automated merging of sliding slicesbased on the interest function.
• Performance Evaluation for a large networkusing security operational data stream.
• Responsiveness issueswhen increasing the number of complex interactive visualizations.
• Data retention and rotationfor the visualization interface.
![Page 24: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/24.jpg)
24Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness
Contributions
• NStreamAware – Building a web-based visualanalytics system using scalable technologies.
• NVisAware – Sliding Slices Visualizationwith embedded visualization widgets.
• NVisAware – Summarized Sliding Slicessteered using interactive visualizations.
EXPLORATION CHALLENGE
How to explore manysliding slices?
SA CHALLENGE
How to reduce thecognitive load?
DATA CHALLENGE
How to make stream analysis scalable?
![Page 25: NStreamAware: Real-Time Visual Analytics for Data Streams ... · Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness 4 NStreamAware:](https://reader034.vdocuments.us/reader034/viewer/2022050323/5f7cafeca4ab1458442063b9/html5/thumbnails/25.jpg)
25Fabian Fischer | NStreamAware: Real-Time Visual Analytics for Data Streams to Enhance Situational Awareness
Thank you very much for your attention!
Questions?
For more informationabout this work please contact
Fabian FischerTel. +49 7531 88-2780
http://ff.cx/
@f2cx