a streaming architecture for cyber security - apache metron

A streaming architecture for Cyber Security

with NiFi, Hadoop, Storm and Metron

Simon Elliston Ball

• Product Manager

• Data Scientist

• Elephant herder

• @sireb

IoT: Mirai

Reports of 1.2 Tbps

500,000 devices at peak

DDoS attacks on Dyn DNS services

Drowning in Data

The value of real time

Data in Motion: why wait until it’s at rest?

Correct context: the world moved on

Better data = analyst efficiency

Fully enriched data

Real context

Consistency

= faster triage and better coverage

Network Level Taps

Data Sources and Aggregation

Open standards for data models = more productive data scientists + shareable models

Business level data sources link security to real business risk.

9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Data Se

rvices an

d In

tegratio

n Laye

r

ModulesReal-time ProcessingCyber Security Engine

TelemetryParsers

Apache Metron: a framework for Big Data Driven cyber security

Tele

metry In

gest B

uffe

r

TelemetryData Collectors

Real-timeEnrich / ThreatIntel Streams

PerformanceNetwork

IngestProbes

/ OtherMachine Generated Logs(AD, App / Web Server,

firewall, VPN, etc.)

Security Endpoint Devices (Fireye, Palo Alto,

BlueCoat, etc.)

Network Data(PCAP, Netflow, Bro, etc.)

IDS(Suricata, Snort, etc.)

Threat Intelligence Feeds(Soltra, OpenTaxi,third-party feeds)

TelemetryData Sources

Data Vault

Real-Time Search

Evidentiary Store

Threat Intelligence Platform

Model as a Service

Community Models

Data Science Workbench

PCAP Forensics

Threat IntelligenceEnrichment

Indexers and WriterProfiler Alert Triage

Cyber SecurityStream Processing Pipeline


Enrichment is the key to context

Human Resources Database

Metron Data

Metron Data

App

Logs

Active Directory

Network Traffic

Logs

IoT

Asset Database

Geo, Threat, Traditional Security

data sources

Business Risk DataMetron

Data

Standard, Consistent Data Format

Streaming enrichment

Batch enrichment

Fully Enriched data ready for analysis

Wide variety of real-time and batch

sources


But time is context too… profiling by time

t = 1 t = 2 t = 3 t = n

Wide range of algorithms including:

HyperLogLogPlus

Bloom filters

T-digests

Statistical Baselining

Hashing functions

Outlier detection

GeoHashing over time

Locality Sensitive HashingApprox.

Data SketchApprox.

Data SketchApprox.

Data SketchApprox.

Data Sketch

Combined Baseline

Statistic


Stellar: Excel functions for Cyber security

{"profile": "auth_distribution","foreach": "'global'","onlyif": "profile == 'attempts_by_user'","init": {

"s": "STATS_INIT()"},"update": {

"s": "STATS_ADD(s, total_count)"},"result": "s"

}

Building a Profile Using a Profile

window := PROFILE_WINDOW('...')profile := PROFILE_GET('attempts_by_user', user, window)distinct_auth_attempts := HLLP_CARDINALITY(GET_LAST(profile))distribution_profile := PROFILE_GET('auth_distribution', 'global', window)stats := STATS_MERGE(distribution_profile)distinct_auth_attempts_median := STATS_PERCENTILE(stats, 0.5)distinct_auth_attempts_stddev := STATS_SD(stats)

• Simple• Expression based• Function composition• Boolean operators• In-stream

Thank you!

Apache Metron: http://metron.apache.org

Twitter: @sireb

a streaming architecture for cyber security - apache metron

Technology