hortonworks cybersecurity platform deep divedataplatform.jp/program/files/a-7.pdf · hortonworks...

Hortonworks Cybersecurity PlatformDeep DivePowered by Apache Metron

Dave Russell

Global SME Cybersecurity

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

0 1 2 3 4 5 6 7 8 9

BREACH DURATION VS DATA RETENTION

Average silo retention Average breach unnoticed

3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

0 5 10 15 20 25 30 35 40

Average breach unnoticed

Average silo retention

Yahoo (US)

PoliceOne

BREACH DURATION VS DATA RETENTION

Duration in months

4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

“Sometime in the next few years we're going to have our first

category-one cyber-incident; one that will need a national response.”

Ian LevyTechnical Director

National Cyber Security Centre

5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Andhra Pradesh Police, IndiaAristotle University of Thessaloniki, GreeceAutomobile Dacia, RomaniaCambrian College, CanadaChinese public security bureauCJ CGVDalian Maritime UniversityDeutsche BahnDharmais Hospital, IndonesiaFaculty Hospital, Nitra, SlovakiaFedExGarena Blade and SoulGuilin University Of Aerospace TechnologyGuilin University Of Electronic TechnologyHarapan Kita Hospital[disambiguation needed], IndonesiaHezhou University

SandvikSão Paulo Court of JusticeSaudi Telecom CompanySberbankShandong UniversityState Governments of India Government of GujaratGovernment of KeralaGovernment of MaharashtraGovernment of West BengalSuzhou Vehicle AdministrationSun Yat-sen University, ChinaTelefónicaTelenor Hungary, HungaryTelkom (South Africa)Timrå Municipality, SwedenUniversitas Jember, IndonesiaUniversity of Milano-Bicocca, ItalyUniversity of Montreal, CanadaVivo, Brazil

HitachiHondaInstituto Nacional de Salud, ColombiaLakeridge HealthLAKSLATAM Airlines GroupMegaFonMinistry of Internal Affairs of the Russian FederationMinistry of Foreign Affairs (Romania)National Health Service (England)NHS ScotlandNissan Motor Manufacturing UKO2, GermanyPetrobrásPetroChinaPortugal TelecomPulse FMQ-ParkRenaultRussian Railways

6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

https://pixabay.com/p-906036/

7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

http://www.thebluediamondgallery.com/wooden-tile/images/failure.jpg

8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Problem Posed For Security Analysts

Too many disparate tools

Too many alerts to process

Too much noise

Too slow

Unnecessary plumbing work

9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Timeline

Sept 2014

June2015

Dec 2015

April 2016

Feb 2017

Jun 2017

Sept2017

OpenSOC

Beta

OpenSOC

Community Edition

Metron enters Apache Incubator

Apache Metron 0.1 Apache Metron 0.3

HCP 1.1

Apache Metron 0.4

HCP 1.2

Secure Cluster

Apache Metron0.4.1

HCP 1.3

Alerts UI

April2017

Apache Metronexits Apache

Incubator

10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Hortonworks Cybersecurity Platform – Powered By Apache Metron

Real-Time ingestion of application and system logs

Real-Time cyber security dashboard and cyber workbench

Real-Time ingestion, correlation and enrichment of PCAP and NetFlow telemetries

Real-Time integration of Cyber security feeds

Advanced statistical and machine learning models to detect cyber security attacks

Integration with existing SIEMs and enterprise assets

ApacheMetron

Cyber Security Data Ingestion

Package

Cyber Security Analytics

11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Data Se

rvices an

d In

tegratio

n Laye

r

ModulesReal-time ProcessingCyber Security Engine

TelemetryParsers

Apache Metron: Overview

Tele

metry In

gest B

uffe

r

TelemetryData Collectors

Real-timeEnrich / ThreatIntel Streams

PerformanceNetwork

IngestProbes

/ OtherMachine Generated Logs(AD, App / Web Server,

firewall, VPN, etc.)

Security Endpoint Devices (Fireye, Palo Alto,

BlueCoat, etc.)

Network Data(PCAP, Netflow, Bro, etc.)

IDS(Suricata, Snort, etc.)

Threat Intelligence Feeds(Soltra, OpenTaxi,third-party feeds)

TelemetryData Sources

Data Vault

Real-Time Search

Evidentiary Store

Threat Intelligence Platform

Model as a Service

Community Models

Data Science Workbench

PCAP Forensics

Threat IntelligenceEnrichment

Indexers and WriterProfiler Alert Triage

Cyber SecurityStream Processing Pipeline

12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Profiler: User and Entity Behavior analytics

HBaseProfiler Bolt

• HyperLogLogPlus

• T-Digest

• Bloom filter

• MAD outlier

Cardinality

Statistics

Presence

Outliers

How many servers connected?

Average over different periods

Finding small needles in big haystacks

Detecting unusual events in streams

Triage Scoring Model features Aggregations over Time

Fast Cache

13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

YARN

Model as a Service

Historical Data Store

Model ServiceREST interface

Model Store

ZookeeperStorm Enrichment Bolt Service Discovery

HDFS

Trai

n /

Up

dat

e

HBase

Metron JSON Object

Metron JSON Object with added score, confidence

etc. from model

• Real-time scoring• Versioned deployment of

models in any tech• Service discovery

14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

A deeper look at Hortonworks Cybersecurity Platform…

15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

“stop blaming the users, and make the systems usable.”

Ian LevyTechnical Director

National Cyber Security Centre

16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Questions?

17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

FAQ

Combined or separate cybersecurity datalake?

Cloud or on premise?

Streaming engine, why not spark/flink/any other hotness?

What ML in Model as a Service?

Where can I get more information?– https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.3.0/index.html

– http://metron.apache.org/

– https://cwiki.apache.org/confluence/display/METRON/Community+Resources

– https://github.com/apache/metron

Hot? Warm? Cold?

What is supported out of the box?

18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Metron Datasheet Ingest:

– Apache NiFi: syslog, socket, file, web services, SQL, RDBMS, Windows Event Log, FTP, MQ, JMS

– High-performance DPDK Packet Capture

Parsers:

– Cisco ASA

– Bluecoat

– Fireeye

– Palo Alto

– SourceFire

– WebSphere

– Snort

– Bro

– YAF (Netflow, IPFIX)

– Grok (Custom)

– Java (Custom)

– JSON

– Applications: DHCPD, AD

Enrichments and threat feeds:

– Geo

– Whois

– HBase

– JDBC

– Stellar

– CSV

– Stix, Taxii threat intel feeds

Analytics features:

– Profiler

– Model Services

– Threat Triage

Indexing and search:

– Elasticsearch, Kibana

– Solr

– HDFS

– Kafka

Data science features:

– Spark Machine Learning

– Zeppelin notebooks and reporting

– Wide partner eco-system

Forensic features:

– PCAP inspector

– PCAP query

– Long term data store

19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Thankyou