tracing your security telemetry with apache metron

Tracing Your Security Telemetry With Apache Metron

Justin LeetSystems ArchitectJune 29, 2016

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

What is Apache Metron?

3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

What Apache Metron Does?

“Apache Metron provides a scalable advanced security analytics framework built with the Hadoop Community evolving from the Cisco OpenSOC Project.

A cyber security application framework that provides organizations the ability to detect cyber anomalies and enable organizations to rapidly respond to identified anomalies.”

4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Metron TimelineSep 2014 •OpenSOC Beta

June 2015 •OpenSOC Community Edition

Dec 2015 •Metron enters Apache Incubator

April 2016 •Apache Metron 0.1

Now •Working towards 0.2 release

5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Who is Metron for?

6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Core Capabilities

7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Architecture

8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Streaming Parsing and Enrichment

9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Metron’s parsing bolt can be configured two ways– And outputs JSON

Grok Parser– Less work to implement– Regex-like syntax– Good for lower volumes of data

Java Parser– More work to implement– Good for higher volumes of data

Parsing

10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Enrichment / Threat Intel

11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Add additional information to raw source during streaming Adding it during streaming allows ML models to score in real time instead of

batch Primarily stored in HBase Several enrichments

– GeoIP– Host– Threat Intelligence

Enrichment

12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Occurs in the same Storm topology as enrichment

Very similar process and flow

Use a threat feed aggregator!– Soltra adapter is provided to read feed and stream into HBase– Flat File loader and Stix bulk loader available without threat feed aggregator

Threat Intel

13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Field Description

ip_src_addr Octet source IPip_dest_addr Octet destination IPip_src_port Integer source portip_dest_port Integer destination portprotocol String protocol (e.g. TCP)timestamp Sensor epoch timestampsource.type yaf, snort, etc.start_time Metron epoch timestampend_time Metron epoch timestamp

Metron JSON

14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Standalone Storm topology

Reads from Kafka

Writes packets to HDFS

Kibana panel forwards request to REST PCAP service– MR Job launched– Delivers results back to Kibana

PCAP

15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

PCAP

16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Tracing a Source Through Metron

17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Sensor to Parser

18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Caching proxy– Mostly useful as a source of easy to get and easily readable logs

Squid

1467125585.752 5288 127.0.0.1 TCP_MISS/200 32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html

Time Elapsed Remote Host Code/Status Bytes Method URL rfc931 Peer Status/ Peer Host Type

1467125585.752 5288 127.0.0.1 TCP_MISS/200

32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html

19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Squid - Grok

Time Elapsed Remote Host Code/Status Bytes Method URL rfc931 Peer Status/ Peer Host Type

1467125585.752 5288 127.0.0.1 TCP_MISS/200

32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html

SQUID_DELIMITED %{NUMBER:timestamp}%{SPACE:UNWANTED} %{INT:elapsed}%{SPACE:UNWANTED}%{IPV4:ip_src_addr} %{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url} - %{WORD:UNWANTED}\/%{IPV4:ip_dst_addr} %{WORD:UNWANTED}\/%{WORD:UNWANTED}

20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Squid – Topology Definition

{  "parserClassName": "org.apache.metron.parsers.GrokParser",  "sensorTopic": "squid",  "parserConfig": {    "grokPath": "/apps/metron/patterns/squid",    "patternLabel": "SQUID_DELIMITED",    "timestampField": "timestamp"  },  "fieldTransformations" : [     {      "transformation" : "MTL"    ,"output" : [ "full_hostname", "domain_without_subdomains" ]    ,"config" : {                    "full_hostname" : "URL_TO_HOST(url)"                    ,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"                    }     } ]}

21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Squid – Topology Result

22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Enrichment Topology

23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Loading some WHOIS derived data.– Not directly making WHOIS query, just using a CSV containing a few rows of data.

Squid – Enrichment Definition

{ "zkQuorum" : ”localhost:2181" ,"sensorToFieldList" : { "squid" : { "type" : "ENRICHMENT" ,"fieldToEnrichmentTypes" : { "domain_without_subdomains" : [ "whois" ] } } }}

24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Squid – Enrichment Result

25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Enrichment Topology

26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Loading a list of malicious domains– ZeuS tracker

Squid – Enrichment Definition

{ "zkQuorum": "localhost:2181", "sensorToFieldList": { "squid": { "type": "THREAT_INTEL", "fieldToEnrichmentTypes": { "url": ["zeusList”] } } }}

27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Squid – Threat Intel Result

28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Questions?Justin LeetSystems Architect

jleet@hortonworks.comjustinjleet@gmail.com