tracing your security telemetry with apache metron

Tracing Your Security Telemetry With Apache Metron

Justin LeetSystems ArchitectJune 29, 2016

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

What is Apache Metron?


What Apache Metron Does?

“Apache Metron provides a scalable advanced security analytics framework built with the Hadoop Community evolving from the Cisco OpenSOC Project.

A cyber security application framework that provides organizations the ability to detect cyber anomalies and enable organizations to rapidly respond to identified anomalies.”


Apache Metron TimelineSep 2014 •OpenSOC Beta

June 2015 •OpenSOC Community Edition

Dec 2015 •Metron enters Apache Incubator

April 2016 •Apache Metron 0.1

Now •Working towards 0.2 release


Who is Metron for?


Core Capabilities


Architecture


Streaming Parsing and Enrichment


Metron’s parsing bolt can be configured two ways– And outputs JSON

Grok Parser– Less work to implement– Regex-like syntax– Good for lower volumes of data

Java Parser– More work to implement– Good for higher volumes of data

Parsing


Enrichment / Threat Intel


Add additional information to raw source during streaming Adding it during streaming allows ML models to score in real time instead of

batch Primarily stored in HBase Several enrichments

– GeoIP– Host– Threat Intelligence

Enrichment


Occurs in the same Storm topology as enrichment

Very similar process and flow

Use a threat feed aggregator!– Soltra adapter is provided to read feed and stream into HBase– Flat File loader and Stix bulk loader available without threat feed aggregator

Threat Intel


Field Description

ip_src_addr Octet source IPip_dest_addr Octet destination IPip_src_port Integer source portip_dest_port Integer destination portprotocol String protocol (e.g. TCP)timestamp Sensor epoch timestampsource.type yaf, snort, etc.start_time Metron epoch timestampend_time Metron epoch timestamp

Metron JSON


Standalone Storm topology

Reads from Kafka

Writes packets to HDFS

Kibana panel forwards request to REST PCAP service– MR Job launched– Delivers results back to Kibana

PCAP


PCAP


Tracing a Source Through Metron


Sensor to Parser


Caching proxy– Mostly useful as a source of easy to get and easily readable logs

Squid

1467125585.752 5288 127.0.0.1 TCP_MISS/200 32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html

Time Elapsed Remote Host Code/Status Bytes Method URL rfc931 Peer Status/ Peer Host Type

1467125585.752 5288 127.0.0.1 TCP_MISS/200

32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html


Squid - Grok

Time Elapsed Remote Host Code/Status Bytes Method URL rfc931 Peer Status/ Peer Host Type

1467125585.752 5288 127.0.0.1 TCP_MISS/200

32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html

SQUID_DELIMITED %{NUMBER:timestamp}%{SPACE:UNWANTED} %{INT:elapsed}%{SPACE:UNWANTED}%{IPV4:ip_src_addr} %{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url} - %{WORD:UNWANTED}\/%{IPV4:ip_dst_addr} %{WORD:UNWANTED}\/%{WORD:UNWANTED}


Squid – Topology Definition

{ "parserClassName": "org.apache.metron.parsers.GrokParser", "sensorTopic": "squid", "parserConfig": { "grokPath": "/apps/metron/patterns/squid", "patternLabel": "SQUID_DELIMITED", "timestampField": "timestamp" }, "fieldTransformations" : [ { "transformation" : "MTL" ,"output" : [ "full_hostname", "domain_without_subdomains" ] ,"config" : { "full_hostname" : "URL_TO_HOST(url)" ,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)" } } ]}


Squid – Topology Result


Enrichment Topology


Loading some WHOIS derived data.– Not directly making WHOIS query, just using a CSV containing a few rows of data.

Squid – Enrichment Definition

{ "zkQuorum" : ”localhost:2181" ,"sensorToFieldList" : { "squid" : { "type" : "ENRICHMENT" ,"fieldToEnrichmentTypes" : { "domain_without_subdomains" : [ "whois" ] } } }}


Squid – Enrichment Result


Enrichment Topology


Loading a list of malicious domains– ZeuS tracker

Squid – Enrichment Definition

{ "zkQuorum": "localhost:2181", "sensorToFieldList": { "squid": { "type": "THREAT_INTEL", "fieldToEnrichmentTypes": { "url": ["zeusList”] } } }}


Squid – Threat Intel Result


Questions?Justin LeetSystems Architect

[email protected]@gmail.com

tracing your security telemetry with apache metron

Technology