![Page 1: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/1.jpg)
1 © Hortonworks Inc. 2011–2018. All rights reserved
Apache Metron in the Real WorldDave Russell - Hortonworks
www.roaringelephant.org
![Page 2: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/2.jpg)
2 © Hortonworks Inc. 2011–2018. All rights reserved
Who am I?
![Page 3: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/3.jpg)
3 © Hortonworks Inc. 2011–2018. All rights reserved
![Page 4: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/4.jpg)
4 © Hortonworks Inc. 2011–2018. All rights reserved
Why Apache Metron?
![Page 5: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/5.jpg)
5 © Hortonworks Inc. 2011–2018. All rights reserved
Months until breach noticed
Avg. months log retention
9 6
VS
3Months missing
![Page 6: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/6.jpg)
6 © Hortonworks Inc. 2011–2018. All rights reserved
28 Months
Police One/Berkut
Yahoo/? FB/Cambridge Analytica
35 Months 48 Months
Time until breach actually noticed
![Page 7: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/7.jpg)
7 © Hortonworks Inc. 2011–2018. All rights reserved
“Sometime in the next few years we're going to have our first
category-one cyber-incident; one that will need a national response.”
Ian LevyTechnical Director
National Cyber Security Centre
![Page 8: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/8.jpg)
8 © Hortonworks Inc. 2011–2018. All rights reserved
Andhra Pradesh Police, IndiaAristotle University of Thessaloniki, GreeceAutomobile Dacia, RomaniaCambrian College, CanadaChinese public security bureauCJ CGVDalian Maritime UniversityDeutsche BahnDharmais Hospital, IndonesiaFaculty Hospital, Nitra, SlovakiaFedExGarena Blade and SoulGuilin University Of Aerospace TechnologyGuilin University Of Electronic TechnologyHarapan Kita Hospital[disambiguation needed], IndonesiaHezhou University
SandvikSão Paulo Court of JusticeSaudi Telecom CompanySberbankShandong UniversityState Governments of India Government of GujaratGovernment of KeralaGovernment of MaharashtraGovernment of West BengalSuzhou Vehicle AdministrationSun Yat-sen University, ChinaTelefónicaTelenor Hungary, HungaryTelkom (South Africa)Timrå Municipality, SwedenUniversitas Jember, IndonesiaUniversity of Milano-Bicocca, ItalyUniversity of Montreal, CanadaVivo, Brazil
HitachiHondaInstituto Nacional de Salud, ColombiaLakeridge HealthLAKSLATAM Airlines GroupMegaFonMinistry of Internal Affairs of the Russian FederationMinistry of Foreign Affairs (Romania)National Health Service (England)NHS ScotlandNissan Motor Manufacturing UKO2, GermanyPetrobrásPetroChinaPortugal TelecomPulse FMQ-ParkRenaultRussian Railways
![Page 9: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/9.jpg)
9 © Hortonworks Inc. 2011–2018. All rights reserved
2018 so far...
340M Records
150M Records
92M RecordsAnd many, many, many more..https://en.wikipedia.org/wiki/List_of_data_breaches
![Page 10: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/10.jpg)
10 © Hortonworks Inc. 2011–2018. All rights reserved
What Does Apache Metron Look Like?
![Page 11: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/11.jpg)
11 © Hortonworks Inc. 2011–2018. All rights reserved
Security telemetry source: authentication logsSecurity telemetry source: authentication logs
![Page 12: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/12.jpg)
12 © Hortonworks Inc. 2011–2018. All rights reserved
![Page 13: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/13.jpg)
13 © Hortonworks Inc. 2011–2018. All rights reserved
![Page 14: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/14.jpg)
14 © Hortonworks Inc. 2011–2018. All rights reserved
![Page 15: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/15.jpg)
15 © Hortonworks Inc. 2011–2018. All rights reserved
What is Apache Metron?
![Page 16: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/16.jpg)
16 © Hortonworks Inc. 2011–2018. All rights reserved
Built on top on proven open source big data technology
![Page 17: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/17.jpg)
17 © Hortonworks Inc. 2011–2018. All rights reserved
An architecture for real-time cybersecurity analytics
![Page 18: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/18.jpg)
18 © Hortonworks Inc. 2011–2018. All rights reserved
Telemetry Data Source
![Page 19: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/19.jpg)
19 © Hortonworks Inc. 2011–2018. All rights reserved
Telemetry Data Collectors
![Page 20: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/20.jpg)
20 © Hortonworks Inc. 2011–2018. All rights reserved
Cyber Security Stream Processing Pipeline
![Page 21: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/21.jpg)
21 © Hortonworks Inc. 2011–2018. All rights reserved
Profiling by time
t = 1 t = 2 t = 3 t = n
Wide range of algorithms including:à HyperLogLogPlusà Bloom filtersà T-digestsà Statistical Baseliningà Hashing functionsà Outlier detectionà GeoHashing over timeà Locality Sensitive HashingApprox. Data
SketchApprox. Data
SketchApprox. Data
SketchApprox. Data
Sketch
Combined Baseline
Statistic
![Page 22: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/22.jpg)
22 © Hortonworks Inc. 2011–2018. All rights reserved
Cyber Security Stream Processing Pipeline
![Page 23: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/23.jpg)
23 © Hortonworks Inc. 2011–2018. All rights reserved
Apache Metron Modules
![Page 24: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/24.jpg)
24 © Hortonworks Inc. 2011–2018. All rights reserved
Who is Using Apache Metron (Part 1)
![Page 25: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/25.jpg)
25 © Hortonworks Inc. 2011–2018. All rights reserved
![Page 26: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/26.jpg)
26 © Hortonworks Inc. 2011–2018. All rights reserved
![Page 27: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/27.jpg)
27 © Hortonworks Inc. 2011–2018. All rights reserved
![Page 28: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/28.jpg)
28 © Hortonworks Inc. 2011–2018. All rights reserved
The Wider Apache Metron Ecosystem
![Page 29: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/29.jpg)
29 © Hortonworks Inc. 2011–2018. All rights reserved
![Page 30: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/30.jpg)
30 © Hortonworks Inc. 2011–2018. All rights reserved
Demo / Video
![Page 31: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/31.jpg)
31 © Hortonworks Inc. 2011–2018. All rights reserved
Who is Using Apache Metron (Part 2)
![Page 32: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/32.jpg)
32 © Hortonworks Inc. 2011–2018. All rights reserved
![Page 33: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/33.jpg)
33 © Hortonworks Inc. 2011–2018. All rights reserved
![Page 34: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/34.jpg)
34 © Hortonworks Inc. 2011–2018. All rights reserved
![Page 35: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/35.jpg)
35 © Hortonworks Inc. 2011–2018. All rights reserved
![Page 36: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/36.jpg)
36 © Hortonworks Inc. 2011–2018. All rights reserved
Deploying Apache Metron
![Page 37: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/37.jpg)
37 © Hortonworks Inc. 2011–2018. All rights reserved
AD/AssetDB/HR/Threat
HDF
HDFS
NiFi - Ingest
HDP
Phase 0 – Current State
ADP Event Broker (Kafka)
ADP Smart Connectors
ADP Logger
ArcSight ESM
Security Assets
3
1
2
4
5
![Page 38: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/38.jpg)
38 © Hortonworks Inc. 2011–2018. All rights reserved
HDF
Zeppelin
HDFS
NiFi - Ingest
Kafka MQ
Storm Parse / Enrich / GeoIP / Index
SolrInvestigator UI
HDP
Phase 1 - Ingest and Archive
ADP Event Broker (Kafka)
ADP Smart Connectors
ADP Logger
ArcSight ESM
Security Assets AD/AssetDB/HR/Threat
Spark
Historical Analysis
10
6
87
9
11
12
13
Banana
![Page 39: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/39.jpg)
39 © Hortonworks Inc. 2011–2018. All rights reserved
HDF
Zeppelin
HDFS
NiFi - Ingest
Kafka MQ
Storm Parse / Enrich / GeoIP / Index
Solr
Enrichment Data
Investigator UI
HDP
Phase 2 – Enrich and Threat Intel
ADP Event Broker (Kafka)
ADP Smart Connectors
ADP Logger
ArcSight ESM
Security Assets AD/AssetDB/HR/Threat
Spark
Historical Analysis
14
Banana / Kibana / ZoomData
![Page 40: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/40.jpg)
40 © Hortonworks Inc. 2011–2018. All rights reserved
HDF
Zeppelin
HDFS
NiFi - Ingest
Kafka MQ
Storm Parse / Enrich / GeoIP / Index
Solr
Enrichment Data
Metron Profiler
Triage
Alert
Investigator UI
HDP
Phase 3 – NiFi Data Ingestion + Analytics / UEBA Profiling
ADP Event Broker (Kafka)
ADP Smart Connectors
ADP Logger
ArcSight ESM
Security Assets AD/AssetDB/HR/Threat
Spark
Historical Analysis
Source Data (via NiFi)
15
16
Banana
![Page 41: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/41.jpg)
41 © Hortonworks Inc. 2011–2018. All rights reserved
HDF
Source Data (via NiFi)
Zeppelin
HDFS
Spark
Historical Analysis
Model as a Service
NiFi - Ingest
Kafka MQ
Storm Parse / Enrich / GeoIP / Index
Automated Response
Solr
Enrichment Data
Netflow / PCAP /
Snort (Kafka direct)
Metron Profiler
Triage
Alert
Investigator UI
HDP
Phase 4 – ArcSight Logger Migration + New Data Sources
ADP Event Broker (Kafka)
ADP Smart
Connectors
ADP Logger
ArcSight ESM
Security Assets
Banana
AD/AssetDB/HR/Threat
17
18
19
20
![Page 42: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/42.jpg)
42 © Hortonworks Inc. 2011–2018. All rights reserved
Questions?
![Page 43: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/43.jpg)
43 © Hortonworks Inc. 2011–2018. All rights reserved
![Page 44: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/44.jpg)
44 © Hortonworks Inc. 2011–2018. All rights reserved
Appendix
![Page 45: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/45.jpg)
45 © Hortonworks Inc. 2011–2018. All rights reserved
Considerations for Sizing Apache Metron
![Page 46: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/46.jpg)
46 © Hortonworks Inc. 2011–2018. All rights reserved
• Events per second (average and peak)
• Retention time for Hot / Warm / Cold zones
• Enrichments
• Node sizing
• I/O Considerations
• PCAP?
Sizing an HCP deployment
![Page 47: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/47.jpg)
47 © Hortonworks Inc. 2011–2018. All rights reserved
3 Months
Hot
Warm
Fast indexed layer (Solr / ES) ~3 months
Warm HDFS layer ~3 months
![Page 48: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/48.jpg)
48 © Hortonworks Inc. 2011–2018. All rights reserved
12 Months
Hot
Warm
Fast indexed layer (Solr / ES) ~3 months
Warm HDFS layer ~12 months
![Page 49: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/49.jpg)
49 © Hortonworks Inc. 2011–2018. All rights reserved
Hot
Warm Cold
Fast indexed layer (Solr / ES) ~3 months
Warm HDFS layer ~12 months
Cold HDFS layer +12 months
24 Months
![Page 50: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/50.jpg)
50 © Hortonworks Inc. 2011–2018. All rights reserved
Cold
Beyond 24 months
Hot
Warm ColdColdColdCold
Fast indexed layer (Solr / ES) ~3 months
Warm HDFS layer ~12 months
Cold HDFS layer +12 months
![Page 51: Apache Metron in the Real World...Automobile Dacia, Romania Cambrian College, Canada Chinese public security bureau CJ CGV Dalian Maritime University Deutsche Bahn DharmaisHospital,](https://reader030.vdocuments.us/reader030/viewer/2022040504/5e358e97d16b7d631e57b245/html5/thumbnails/51.jpg)
51 © Hortonworks Inc. 2011–2018. All rights reserved
Apache Metron Data Sheet• Ingest:
• Apache NiFi: syslog, socket, file, web services, SQL, RDBMS, Windows Event Log, FTP, MQ, JMS, Splunk and others
• High-performance DPDK Packet Capture
• Parsers:• Cisco ASA• Bluecoat• Fireeye• Palo Alto• SourceFire• WebSphere• Snort IDS• Bro DPI• Netflow, IPFIX• Grok (Custom)
• Java (Custom)• JSON• CEF, LEEF (ArcSight, Qradar compat.)• Applications: DHCPD, AD
• Enrichments and threat feeds:• Geo• Whois• HBase• JDBC• Stellar• CSV• Stix, Taxii threat intel feeds
• Analytics features:• Profiler and statistical baselining
engine• Model Services for advanced ML
• Threat Triage rules and scoring engine
• Indexing and search:• Elasticsearch, Kibana• Solr• HDFS• Kafka
• Data science features:• Spark Machine Learning• Zeppelin notebooks and reporting• Wide partner eco-system
• Forensic features:• PCAP inspector• PCAP query• Long term data store