Download - Solving Cyber at Scale
![Page 1: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/1.jpg)
Solving Cyber at Scalewith Hadoop, Storm and Metron
![Page 2: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/2.jpg)
Simon Elliston Ball
• Product Manager
• Data Scientist
• Elephant herder
• @sireb
![Page 3: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/3.jpg)
Threat Sources
![Page 4: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/4.jpg)
IoT: Mirai
Reports of 1.2 Tbps
500,000 devices at peak
DDoS attacks on Dyn DNS services
![Page 5: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/5.jpg)
Insiders
![Page 6: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/6.jpg)
Ransomware and spears
![Page 7: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/7.jpg)
Who are we up against?
![Page 8: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/8.jpg)
MEECES
Money
Ego
Entertainment
Cause
Entrance (social acceptance)
Status
![Page 9: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/9.jpg)
Big Business
• $tn market
• Access is bought and sold: 5 bitcoin for 100m accounts
• Sharing networks
• Criminals as a Service
• DDoS attacks: cost attackers $5 per hour, defenders ~$40k
Sources: BT and KPMG Report, Taking the Offensive
![Page 10: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/10.jpg)
Challenges for the Modern SOC
![Page 11: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/11.jpg)
Drowning in Data
![Page 12: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/12.jpg)
Staff shortage
![Page 13: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/13.jpg)
Long tail problem
![Page 14: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/14.jpg)
What we have now
![Page 15: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/15.jpg)
Silos
Packet
Store
SIEM
Log
StoreForensics Tools
Endpoint Agents
Cases
Threat Intel
UEBA
Anti Virus
Email filter
![Page 16: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/16.jpg)
Rules: Asset or
Liability
![Page 17: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/17.jpg)
Shiny
new tools
![Page 18: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/18.jpg)
Solutions: machine learning! magic!
Triage Automation
Detecting the unknown unknowns
Explaining yourself
![Page 19: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/19.jpg)
The value of real time
Data in Motion: why wait until it’s at rest?
Correct context: the world moved on
![Page 20: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/20.jpg)
Better data = analyst efficiency
Fully enriched data
Real context
Consistency
= faster triage and better coverage
![Page 21: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/21.jpg)
Single View of Business & Security
Risks
HR
Finance
Web Logs
Security Appliances
Syslogs
Geolocation
Network Data
IoT
Telemetry Data
Operations
CRM
![Page 22: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/22.jpg)
Longer term data
• Attacks last months
• So should your queryable data
![Page 23: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/23.jpg)
Executable solutions
• Orchestration
• Machine-time response
![Page 24: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/24.jpg)
How to do it
![Page 25: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/25.jpg)
Network Level Taps
![Page 26: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/26.jpg)
Data Sources and Aggregation
Open standards for data models = more productive data scientists + shareable models
Business level data sources link security to real business risk.
![Page 27: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/27.jpg)
Massively scalable platforms
![Page 28: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/28.jpg)
![Page 29: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/29.jpg)
29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Data Se
rvices an
d In
tegratio
n Laye
r
Search andDashboarding
Portal
Security Data Vault
CommunityAnalytical
Models
Provisioning,Management
and Monitoring
ModulesReal-time ProcessingCyber Security Engine
TelemetryParsers Enrichment
ThreatIntel
AlertTriage
Indexersand
Writers
Cyber SecurityStream Processing Pipeline
Apache Metron: a framework for Big Data Driven cyber security
Tele
metry In
gest B
uffe
r
TelemetryData Collectors
Real-timeEnrich / ThreatIntel Streams
PerformanceNetwork
IngestProbes
/ OtherMachine Generated Logs(AD, App / Web Server,
firewall, VPN, etc.)
Security Endpoint Devices (Fireye, Palo Alto,
BlueCoat, etc.)
Network Data(PCAP, Netflow, Bro, etc.)
IDS(Suricata, Snort, etc.)
Threat Intelligence Feeds(Soltra, OpenTaxi,third-party feeds)
TelemetryData Sources
![Page 30: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/30.jpg)
Community Development
• http://metron.apache.org
• https://github.com/apache/incubator-metron/
![Page 31: Solving Cyber at Scale](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6515b27f8b9aa2548b6c83/html5/thumbnails/31.jpg)
Thank you!
• Apache Metron: http://metron.apache.org
• Twitter: @sireb