security analytics for data discovery - closing the siem gap
TRANSCRIPT
Security Analytics for Data Discovery:Closing the SIEM Gap
Eric JohansenSr. Solutions Architect
Background
Virus CERT / Incident Response @ IBM
MSS Architect @ IBM Internet Security Systems SME: IBM SELM (Security Event & Log Management)
MSS Architect @ FishNet Security Launched Hosted SIEM and Co-managed SIEM Services
MSS Biz Dev @ Optiv
Overview
- Hunting- The SIEM Gap- The Problem with Hunting (and the Solution)- Unknowns (and how to turn them into
Knowns)- Wrap up
Hunting Defined
Proactive versus reactive approach to identifying incidents
Reactive: incident starts when a notification comes in.
Proactive: actively looking for incidents - based on patterns, intelligence, or even hunches.Source: Scott J. Roberts - http://sroberts.github.io/2015/04/14/ir-is-dead-long-live-ir/.
Hunting Maturity Model
3 Factors Contribute:1) Quality of data – the more data the better2) Tools provided to access and analyze the data3) Skills of the analysts using the data (hunting)
Source: David Bianco - http://detect-respond.blogspot.com.au/2015/10/a-simple-hunting-maturity-model.html.
Maturity Indicators:Threat Intel
Data Analysis ProceduresAutomation
Security Analytics – A Path to Hunting Maturity
“Advanced analytics are being integrated into security markets after rule- and signature-based prevention systems and tuning processes struggled to detect or stop most security breaches over the past few years”
Source: Gartner - The Fast-Evolving State of Security Analytics, 2016 – April 4, 2016.
The SIEM Gap Defined
Designed for the known- Normalize / parse logs with defined compatibility- Alerts based on policy - Pre-defined reporting- Automated Data Analysis (for compliance / audit)
If there’s not a rule, policy, report, or alert – nothing gets detected.
Architectural decisions made then - now fundamentally limit SIEM.(Technology advancements have enabled Security Analytics).
Not really designed for human interaction – i.e. hunting and incident response.
The SIEM Gap - Industry Analyst Perspectives
• Requires advanced skills and knowledge• Custom queries are difficult• Challenges collecting certain types of data• Lacks context for collected data• Too many false positive alerts
• Primary challenge is complexity• Performance limits galore• Data variety challenges• New environment explosion• Analysis? Where is that?
Data Analysis Evolution
Example Products
Delivery
Create Views
Use Cases
Predefined Reports
HP Arcsight
Vendors
Compliance
Structured Data Aggregation
Data Scientists
Visualize the Known
Custom Dashboards
SecurityOperations
Splunk
SMEs
Discover the Unknown
Security Analytics
Integrated Operations
Data Discovery Workflow
The Problem with Hunting
“Effective threat hunting remains the domain of the well-resourced, super-security-mature, extra-skilled security 1%-ers…”
Source: Anton Chuvakin – http://blogs.gartner.com/anton-chuvakin/2016/03/21/antons-favorite-threat-hunting-links/.
The Most Sophisticated Analytic on the Planet
A Profound Shift – Known to Unknown
Repor
t on answersCollect only
data required to answer
questions
Develop list of questions
Known
Analytics-enabled exploration and discovery
Collect everything
No list of questions
Cloud
Virtual
Unknown
Security Analytics – Techniques for the Unknown
Event Clusters
Rapidly analyze large data sets with machine learning – event clusters technology summarizes the data set based on commonality to allow for quick human analysis.
Security Analytics – Techniques for the Unknown
Association Analytics
Explore frequency in your data in different categories, i.e. IP addresses, geolocations, usernames, applications, etc.
Security Analytics – Techniques for the Unknown
Activity & Change
Compare datasets and timeframes for differences – trending up/down, what’s new, etc.
Security Analytics – Techniques for the Unknown
Cohort Analysis
“Guilt by association”
Security Analytics – Techniques for the Unknown
Visualization / Perspective
See the data – find outliers - explore
Security Analytics – Techniques for the Unknown
Natural Language Processing
Deconstruct messages to attempt to find the direct and implied informationcontent.
- Actions (verbs) – allow, deny, block, fail, etc.- Subjects (proper nouns) – addresses, usernames, etc.- Various other parts of speech (direct objects, prepositions, adjectives, etc.) that add nuance- Fuzzy
Security Analytics Search Engine- Much like Google – to the user Google looks like one big bucket of one big field.- Under the covers - adding in metadata to add hints and help improve relevance.
Security Analytics – Techniques for the Unknown
Clustering (Big Data) and Federation (Data Politics)
Security Analytics – Techniques for the Unknown
Flexible Real-time Data Collection
- Streaming Packet Capture: Forensic analysis on demand- Any TCP/UDP Port- All usual suspects (syslog, flat files, netflow, etc.)- Define repository, TTL, rate limit
Security Analytics – Techniques for the Unknown
Drag and Drop Import
- Simple browser interface to bring in disparate data- Define repository, TTL, delimiters, time (now versus time discovered in data)
- Take in anything human readable- Office files, Outlook PST, PDF, PCAP, configuration files, and much more.- Threat Intel and CMDB Data
Security Analytics – Techniques for the Unknown
Collaboration
- Pinboard- Save and share commonly used queries.
- Tags, Notes- Rapidly record observations in data
Security Analytics – Techniques for the Unknown
Automation
- Workflow- Create repeatable processes within your data.
- Remotes- Tie remote agent based actions into Workflow or use ad-hoc.
Security Analytics – A Path to Hunting Maturity
HuntingDiscover The Unknown
Rapid Event TriageDiscover The Cause
Incident ResponseDiscover Incident Context
Data AccessibilityDiscover From More Data
Search for outbound deny events and view clusters, trends and associations to spot high risk activity.
Drag log files from multiple sources into the system, retain original date, create time-correlated views.
Automatically correlate alerts and human data with automatically enriched infrastructure data.
Drag the 2G log file and 4G PCAP into the system as easy as uploading to Dropbox.
Clusters, comparisons and associations are automatic.
Hunting Maturity Model Revisited
3 Factors Contribute:1) Quality of data – the more data the better2) Tools provided to access and analyze the data3) Skills of the analysts using the data (hunting)
Source: David Bianco - http://detect-respond.blogspot.com.au/2015/10/a-simple-hunting-maturity-model.html.
Maturity Indicators:Threat Intel
Data Analysis ProceduresAutomation
