ibm big data_security_analitycs_2014_scavanna

© 2012 IBM Corporation

IBM Security Systems

1 IBM Security Systems © 2012 IBM Corporation © 2013 IBM Corporation

IBM BIG Data Security analitycs Santiago Cavanna IBM Security Sales Specialist Argentina, Uruguay, Paraguay.

© 2011 IBM Corporation 2 IBM Confidential

IBM Security Strategy

Confidential – for division executives only


Why Security Intelligence?



3 IBM Security Systems

Big Data

Step 1: Collect and store massive amounts of security data

For Large Customer

250,000 managed firewalls 30,000 network devices

500,000 open port combinations

410,455 Windows client systems

36,109 Windows servers 24,000 *NIX servers

1200+ products for VA Large Scale Size Estimates

1.5 – 2 TB per month per major security service 200 – 750GB total per minor security service

900 K – 1M events per second; 86 B events per day

E-mail and social activity

System audit trails and logs

Configuration data from

infrastructure

Full packet and DNS captures

Business process data

External threat feeds

Network flows and anomalies

Malware samples and

behavior

Alerts from security sensors

Vulnerability and patch information

Security logs from servers

Traditional Security Operations and Technology

Next Generation Security Operations and Technology




Captured

Detected

Inferred

Step 2: Apply real-time and historical analytics to build insights beyond traditional security tools

Descriptive Analytics

Decision modeling

Historic and Predictive Analytics

Forecasting What if these

trends continue?

Risk mitigation & avoidance How can we mitigate risk? How can we avoid this?

What happened?

What exactly is the problem?

How many, how often, where?

Is it significant? Risk impact?

Historic Has it happened before? How did we deal with it?

Decision making

What actions are to be taken?

Predictive Modelling What will

happen next if?

Data Information




The Security Intelligence, Analytics and Big Data

Volume: Scale data volumes from terabytes to multiple petabytes – identifying security evidence and anomalies within vast amounts of data using ad-hoc and historical queries

Velocity: Increase the velocity of critical decision making by performing analytics on real-time data – detecting and responding to threats before they cause damage

Variety:

Derive actionable security intelligence by analyzing a variety of sources –ranging from structured events, audit logs, network flows, raw text and other forms of unstructured data

Significantly improve the accuracy, speed and depth of insights used to identify and remediate unknown threats




IBM Big Data Platform: Move Analytics Closer to the Data

Permite partir pequeño y escalar Componentes compartidos y sistemas

integrados disminuye costos, tiempo y riesgos.

Características Claves

•  Aceleradores construidos de múltiples componentes

•  Integraciones Pre-Construidas entre componentes usando conectores abiertos

•  Motores analíticos comunes (i.e. text analytics)

•  Metadata común, diseño de integración y governance a través de la plataforma




A continued journey towards total Security Intelligence





IBM QRadar Security Intelligence Platform




Explosión de datos IBM se integra en los silos de TI con soluciones de Inteligencia de seguridad.

Fuentes Inteligencia Información más exacta y accionable + =




Southbound APIs

Northbound APIs

Security Intelligence architecture provides continuity with total context


Real Time Structured Security Data Unstructured Operational / Security Data

Log Management

NextGen SIEM

Network Activity

Monitoring Risk

Management Vulnerability Management

Network Forensics Future

Real Time and Analyst-driven Work Flow

Real Time Correlation / Automated Security Analytics

Big Data Store / Warehouse / Archival

Security Intelligence

Operating System




Organizations need end-to-end visibility to succeed

People Safeguard and monitor access to IT systems,

applications and information

Data Continuous monitoring and assessment of databases,

warehouses, file shares and big data environments

Applications Identify and remediate critical web and mobile

application vulnerabilities before they have an impact

Infrastructure Discover, remediate and

block threats to constantly changing networks, servers

and endpoints

Research Constantly monitor the

threat landscape for new

vulnerabilities




Fully Integrated Security Intelligence

•  Turn-key log management and reporting •  SME to Enterprise •  Upgradeable to enterprise SIEM

•  Log, flow, vulnerability & identity correlation •  Sophisticated asset profiling •  Offense management and workflow

•  Network security configuration monitoring •  Vulnerability prioritization •  Predictive threat modeling & simulation

SIEM

Log Management

Configuration & Vulnerability

Management

Network Activity & Anomaly Detection

Network and Application

Visibility

•  Network analytics •  Behavioral anomaly detection •  Fully integrated in SIEM

•  Layer 7 application monitoring •  Content capture for deep insight & forensics •  Physical and virtual environments

•  Bringing rich context to Vulnerability Management

•  Improves visibility •  Unified vulnerability view across all

products

Vulnerability Manager





QRadar & Big Data




IBM’s Security Intelligence, Analytics and Big Data portfolio

1 IBM QRadar Security Intelligence unified architecture for collecting, storing, analyzing and querying log, threat, vulnerability and risk related data

2 IBM Big Data Platform (Streams, Big Insights, Netezza) addresses the speed and flexibility required for customized data exploration, discovery and unstructured analysis

3 IBM i2 Analyst Notebook helps analysts investigate fraud by discovering patterns and trends across volumes of data

4 IBM SPSS unified product family to help capture, predict, discover trends, and automatically deliver high-volume, optimized decisions




IBM Big Data for Security Solution

Real-time Processing •  Focus on HOT, real-time data •  Event normalization •  Real-time correlation •  Data enrichment

Security Operations •  Detailed security metrics •  Activity & event graphs •  Incident management •  Compliance reporting

Big Data Warehouse •  Storage for HOT, Warm & cold data •  Unstructured and structured •  Distributed infrastructure •  Preserves raw data •  Scalable platform •  Large-scale

machine learning •  Hadoop-based

backend

Big Data Analytics and Forensics •  Advanced visuals and interaction •  Predictive and decision modeling •  Ad hoc and historical queries •  Transaction and geo analysis •  Custom reports and dashboards •  Pluggable UI •  Collaborative

sharing tools

Analyze Store & Process Collect

Data Sources •  Security Devices

•  Server and Host Logs

•  Network and Virtual Activity

•  Database Activity

•  Application Activity

•  Vulnerability and Config Data

•  Threat Intelligence Feeds

•  User Activity and Behavior

•  Web, Blogs, & Social Activity

•  Business Transactions

•  Unstructured data (e.g. Email)

Advanced Security Analytics & Correlation Engine

Big Data Security Workbench





Architecture & Use Cases




IBM QRadar provides out-of-the-box security intelligence, while the IBM Big Data Platform is customizable to satisfy unique needs

IBM Big Data Platform

No se puede mostrar la imagen. Puede que su equipo no tenga suficiente memoria para abrir la imagen o que ésta esté dañada. Reinicie el equipo y, a continuación, abra el archivo de nuevo. Si sigue apareciendo la x roja, puede que tenga que borrar la imagen e insertarla de nuevo.



Security use cases Turnkey Custom

User Interface All-in-one console Purpose-built applications

Data Sources 450+ preconfigured (and growing) Everything else

Data Volume 100+ Terabyte range Peta-byte range

Real-time Analysis Seconds Milliseconds

Analytics Pre-built, learned Custom, learning

Required Expertise Average - Security practitioners Skilled analysts

No se puede mostrar la imagen. Puede que su equipo no tenga suficiente memoria para abrir la imagen o que ésta esté dañada. Reinicie el

InfoSphere Big Insights and Streams




From NetFlow to QFlow to… …QRadar Incident Forensics

Internet/ intranet

packet

Netflow: packet oriented, identifies unidirectional sequences sharing source and destination IPs, ports, and type of service

Internet/ intranet

QFlow: packet oriented, identifies bi-directional sequences aggregated into sessions, also identifies applications by capturing the beginning of a flow.

Internet/ intranet

Competitive solutions: session oriented, some only capture a subset of each flow and index only the metadata—not the payload.

Internet/ intranet

QRadar Incident Forensics: session oriented, captures all packets in a flow indexing the metadata and payload to enable fast search driven data exploration




IBM Big Data for Security Solution – Components and data flow

Real-time Processing

Security Operations

Big Data Warehouse Big Data Analytics and Forensics

Analyze Store & Process Collect

Data Sources

Security and Infrastructure Data Sources

External Threat Intelligence Feeds

Email, Web, Blogs, and Social Activity

Relational Store •  High-value Information

Hadoop Store •  Raw Data

InfoSphere BigInsights

i2 Intelligence Analysis

InfoSphere BigSheets

QRadar Console (Web interface)

2 Real-time insights (HOT)

3 Forward (HOT) & Store (HOT, Warm, cold) data

6 Enrich / Adapt / Improve

5 Advanced Visualizations and Investigation – (Warm and cold)

4

1 Data Collection & Enrichment (HOT) Flow of data/information

Flow of knowledge

•  Watch List •  Custom Rules

QRadar Security Intelligence Platform

Big Data Analysis, Trends & History (Warm and cold)




Big Data Warehouse


Data Sources

Use case #1 – Detection of an internal compromised system

Security Operations

Big Data Analytics and Forensics

Hadoop Store

Firewall

Router

QFlow Collector

Requirements Source: Netflow

Sample Size: >100GB /src

Query time: <30sec

Analytics: Time interval and network flow size

Relational Store

1

1

1 2

5

9

4

6

8 7 Compromised System

3




Big Data Analytics and Forensics

Data Sources

Security Operations Real-time Processing

Use case #2 – Detection of a malicious external subnet

Big Data Warehouse

Firewall

Web Proxy

Email Gateway

Requirements Source: FW, email, proxy


Query time: <30sec

Analytics: Graphical view of malicious subnet

Relational Store

1

1

1 2

5

9

8 7

Malicious Subnet

4 3

Hadoop Store

6





Security Operations Data Sources

Use case #3 – User profiling based on multiple sources


Web Proxy

Email Gateway

Requirements Source: proxy, email, unstructured text


Query time: <45sec

Analytics: Multiple

Relational Store

4

1

1

9

8 7

Unstructured Data

5

Suspicious User(s)

3

2

Hadoop Store

6




Data Sources

Web and Email Proxy


Use case #4 – Ad hoc query for specific data on multiple sources

Security Operations


Netflow

Requirements Source: All


Query time: <45sec

Analytics: Search for IP, FQDN and/or email address

Relational Store

Unstructured Data

4

1

1

9

8 7

5

“Needle in a Haystack”

“Needle in a Haystack”


3 No se puede mostrar la imagen. Puede que su equipo no tenga suficiente memoria para abrir la


2

Hadoop Store

6





IBM Big Data Platform





§  Flexible, enterprise-class support for processing large volumes of data – Based on Google’s MapReduce technology – Foundation of Apache Hadoop; 100% compatible with its ecosystem and distribution – Well-suited to batch-oriented, read-intensive applications – Supports wide variety of data

§  Enables applications to work with thousands of nodes and petabytes of data in a highly parallel, cost effective manner

– CPU + disks = “node” – Nodes can be combined into clusters – New nodes can be added as needed without changing

•  Data formats •  How data is loaded •  How jobs are written

Traditional / Non-traditional

data sources

Internet-Scale Analytics




26

InfoSphere BigInsights En

terp

rise

Valu

e

Com

mod

itize

d

Mon

etiz

atio

n po

tent

ial

Core Hadoop

BigInsights Basic Edition

BigInsights Enterprise Edition

Free download with web support Limit to <= 10 TB of data

(Optional: 24x7 paid support

Fixed Term License)

Professional Services Offerings QuickStart, Bootcamp, Education, Custom Development

Enterprise-grade features:

Tiered Terabyte-based pricing

Easy Installation And programming

Analytics tooling/visualization Administration tooling Development tooling High Availability Flexible storage Recoverability Security




BigInsights Enterprise Edition

Connectivity and Integration Streams

Netezza

Text processing engine and library

JDBC

Flume

Infrastructure Jaql

Hive

Pig

HBase

MapReduce

HDFS

ZooKeeper Indexing Lucene

Adaptive MapReduce

Oozie

Text compression

Enhanced security

Flexible scheduler

Optional IBM and partner offerings

Analytics and discovery “Apps”

DB2

BigSheets

Web Crawler

Distrib file copy

DB export

Boardreader

DB import

Ad hoc query

Machine learning

Data processing

. . .

Administrative and development tools

Web console •  Monitor cluster health, jobs, etc. •  Add / remove nodes •  Start / stop services •  Inspect job status •  Inspect workflow status •  Deploy applications •  Launch apps / jobs •  Work with distrib file system • Work with spreadsheet interface • Support REST-based API •  . . .

R

Eclipse tools •  Text analytics •  MapReduce programming •  Jaql, Hive, Pig development •  BigSheets plug-in development •  Oozie workflow generation

Integrated installer

Open Source IBM IBM

Cognos BI

Big SQL

Accelerator for machine data analysis

Accelerator for social data analysis

Guardium DataStage Data Explorer

Sqoop

HCatalog GPFS –FPO




InfoSphere Streams

§  Applications that require on-the-fly processing, filtering and analysis of streaming data – Sensors: environmental, industrial, surveillance video, GPS, … –  “Data exhaust”: network/system/web server/app server log files – High-rate transaction data: financial transactions, call detail records

§  Criteria: two or more of the following – Messages are processed in isolation or in limited data windows – Sources include non-traditional data (spatial, imagery, text, …) – Sources vary in connection methods, data rates, and

processing requirements, presenting integration challenges – Data rates/volumes require the resources of

multiple processing nodes – Analysis and response are needed with

sub-millisecond latency – Data rates and volumes are too great

for store-and-mine approaches Traditional / Non-traditional

data sources

Millions of events per

second

Microsecond Latency

Real-time decisions




à continuous ingestion à Continuous ingestion à Continuous analysis

IBM INFOSPHERE STREAMS




Achieve scale: By partitioning applications into software components By distributing across stream-connected hardware hosts

Infrastructure provides services for Scheduling analytics across hardware hosts, Establishing streaming connectivity

Transform Filter / Sample

Classify Correlate

Annotate

Where appropriate: Elements can be fused together for lower communication latency

à Continuous ingestion à Continuous analysis

IBM INFOSPHERE STREAMS




Scalable Stream Processing

§  InfoSphere Streams provides – a programming model for defining data flow graphs consisting of data sources (inputs),

operators, and sinks (outputs) – controls for fusing operators into processing elements (PEs) –  infrastructure to support the composition of scalable stream processing applications

from these components – deployment and operation of these applications

across distributed x86 processing nodes, when scaled-up processing is required




ibm.com/security

© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response to improper access from within and outside your enterprise. Improper access can result in information being altered, destroyed or misappropriated or can result in damage to or misuse of your systems, including to attack others. No IT system or product should be considered completely secure and no single product or security measure can be completely effective in preventing improper access. IBM systems and products are designed to be part of a comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems, products or services to be most effective. IBM DOES NOT WARRANT THAT SYSTEMS AND PRODUCTS ARE IMMUNE FROM THE MALICIOUS OR ILLEGAL CONDUCT OF ANY PARTY.





BACKUP Slides




Prediction & Prevention Reaction & Remediation Network and Host Intrusion Prevention.

Network Anomaly Detection. Packet Forensics. Database Activity Monitoring. Data Leak Prevention.

SIEM. Log Management. Incident Response.

Risk Management. Vulnerability Management. Configuration and Patch Management.

X-Force Research and Threat Intelligence. Compliance Management. Reporting and Scorecards.

What are the external and internal threats?

Are we configured to protect against

these threats? What is happening

right now? What was the

impact?

IBM QRadar - solutions for the full Security Intelligence timeline




Inteligente, integrado y automatizado - Una consola de Seguridad

QRadar Log Manager

QRadar SIEM

QRadar QFlow

QRadar Risk

Manager

QRadar VFlow

Reporting Engine Workflow Rules Engine Real-Time

Viewer

Analytics Engine

Warehouse Archival

Reporting API

Forensics API

LEEF AXIS Configuration Netflow Offense

Soluciones de Inteligencia

en Seguridad

Plataforma para

Inteligencia en Seguridad

Normalization

IBM (Q1 Labs) y su familia de soluciones QRadar

QRadar Vulnerability

Manager




Uso para detección de ataques

¿Cuál fue el ataque?!

¿Quién fue el responsable?!

¿Cuáles fueron los objetivos

involucrados?!

¿Fue exitoso?!

¿Dónde puedo

encontrarlos?!

¿Alguno de ellos era vulnerable?!

¿Qué tan importante son los activos para

el negocio?!

¿Dónde se encuentra toda la evidencia?!

Toda la información relevante en un mismo lugar




Uso para identificación de actividades maliciosas

Aplicación IRC en el puerto 80?!

QFlow permite la detección de tráfico infilrado!

Prueba Irrefutable del Botet!Tráfico de capa 7 contiene instrucciones de comando y control de una red Botnet.!

Detección posible de Botnet?!

Lo que tipicamente es capaz una solucion de SIEM!





Quick Overview




Advanced Threat Detection

Traditional Data Sources + Network + Context

Security Intelligence Platform

What? Solving new Security Intelligence challenges with Big Data

QRadar •  Data collection •  Event correlation •  Real-time analytics •  Offense prioritization

What customers are asking us:

1. Significantly increase the volume of data stored for forensics and historic analysis

2. Analyze a variety of non-traditional and unstructured datasets - such as email, web content, files and full packets

3. Visualize data in new ways, using custom queries, graphs, linguistics, maps, etc.

4. Integrate this capability with my current security operations




Data Ingest

Insights

QRadar •  Hadoop-based

data integration •  Data mining •  Custom analytics •  Machine learning

•  Data collection •  Event correlation •  Real-time analytics •  Offense prioritization

Big Data / Analyst Workbench

Custom Use Cases Advanced Threat Detection

Traditional Data Sources + Network + Context


Non-traditional

How? By integrating QRadar with IBM’s Hadoop-based offering

Security Intelligence Platform

ibm big data_security_analitycs_2014_scavanna

Technology

security evidence

generation security

major security service

minor security service

security sensors vulnerability

data information

big data volume

logs configuration data