security and audit for big data
Post on 15-Jan-2015
489 Views
Preview:
DESCRIPTION
TRANSCRIPT
© 2013 IBM Corporation
IBM Security Systems
1
© 2013 IBM Corporation
Security and Audit for Big Data
Tina ChenGuardium Enablement
chenti@us.ibm.com
© 2013 IBM Corporation
IBM Security Systems
2
Please note
� IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion.
� Information regarding potential future products is intended to outline our general product
direction and it should not be relied on in making a purchasing decision.
� The information mentioned regarding potential future products is not a commitment,
promise, or legal obligation to deliver any material, code or functionality. Information about
potential future products may not be incorporated into any contract. The development,
release, and timing of any future features or functionality described for our products remains
at our sole discretion.
© 2013 IBM Corporation
IBM Security Systems
3
Bring your own IT
Social business
Cloud and virtualization
1 billion mobile workers
1 trillion connected
objects
Innovative technology changes everything
© 2013 IBM Corporation
IBM Security Systems
4
Compromises take weeks and months to discover & remediate
http://www.verizonbusiness.com/resources/reports/rp_data-breach-investigations-report-2012_en_xg.pdf?CMP=DMC-SMB_Z_ZZ_ZZ_Z_TV_N_Z038
Time span of events by percent of breaches
© 2013 IBM Corporation
IBM Security Systems
5
Four key drivers for data security
CLOUD ADOPTIONEnterprise Security #1 Inhibitor1
APTs DATA BREACHES98% Stolen RecordsFrom Large Orgs2
From Vormetric1. Global State of Information Security® Survey by PwC, CIO magazine, and CSO magazine – October 2012 2. Verizon Data Breach Investigation Report – March 2012
BIG DATABig Data is a Big Target
GLOBALCOMPLIANCEAggressive NewRegulations
� Compliance– PCI-DSS, HIPAA/HITECH, SOX, GLBA, Basel III and others
– National Data Encryption Laws: UK Data Protection Act, EU Data Protection Directive, South Korean PIPA and others
– Require encryption, separation of duties, privileged user controls
� Data Breaches – Valuable data is being targeted by sophisticated attackers - data
breaches increasingly result
– IP Protection, US Federal and State Data Protection laws, Data across Borders
– Encryption + access controls limit risk, meet safe harbor requirements
� Cloud Adoption – Security the #1 concern– Cloud efficiency and flexibility highly desired
– Encryption + access control limits exposure to cloud adminsand other new security risks
� Big Data = Big Risks– Large data sets inevitably include sensitive data
– All data stores and report locations require protection
“To mitigate business risk, you must proactively protect what matters — customer data, financial data and intellectual property — from both outside attackers and privileged insiders.”
© 2013 IBM Corporation
IBM Security Systems
6
Structured
Unstructured
Streaming
Big Data Platform
Hadoop Cluster
Clients
The importance of monitoring Can you answer these questions?
- Who is running map reduce jobs and what are those jobs accessing?
- Is there a new job in the system that hasn’t been vetted?
- Is someone possibly trying to hack into the file system?
© 2013 IBM Corporation
IBM Security Systems
7
© 2013 IBM Corporation
IBM Security Systems
8
Complement existing security with secure databases
© 2013 IBM Corporation
IBM Security Systems
9
InfoSphere Guardium protects sensitive data in Hadoop environments and helps ensure compliance
o Protect your sensitive data with real time activity monitoring
o Gain insights into data activity throughout the stack: Hive, MapReduce,
HBase and HDFS
o Detect unauthorized applications or users
o Real time alerts reduce time to discovery for possible breach or infraction of
compliance
o Automate compliance and management tasks
o Infrastructure in place to provide additional real-time controls over time
Introducing HadoopActivity Monitoring
Application
Storage
MapReduce
Oozie
HDFS
HBase
HiveApplication
Storage
MapReduce
HDFS
HBase
Hive
Monitor and Audit
© 2013 IBM Corporation
IBM Security Systems
10
How it’s done
InfoSphere Guardium monitors key Hadoop events:•Session and user information
•HDFS operations – Commands (cat, tail, chmod, chown, expunge, etc), files, permissions
•MapReduce jobs - Job, operations, permissions
•Exceptions, such as authorization and access control failures
•Hive/HBase queries , - Alter, count, create, drop, get, put, list,..
Heavy lifting occurs on Guardium collector! Very low
overhead on monitored nodes. Architecture supports separation of duties
Hadoop ClusterClients
InfoSphereGuardiumCollector Appliance
S-TAPs
InfoSphere Guardium Reports
Sensitive data alert!
© 2013 IBM Corporation
IBM Security Systems
11
How it is done
© 2013 IBM Corporation
IBM Security Systems
1212
Capture and Parsing Overview
HadoopClient
GuardiumCollector
Analysis engine
Hadoop fs –mkdir /user/data/sundari
Hadoop fs –mkdir ….
Sessions
Commands
Objects
Read Only Hardened Repository
(no direct access)
Hadoop commands
mkdirs
Joe /user/data/sundari
Parse commands
then log
Joe
Namenode
S-TAP
Hadoop fs –mkdir …
Hadoop fs –mkdir/user/data/sundari
© 2013 IBM Corporation
IBM Security Systems
13
A recommended approach
1. Identify users and classes of users – “privileged” users, data scientists…Who is allowed to access sensitive data
� Validate with activity monitoring
2. Identify the applications, jobs, ad-hoc analysis
� Validate with activity monitoring
3. When possible identify and mask sensitive data before it enters the cluster and identify specific directory location in cluster for that data. Put tighter monitoring controls around that data.
4. Look at exceptions – permission exceptions, other operational errors
© 2013 IBM Corporation
IBM Security Systems
14
Use cases
Let’s do the following:
• Log and/or alert access to sensitive files by “unauthorized or unknown” user
• Reporting on new jobs entering the system (identify new MapReduce jobs in the
system)
• Exception reporting for permission errors on sensitive data
Plan
Monitor
Automate
And for each scenario, how to:
© 2013 IBM Corporation
IBM Security Systems
15
Planning for sensitive data access and monitoring
o Do you have PCI or other sensitive data? Is sensitive data already identified in “source” systems?
o How do you carry that sensitive identification over to Hadoop?
o What are the internal and external compliance requirements for monitoring sensitive data access?
o What is the plan for handling violations? Who need to be alerted and when?
PlanPlan
© 2013 IBM Corporation
IBM Security Systems
16
Where is sensitive data?
Configuration
files
Sensitive data or
unknown
PlanKeep sensitive data localized, encrypted, and
under monitoring control
Non-Sensitive
data or known
© 2013 IBM Corporation
IBM Security Systems
17
Monitoring sensitive data
Your policy rules go here, such as sensitive data alerting…
Real-time security policies
Default Hadoop Policy
Flexible, granular rules….
MonitorMonitor
© 2013 IBM Corporation
IBM Security Systems
18
Determine who is accessing sensitive data
When?From where?
Who?
MonitorMonitor
What?
© 2013 IBM Corporation
IBM Security Systems
19
Alerts reduce time to discovery
Incorporate Data Events
into QRadar unified view and real time analytics
AutomateAutomate
Unauthorized access to sensitive data!
© 2013 IBM Corporation
IBM Security Systems
20
Planning for application access
What MapReduce jobs are being used? What kind of ad hoc analysis is allowed on the system?
Have they been vetted for access to sensitive data?
What is a normal pattern of activity?
What process should the team use to communicate new deployments?
No human communication process is infallible…
Task Map(break task into
small parts)
Reduce(many results to a
single result set)
TaskTask Map(break task into
small parts)
Map(break task into
small parts)
Reduce(many results to a
single result set)
Reduce(many results to a
single result set)
Plan
© 2013 IBM Corporation
IBM Security Systems
21
What applications are using the data?
Now, reduce the noise by filtering out authorized jobs….
MonitorMonitor
MapReduce reports ….
© 2013 IBM Corporation
IBM Security Systems
22
What applications are using the data?
Focus your resources on the unknown –
unauthorized MapReduce jobs
MonitorMonitor
© 2013 IBM Corporation
IBM Security Systems
23
What applications are using the data?
Audit process workflow and administrative automation AutomateAutomate
Should this job be approved?
© 2013 IBM Corporation
IBM Security Systems
24
What applications are using the data?
Audit process workflow and administrative automation
Business Owner approves or rejects new applications/jobs
AutomateAutomate
© 2013 IBM Corporation
IBM Security Systems
25
What applications are using the data?
Audit process workflow and administrative automation
Business Owner approves or rejects new applications/jobs
Information Security confirms Business Owner recommendation
AutomateAutomate
© 2013 IBM Corporation
IBM Security Systems
26
What applications are using the data?
Audit process workflow and administrative automation
Business Owner approves or rejects new applications/jobs
Information Security confirms Business Owner recommendation
Guardium Admin adds authorized jobs to “authorized job list”
AutomateAutomate
© 2013 IBM Corporation
IBM Security Systems
27
What applications are using the data?
Populate new vetted applications automatically AutomateAutomate
© 2013 IBM Corporation
IBM Security Systems
28
What applications are using the data?
Populate new vetted applications automatically AutomateAutomate
© 2013 IBM Corporation
IBM Security Systems
29
What applications are using the data?
Populate new vetted applications automatically AutomateAutomate
© 2013 IBM Corporation
IBM Security Systems
30
Alerting off the Exception reports
IO exception
MonitorMonitor AutomateAutomate
Table already exists
© 2013 IBM Corporation
IBM Security Systems
31
• PCI, SOX and HIPAA accelerators included with DAM (guidance, reports, and more)
Continuous database activity monitoring
© 2013 IBM Corporation
IBM Security Systems
32
PCI – Data Access Report
© 2013 IBM Corporation
IBM Security Systems
33
Streamline and simplify compliance processes for Hadoop
Proven track record in data security
Complete separation of duties
Sensitive data monitoring to pass compliance audits
Privileged user monitoring
Real-time alerting for abnormal/suspicious activity
Full forensics: Any activity – views, changes, updates…
Heterogeneous support – IBM, HortonWorks, Cloudera,
Greenplum…
Same platform for all databases in your enterprise
Proven track record in data security
Complete separation of duties
Sensitive data monitoring to pass compliance audits
Privileged user monitoring
Real-time alerting for abnormal/suspicious activity
Full forensics: Any activity – views, changes, updates…
Heterogeneous support – IBM, HortonWorks, Cloudera,
Greenplum…
Same platform for all databases in your enterprise
InfoSphere Guardium Top BenefitsInfoSphereInfoSphere GuardiumGuardium Top BenefitsTop Benefits
© 2013 IBM Corporation
IBM Security Systems
34
© 2013 IBM Corporation
IBM Security Systems
35
IBM Securing all types of data…
� Data privacy for non-production environments:
�Optim Data Privacy (DP)
�Optim Test Data Management (TDM)
� Securing static data on repository :
�Guardium Data Encryption
Stored(Databases, File Servers, Big Data, Data
Warehouses, Application Servers, Cloud/Virtual ..)
Over Network(SQL, HTTP, SSH, FTP, email,. …)
Data in MotionData in Motion
Repository Vulnerability(Database Configuration, Patch Level,
OS Security, …)
ConfigurationConfiguration
� Data privacy for unstructured data (documents)– Guardium Data Redaction
� Data privacy for production environments:– Guardium Data Activity Monitoring
� Ensuring database is configured and patched properly
– Guardium Vulnerability Assessment
– QRadar/QVM
Data at Rest
© 2013 IBM Corporation
IBM Security Systems
36
Data Masking for Data PrivacyMask confidential data to avoid data breach & meet privacy compliance
Database
InfoSphere Optim
JASON MICHAELSJASON MICHAELS ROBERT SMITHROBERT SMITH
Mask
Before Masking After Masking
• Protect sensitive information from misuse and fraud
• Prevent data breaches and associated fines
• Achieve better information governance
• Protect confidential data while preserving analytics
• Mask data anytime, anywhere
• Mask data in Hadoopusing MapReduce
• Implement proven built-in masking algorithms
• Support compliance with privacy regulations
Requirements
Benefits
Mask in-databaseMask in-database Mask in-Hadoopusing MapReduce
Mask in-Hadoopusing MapReduce
Extract, mask & loadExtract, mask & load
IMS
VSAM
More…
Mask filesMask files
Hadoop
© 2013 IBM Corporation
IBM Security Systems
37
Optim Data Masking implementation in Hadoop
JASON MICHAELSJASON MICHAELS ROBERT SMITHROBERT SMITH
OPTIM
Before Masking After Masking
� Optim Masking can also be executed in Hadoop for delimited files.
� Java application/interface for masking.
• MapReduce base classes and helpers.
• Distributed cache.
• Shared libraries.
• Use of masking in Reducers.
� Declarative specification of:
• Metadata of data files
• Masking rules
© 2013 IBM Corporation
IBM Security Systems
38Products Services
Intelligence: A comprehensive portfolio of products and services
New in 2012
© 2013 IBM Corporation
IBM Security Systems
39
All domains feed Security Intelligence
Endpoint Management vulnerabilities enrich QRadar’s
vulnerability database
AppScan Enterprise
AppScan vulnerability results feed QRadar SIEM for improved
asset risk assessment
Tivoli Endpoint Manager
Guardium Identity and Access Management
IBM Security NetworkIntrusion Prevention System
Flow data into QRadar turns NIPS
devices into activity sensors
Identity context for all security domains w/ QRadar as the dashboard
Database assets, rule logic and database activity information
Correlate new threats based on X-Force IP reputation feeds
Hundreds of 3rd party information sources
© 2013 IBM Corporation
IBM Security Systems
40
Key Business Drivers for InfoSphere GuardiumContinuously Monitor All Access too..
Prevent data breaches
Assure data governance
Reduce cost of compliance
© 2013 IBM Corporation
IBM Security Systems
41
Extend real-time Data Activity Monitoring to also protect sensitive data in data warehouses, Big Data Environments and file shares
Integration with
LDAP, IAM,
SIEM, TSM,
Remedy, …
Big Data Environments
DATA
InfoSphereBigInsights
CouchDB
GreenPlum
© 2013 IBM Corporation
IBM Security Systems
42
Information and community
� InfoSphere Guardium YouTube Channel – includes overviews and technical demos
�developerWorks forum (very active)
�Guardium DAM User Group on Linked-In (very active)
�Community on developerWorks (includes content and links to a myriad of sources, articles, etc)
New! InfoSphere Guardium Virtual User Group. Open, technical discussions with other users.
Send a note to bamealm@us.ibm.com if interested.
42
© 2013 IBM Corporation
IBM Security Systems
43
ibm.com/guardium
© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes
only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use
of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any
warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement
governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in
all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole
discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any
way. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United
States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.
Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response
to improper access from within and outside your enterprise. Improper access can result in information being altered, destroyed or misappropriated
or can result in damage to or misuse of your systems, including to attack others. No IT system or product should be considered completely secure
and no single product or security measure can be completely effective in preventing improper access. IBM systems and products are designed to
be part of a comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems,
products or services to be most effective. IBM DOES NOT WARRANT THAT SYSTEMS AND PRODUCTS ARE IMMUNE FROM THE
MALICIOUS OR ILLEGAL CONDUCT OF ANY PARTY.
© 2013 IBM Corporation
IBM Security Systems
44
Disclaimer
Please Note:
IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general product
direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise,
or legal obligation to deliver any material, code or functionality. Information about potential
future products may not be incorporated into any contract. The development, release, and
timing of any future features or functionality described for our products remains at our sole
discretion.
top related