hadoop security features that make your risk officer happy
TRANSCRIPT
![Page 1: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/1.jpg)
HADOOP SECURITY FEATURES
That make your risk officer happy
By Anurag Shrivastava, ING Commercial Bank, Amsterdam
@shri2201
![Page 2: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/2.jpg)
Security for Hadoop
Source: http://blogs.gartner.com/merv-adrian/2014/01/21/security-for-hadoop-dont-look-now/
Hadoop Security Features 2
![Page 3: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/3.jpg)
Hadoop in Enterprise
Data Lake – an important information assets for enterprise
Data from System of
Records and Logs are stored
in Hadoop
Significant cost
savings for
Enterprise
Diverse types of
users
Picture Source: http://arunkottolli.blogspot.nl/2014/03/understanding-data-in-big-data.html
Hadoop Security Features 3
![Page 4: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/4.jpg)
Operational Security in Enterprise• User Access Management
• Security Event Monitoring
• Application State Monitoring
• Security Testing
• Patch Management
• Data Protection
• Backup and restore
Hadoop Security Features 4
![Page 5: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/5.jpg)
User Access Management
Requirements
Privileged, group and generic accounts
Separation of technical and business users
Separation of environments (DTAP)
Separation of admins and other users
Separation of users in different business roles
Application of four eyes principle when entering or
changing the data
Hadoop Security Features 5
![Page 6: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/6.jpg)
Security Event Monitoring• Definition of application specific events
• All login attempts failed or successful
• Unauthorized attempt to access a table or file
• Operational performance of application
• Name node performance
• CPU, Disk
• Integration with Master Control Room
• Alerting the asset manager
Hadoop Security Features 6
![Page 7: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/7.jpg)
Data Protection (1/2)• Confidentiality
• Protect information from unauthorized
disclosure
• Integrity
• Ensure the accuracy, completeness and
timeliness of information and prevent data
tempering
• Availability
• Ensure that information and service is
available when required
Picture Source:
http://www.attix5.co.uk/thought-
leadership/why-data-protection-software-
essential-good-nights-sleep
Hadoop Security Features 7
![Page 8: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/8.jpg)
Data Protection (2/2)• Confidentiality
• Logon
• Access Control
• Malicious code protection
• Security Event Monitoring
• Encryption
• Integrity
• Message authentication code
• Data Lineage
Picture Source:
http://www.attix5.co.uk/thought-
leadership/why-data-protection-software-
essential-good-nights-sleep
Hadoop Security Features 8
![Page 9: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/9.jpg)
Security under spotlight in Data Lake
• All kinds of enterprise data – structured,
semi-structured and unstructured
• Many groups of users – Data Scientists,
Analysts, Engineers, Marketers,
Managers
• Long term retention of data
• Different types of workloads
• Value of data grows as the data from
different sources are combined in Data
Lake
Picture source: http://beyondplm.com/2014/05/05/plm-downstream-usage-and-future-information-rivers/
Hadoop Security Features 9
![Page 10: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/10.jpg)
Data Lake Risks
• Data Lake is an attractive target of inside and outside attackers
• Security compromise in Data Lake can have major or catastrophic
business impact
IT Risk assessment gives Hadoop implementation
the highest risk rating for Data Lake use case.
Hadoop Security Features 10
![Page 11: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/11.jpg)
Lab Like Security is not Enough
Play Area Big Data Predictive Analytics
Lab
Production
System
Hadoop Security Features 11
![Page 12: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/12.jpg)
Predictive Analytics Lab
Stepping Stone
(Citrix)
18 x Hadoop
Nodes
GIT, Libraries,
Build Tools
Monitoring
Services
Data Files in
Batches
Dedicated VLAN Shared ServicesShared Services
SMTP Relay
Internet via
Corporate
Infrastructure
Firewall Rules
Guard the
Perimeter
Security
Of Hadoop
Cluster
18 x Hadoop
Nodes
Lab like security works for a small group of people
Hadoop Security Features 12
![Page 13: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/13.jpg)
Limitations of Hadoop• No “Data at Rest” Encryption
• A Kerberos-Centric Approach
• Limited Authorization Capabilities
• Complexity of the Security Model and Configuration
Unfortunately this is not sufficient for Data Lake that ingests all the
data and caters to thousands of users.
Hadoop Security Features 13
![Page 14: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/14.jpg)
Hadoop Security
Hadoop Security Solutions from Major Vendors
Hortonworks acquires XASecure to
bring ACLs in Hadoop
Apache Ranger
Apache Knox
Apache Falcon
Cloudera is working on Project Rhino Project Rhino
Apache Sentry
Hadoop Security Features 14
![Page 15: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/15.jpg)
HDP-Apache Ranger
Hadoop Security Features 15
![Page 16: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/16.jpg)
Apache RangerApache Ranger currently supports authorization, auditing and security administration of limited
number of HDP components
Hive
HBase
StormKnox
HDFS
Hadoop Security Features 16
![Page 17: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/17.jpg)
Apache Ranger Goals
1. Centralized security administration to manage all security related tasks in
a central UI or using REST APIs.
2. Fine grained authorization to do a specific action and/or operation with
Hadoop component/tool and managed through a central administration tool
3. Standardize authorization method across all Hadoop components.
4. Enhanced support for different authorization methods - Role based access
control, attribute based access control etc.
5. Centralize auditing of user access and administrative actions (security
related) within all the components of Hadoop.
Hadoop Security Features 17
![Page 18: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/18.jpg)
Apache Knox and Hadoop Services
Hadoop Services
Covered
• WebHDFS (HDFS)
• Templeton
(HCatalog)
• Stargate (HBase)
• Oozie
• Hive/JDBC
Hadoop Security Features 18
![Page 19: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/19.jpg)
Apache Falcon• Visualize Data Pipeline Lineage
• Track Data Pipeline audit logs
• End to End Monitoring of Data
Pipeline
• Policies for Data Replication and
Retention
Hadoop Security Features 19
![Page 20: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/20.jpg)
Apache Sentry and Project Rhino
Hadoop Security Features 20
![Page 21: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/21.jpg)
Goals of Project Rhino
• Provide encryption with hardware-enhanced performance
• Support enterprise-grade authentication and single sign-on for
Hadoop services
• Provide role-based access control in Hadoop with cell-level
granularity in HBase
• Ensure consistent auditing across essential Apache Hadoop
components
Hadoop Security Features 21
![Page 22: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/22.jpg)
Apache Sentry and Project Rhino
Hadoop Security Features 22
![Page 23: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/23.jpg)
Making Risk Officer Happy
• Hadoop security has
more to offer
• Role based access
• Audit logging
• Data encryption
• User Access Management
• Security Event Monitoring
• Application State Monitoring
• Security Testing
• Patch Management
• Data Protection
• Backup and restore
Overlapping efforts of vendors, Lack of complete coverage for all products,
Varying commitment to open source would slow down the adoption of Hadoop.
Hadoop Security Features 23
![Page 24: Hadoop Security Features that make your risk officer happy](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a9320e1a28ab30368b4661/html5/thumbnails/24.jpg)
THANK YOUAnurag Shrivastava
@shri2201