kafka security
TRANSCRIPT
Page 1 © Hortonworks Inc. 2014
Kafka Security
SSL, Kerberos & Authorization
Page 2 © Hortonworks Inc. 2014
Page 3 © Hortonworks Inc. 2014
Who Are We?
Sriharsha ChintalapaniApache Kafka Committer
Apache Storm Committer & PMC
Parth BrahmbhattApache Kafka Contributor
Apache Storm Committer & PMC
Page 4 © Hortonworks Inc. 2014
Kafka Security
• SSL ( wire encryption)
• SASL ( Kerberos )
• Authorizer (Topic/Host/User level Authorization)
Page 5 © Hortonworks Inc. 2014
SSL
Page 6 © Hortonworks Inc. 2014
Kafka Security – SSL
• Kafka networking
• A TCP server listening for incoming connections
• Uses Non-blocking network I/O
• When a client connects to a server it opens a socket channel on
server side and hands it over selector.
• Selector gets polled in a loop. It will wake up whenever there are
connections ready with data to be read or write.
• Long – living connections , once established it will be used to
read/write data until client closed or an exception occurs.
Page 7 © Hortonworks Inc. 2014
Kafka Security – SSL
• Kafka networking
Page 8 © Hortonworks Inc. 2014
Kafka Security – SSL
• Kafka SSL / SASL requirements
• No User-level API changes to clients
• Retain length-encoded Kafka protocols
• Client must authenticate before sending/receiving requests
• Kafka Channel
• Instead of using socket channel, we added KafkaChannel
which consists a TransportLayer, Authenticator.
Page 9 © Hortonworks Inc. 2014
Kafka Security – SSL
• TransportLayer
• Handles network level byte transfers
• PlaintextTransportLayer
• SSLTransportLayer
• Authenticator
• A pluggable interface for authentication implementations
• SaslAuthenticator – Provides SASL handshake and
authenticated user.
Page 10 © Hortonworks Inc. 2014
Kafka Security – SSL
KafkaChannel
TransportLayer
Authenticator
Kafka Server
handshake
authenticate
Page 11 © Hortonworks Inc. 2014
Kafka Security – SSL
• SSL - Handshake
• Kafka Server configures with Keystore and Truststore
• Kafka Client also needs a truststore with Kafka Server
certificate added to the truststore.
• Keystore configuration on client side is optional unless user wants
client side authentication.
Page 12 © Hortonworks Inc. 2014
Kafka Security – SSL
• KafkaChannel
• Before write or read application data , checks if the
channel.ready()
• A channel is ready if its established a connection and
authenticated. No-OP of PlaintextTransportLayer
• If a channel is not ready it goes through channel.prepare()
which internally calls transportLayer.handshake()
Page 13 © Hortonworks Inc. 2014
Kafka Security – SSL
• SSLTransportLayer
• Before sending any application data, both client and server
needs to go though SSL handshake
• SSLTransportLayer uses SSLEngine to establish a non-
blocking handshake.
• SSLEngine provides a state machine to go through several
steps of SSLhandshake
Page 14 © Hortonworks Inc. 2014
Kafka Security – SSL
Page 15 © Hortonworks Inc. 2014
Kafka Security – SSL
• SSLTransportLayer
• SocketChannel read
• Returns encrypted data
• Decrypts the data and returns the length of the data from Kafka protocols
• SocketChannel Write
• Writes encrypted data onto channel
• Regular socketChannel returns length of the data written to socket.
• Incase of SSL since we encrypt the data we can’t return exact length written to
socket which will be more than actual data
• Its important to keep track length of data written to network. This signifies if we
successfully written data to the network or not and move on to next request.
Page 16 © Hortonworks Inc. 2014
Kafka Security – SSL
• Principal Builder
• SSLTransportLayer gives hostname as authenticated user
• X509Certificate has lot more information about a client
identity.
• PrincipalBuilder provides interface to plug in a custom
PrincipalBuilder that has access to X509Certificate and can
construct a user string out of it.
• Authenticator can use this custom principal to add ACLs
Page 17 © Hortonworks Inc. 2014
Kafka Security – SSL
• Performance Impact
• Decrease in throughput by 20%.
• Latency increased by 30 %
• KAFKA-2481 (Ben Stopford) has more details
Page 18 © Hortonworks Inc. 2014
Kafka Security – SSL
• listeners=SSL://host.name:port
• ssl.keystore.location
• ssl.keystore.password
• ssl.key.password
• ssl.truststore.location
• ssl.truststore.password
• security.inter.broker.protocol (optional)
Page 19 © Hortonworks Inc. 2014
SASL/ Kerberos
Page 20 © Hortonworks Inc. 2014
Kafka Security – SASL
• Simple Authentication and Security Layer, or SASL
• Provides flexibility in using Login Mechanisms
• One can use Kerberos , LDAP or simple passwords to authenticate.
• JAAS Login
• Before client & server can handshake , they need to authenticate with Kerberos or other Identity Provider.
• JAAS provides a pluggable way of providing user credentials. One can easily add LDAP or other mechanism just by changing a config file.
Page 21 © Hortonworks Inc. 2014
Kafka Security – SASL
• JAAS Config file
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
serviceName="kafka"
keyTab="/vagrant/keytabs/kafka1.keytab"
principal="kafka/[email protected]";
};
KafkaConfig {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
serviceName="kafka"
keyTab="/vagrant/keytabs/client1.keytab"
principal=”client/[email protected]";
};
Page 22 © Hortonworks Inc. 2014
Kafka Security – SASL
• SASL Authenticator• Uses configured login credentials of JAAS config.
• Non-blocking handshake to establish clients identity
• Once handshake established , Kerberos principal name will be the authenticated user.
• Can be layered with SSL for wire encryption or Plaintext incase of wire encryption not needed.• SASL can provide encryption but it has huge performance penalties
Page 23 © Hortonworks Inc. 2014
Kafka Security – SASL
Client Broker
Connection
Mechanism list
Selected Mechanism & sasl data
Evaluate and Response
Sasl data
Client Authenticated
Page 24 © Hortonworks Inc. 2014
Kafka Security – SASL
• Pass JAAS config file as jvm parameter
• -Djava.security.auth.login.config
Page 25 © Hortonworks Inc. 2014
Kafka Security – Resources
• SSL
• https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka
• SASL
• https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61326390
• Vagrant Setup
• SASL
• https://github.com/harshach/kafka-vagrant/tree/master/
Page 26 © Hortonworks Inc. 2014
Authorization
Page 27 © Hortonworks Inc. 2014
Authorizer
• Controls who can do what
• Pluggable
• Acl based approach
Page 28 © Hortonworks Inc. 2014
Acl
• Alice is Allowed to Read from Orders-topic from Host-1
Principal Permission Operation Resource Host
Alice Allow Read Orders Host-1
Page 29 © Hortonworks Inc. 2014
Principal
• PrincipalType:Name
• Supported types: User and Group
• Extensible so users can add their own types
• Wild Card User:*
Page 30 © Hortonworks Inc. 2014
Operation
• Read, Write, Create, Delete, Alter, Describe,
ClusterAction, All
• Each API as an Operation VS Classification that maps to
APIs.
Page 31 © Hortonworks Inc. 2014
Resource
• ResourceType:ResourceName
• Topic, Cluster and ConsumerGroup
• Wild card resource ResourceType:*
Page 32 © Hortonworks Inc. 2014
Permissions
• Allow and Deny
• Anyone without an explicit Allow ACL is denied
• Then why do we have Deny?
• Deny works as negation
• Deny takes precedence over Allow Acls
Page 33 © Hortonworks Inc. 2014
Hosts
• Why provide this granularity?
• Allows authorizer to provide firewall type security even in
non secure environment.
• * as Wild card.
Page 34 © Hortonworks Inc. 2014
Configuration
• Authorizer class
• Super users
• Authorizer properties
• Default behavior for resources with no ACLs
Page 35 © Hortonworks Inc. 2014
SimpleAclAuthorizer
• Out of box authorizer implementation.
• Stores all of its ACLs in zookeeper.
• In built ACL cache to avoid performance penalty.
• Provides authorizer audit log.
Page 38 © Hortonworks Inc. 2014
CLI
• Add, Remove and List acls
• Convenience options:
--producer and --consumer.
Page 39 © Hortonworks Inc. 2014
Ranger Policy
Page 40 © Hortonworks Inc. 2014
Ranger Auditing
Page 41 © Hortonworks Inc. 2014
Ranger ACL management Audit
Page 42 © Hortonworks Inc. 2014
Unsecure zookeeper
Page 43 © Hortonworks Inc. 2014
Zookeeper
• Kafka’s metadata store
• Has its own security mechanism that supports SASL and
MD5-DIGEST for establishing identity and ACL based
authorization
• Create , Delete directly interacts with zookeeper
Page 44 © Hortonworks Inc. 2014
Securing zookeeper
• Acl on zk nodes: user:cdrwa
• Zookeeper.set.acl
• ZkSecurityMigrator script
• Credit where its due: Flavio Junqueira
Page 45 © Hortonworks Inc. 2014
Client JAAS
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
serviceName="zookeeper"
keyTab="/vagrant/keytabs/kafka.keytab"
principal="kafka/[email protected]";
};
Page 46 © Hortonworks Inc. 2014
Future
• KIP-4: Move everything to server side, no direct
interactions with zookeeper
• Group Support (PR already available)
• Pluggable Auditor
Page 47 © Hortonworks Inc. 2014
Summary
• SSL for wire encryption
• Sasl for authentication
• Authorization
• Secure Zookeeper
Thanks to the community for participation.