secure kafka at salesforce.com
DESCRIPTION
Presented at Kafka meetup 2014TRANSCRIPT
Secure Kafka at Salesforce.com Rajasekar Elango - Lead Developer
What I do?
Work for Monitoring and Management Team We build tools for monitoring health and performance of salesforce.com infrastructure. Tools are used by Site Reliability and R&D development for troubleshooting, performance analysis, etc.
Why Kafka?
We have application servers grouped into multiple clusters and distributed across multiple datacenters.
Build scalable, near real time monitoring framework that collects data from all production datacenters and pushes it to secure DMZ datacenter for aggregation and reporting.
Monitoring data we ship are JMX Metrics, System metrics (cpu, load, memory) from application servers, custom database metrics from database nodes.
Architecture
App Servers
Cluster
Prod DC
App Servers
App Servers
Graphite DMZ
Kafka
Cluster Cluster
MM
Kafka
MM Kafka Kafka
Kafka
MM Kafka
Prod DC Prod DC
Architecture
Zookeeper x 3
Broker x 5
Rest Interfa
ce
Graphite Consumer
Graphite
JMX Metrics Producer
System Metrics Producer
DB Metrics Producer
Mirror maker
x2
Production DMZ
Zookeeper x 3
Broker x 5
Components
Rest Interface for abstracting producers.
AVRO for data format specification and serialization.
Producers - JMX Metric producer, collectd for system metrics, database metric producers.
Consumers - Graphite Consumer.
MirrorMaker - for cross datacenter replication.
Secure Kafka Implementation
We wanted to secure traffic across datacenter to prevent malicious client eavesdropping data
Implemented SSL/TLS Mutual Auth between broker and producer/
consumer to add encryption and authentication
SSL Based socket channel based on JSSE doc
Secure mode can be toggled on/off by secure=true|false property in server.properties.
Broker registers secure property in zookeeper.
Secure Kafka Configuration
server.properties
secure=true
security.config.file=config/server.security.properties
producer.properties & consumer.properties
security.config.file=config/client.security.properties
Secure Kafka Configuration
server.security.properties
want.client.auth=true
need.client.auth=true
# Keystore file
keystore=<path to server keystore>
keystorePwd=<keystore password>
keyPwd=<key password>
# Truststore file
truststore=<path to server truststore>
truststorePwd=<truststore password>
Secure Kafka Configuration
client.security.properties
# Keystore file
keystore=<path to client keystore file>
keystorePwd=<keystore password>
keyPwd=<key password>
# Truststore file
truststore=<path to client truststore file>
truststorePwd=<trust store password>
Scripts
Producer
bin/kafka-console-producer.sh --broker-list localhost:9092:true --security.config.file config/client.security.properties --topic test
Consumer
bin/kafka-console-consumer.sh --topic test --zookeeper localhost:2181 --from-beginning --security.config.file config/client.security.properties
Limitations
Doesn’t provide authorization.
Doesn’t use secure communication with Zookeeper. We implemented secure features branched off from older snapshot version of kafka 0.8 release.
Demo
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
bin/kafka-console-producer.sh --broker-list relango-ltmr.home:9092:true --topic test < messages.txt
bin/kafka-console-consumer.sh --topic test --zookeeper localhost:2181 --from-beginning