secure kafka at salesforce.com

14
Secure Kafka at Salesforce.com Rajasekar Elango - Lead Developer

Upload: rajasekar-elango

Post on 27-Jan-2015

128 views

Category:

Engineering


2 download

DESCRIPTION

Presented at Kafka meetup 2014

TRANSCRIPT

Page 1: Secure Kafka at Salesforce.com

Secure Kafka at Salesforce.com Rajasekar Elango - Lead Developer

Page 2: Secure Kafka at Salesforce.com

What I do?

Work for Monitoring and Management Team We build tools for monitoring health and performance of salesforce.com infrastructure. Tools are used by Site Reliability and R&D development for troubleshooting, performance analysis, etc.

Page 3: Secure Kafka at Salesforce.com

Why Kafka?

We have application servers grouped into multiple clusters and distributed across multiple datacenters.

Build scalable, near real time monitoring framework that collects data from all production datacenters and pushes it to secure DMZ datacenter for aggregation and reporting.

Monitoring data we ship are JMX Metrics, System metrics (cpu, load, memory) from application servers, custom database metrics from database nodes.

Page 4: Secure Kafka at Salesforce.com

Architecture

App Servers

Cluster

Prod DC

App Servers

App Servers

Graphite DMZ

Kafka

Cluster Cluster

MM

Kafka

MM Kafka Kafka

Kafka

MM Kafka

Prod DC Prod DC

Page 5: Secure Kafka at Salesforce.com

Architecture

Zookeeper x 3

Broker x 5

Rest Interfa

ce

Graphite Consumer

Graphite

JMX Metrics Producer

System Metrics Producer

DB Metrics Producer

Mirror maker

x2

Production DMZ

Zookeeper x 3

Broker x 5

Page 6: Secure Kafka at Salesforce.com

Components

Rest Interface for abstracting producers.

AVRO for data format specification and serialization.

Producers - JMX Metric producer, collectd for system metrics, database metric producers.

Consumers - Graphite Consumer.

MirrorMaker - for cross datacenter replication.

Page 7: Secure Kafka at Salesforce.com

Secure Kafka Implementation

We wanted to secure traffic across datacenter to prevent malicious client eavesdropping data

Implemented SSL/TLS Mutual Auth between broker and producer/

consumer to add encryption and authentication

SSL Based socket channel based on JSSE doc

Secure mode can be toggled on/off by secure=true|false property in server.properties.

Broker registers secure property in zookeeper.

Page 8: Secure Kafka at Salesforce.com

Secure Kafka Configuration

server.properties

secure=true

security.config.file=config/server.security.properties

producer.properties & consumer.properties

security.config.file=config/client.security.properties

Page 9: Secure Kafka at Salesforce.com

Secure Kafka Configuration

server.security.properties

want.client.auth=true

need.client.auth=true

# Keystore file

keystore=<path to server keystore>

keystorePwd=<keystore password>

keyPwd=<key password>

# Truststore file

truststore=<path to server truststore>

truststorePwd=<truststore password>

Page 10: Secure Kafka at Salesforce.com

Secure Kafka Configuration

client.security.properties

# Keystore file

keystore=<path to client keystore file>

keystorePwd=<keystore password>

keyPwd=<key password>

# Truststore file

truststore=<path to client truststore file>

truststorePwd=<trust store password>

Page 11: Secure Kafka at Salesforce.com

Scripts

Producer

bin/kafka-console-producer.sh --broker-list localhost:9092:true --security.config.file config/client.security.properties --topic test

Consumer

bin/kafka-console-consumer.sh --topic test --zookeeper localhost:2181 --from-beginning --security.config.file config/client.security.properties

Page 12: Secure Kafka at Salesforce.com

Limitations

Doesn’t provide authorization.

Doesn’t use secure communication with Zookeeper. We implemented secure features branched off from older snapshot version of kafka 0.8 release.

Page 13: Secure Kafka at Salesforce.com

Demo

bin/zookeeper-server-start.sh config/zookeeper.properties

bin/kafka-server-start.sh config/server.properties

bin/kafka-console-producer.sh --broker-list relango-ltmr.home:9092:true --topic test < messages.txt

bin/kafka-console-consumer.sh --topic test --zookeeper localhost:2181 --from-beginning

Page 14: Secure Kafka at Salesforce.com