kafka 0.9, things you should know

©2016 LinkedIn Corporation. All Rights Reserved.

Kafka 0.9, Things you should know

©2016 LinkedIn Corporation. All Rights Reserved.

Ratish RavindranSite Reliability EngineerLinkedIn, Data Infrastructure Streaming

©2016 LinkedIn Corporation. All Rights Reserved. 3

Agenda

Security Kafka Connect User defined quota New consumer Notable improvements and fixes Upgrading from kafka 0.8 Kafka 0.10 - highlights

©2016 LinkedIn Corporation. All Rights Reserved. 4

Security

Why ? Multitenant cluster Multiple clusters Network ACLs

©2016 LinkedIn Corporation. All Rights Reserved. 5

Security

Authentication Kerberos TLS

Unix like permission "Principal P is [Allowed/Denied] Operation O From Host H On Resource R"bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --allow-principal User:Alice --allow-host 198.51.100.0 --allow-host 198.51.100.1 --operation Read --operation Write --topic Test-topic

Encryption

©2016 LinkedIn Corporation. All Rights Reserved. 6

Kafka Connect

Why ? Multiple tools for importing and exporting High engineering and operational overhead Some tools are poor fit for job

©2016 LinkedIn Corporation. All Rights Reserved. 7

Kafka Connect

Kafka

Datasource

C

Datasource

B

Datasource

A

Datasink 3

Datasink 2

Datasink 1T1

T2

T3 T6

T5

T4

©2016 LinkedIn Corporation. All Rights Reserved. 8

Kafka Connect

Kafka

Datasource

C

Datasource

B

Datasource

A

Datasink 3

Datasink 2

Datasink 1

Kaf

ka C

onne

ct

Kaf

ka C

onne

ct

©2016 LinkedIn Corporation. All Rights Reserved. 9

Kafka Connect

Key Properties: Broad copying by default

Streaming and batch

Scales to application

Focus on copying data only

Parallel

Connector API

©2016 LinkedIn Corporation. All Rights Reserved. 10

Kafka Connect

Advantages : Fault tolerance

Partitioning

Offset management

Delivery semantics

Operations

Monitoring

©2016 LinkedIn Corporation. All Rights Reserved. 11

User defined quota

Why ? High reads High writes SLAs

©2016 LinkedIn Corporation. All Rights Reserved. 12

User defined quota

Single large cluster Producer side (quota.producer.default) Consumer side (quota.consumer.default) Per client , Per broker Quota override ./bin/kafka-config.sh --alter --add-config ‘producer_byte_rate=1048576,consumer_byte_rate=1048576’ --entity-type clients --entity-name TestTopic

--zookeeper localhost:2181

©2016 LinkedIn Corporation. All Rights Reserved. 13

New consumer

Motivation : Thin consumer client Central co-ordination Allow manual partition assignment Allow manual offset management Invocation of user specified callback on rebalance Non blocking consumer APIs

©2016 LinkedIn Corporation. All Rights Reserved. 14

New consumer

Features: Group management protocol

Consumer

©2016 LinkedIn Corporation. All Rights Reserved. 15

New consumer

State Diagram - consumer

©2016 LinkedIn Corporation. All Rights Reserved. 16

New consumer

Features: Group management protocol

Consumer Co-ordinator

©2016 LinkedIn Corporation. All Rights Reserved. 17

New consumer

State Diagram – Co-ordinator

©2016 LinkedIn Corporation. All Rights Reserved. 18

New consumer

Features: Group management protocol

Consumer Co-ordinator

Failure detection protocol

©2016 LinkedIn Corporation. All Rights Reserved. 19

New consumer

Interesting scenarios: Co-ordinator failover/connection loss Partition changes for subscribed topics Offset commit during rebalance Hearbeats during rebalance Slow consumers

©2016 LinkedIn Corporation. All Rights Reserved. 20

Notable improvements and fixes

Automated replica lag tuning (replica.lag.time.max.ms) New purgatory design – low memory overhead Auto-assign node ids No data loss in Mirror Maker – unclean shutdown Log compaction for compressed topics Handling of corrupt index files

©2016 LinkedIn Corporation. All Rights Reserved. 21

Upgrading from kafka 0.8

inter.broker.protocol.version=0.8.2.x Update code and restart inter.broker.protocol.version=0.9.0.0 Restart brokers again

©2016 LinkedIn Corporation. All Rights Reserved. 22

Potential Breaking Changes: Java 1.6 and Scala 2.9 are not supported Broker IDs > 1000 ( reserved.broker.max.id and

broker.id.generation.enable ) replica.lag.max.messages removed replica.lag.time.max.ms No compaction for topics without key

Upgrading from kafka 0.8

©2016 LinkedIn Corporation. All Rights Reserved. 23

Potential Breaking Changes contd…. Changes in default JVM options

Upgrading from kafka 0.8

©2016 LinkedIn Corporation. All Rights Reserved. 24

Why not kafka 0.10 ?

©2016 LinkedIn Corporation. All Rights Reserved. 25

Kafka 0.10 - Highlights

Kafka Streams Rack Awareness Timestamps in messages SASL improvements Kafka consumer max record Protocol version improvements

©2016 LinkedIn Corporation. All Rights Reserved. 26

References

http://kafka.apache.org/090/documentation.html

http://www.confluent.io/blog/apache-kafka-0.9-is-released

http://www.confluent.io/blog/announcing-apache-kafka-0.10-and-confluent-platform-3.0

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design

©2016 LinkedIn Corporation. All Rights Reserved.