kafka 0.9, things you should know

©2016 LinkedIn Corporation. All Rights Reserved.

Kafka 0.9, Things you should know


Ratish RavindranSite Reliability EngineerLinkedIn, Data Infrastructure Streaming

©2016 LinkedIn Corporation. All Rights Reserved. 3

Agenda

Security Kafka Connect User defined quota New consumer Notable improvements and fixes Upgrading from kafka 0.8 Kafka 0.10 - highlights


Security

Why ? Multitenant cluster Multiple clusters Network ACLs


Security

Authentication Kerberos TLS

Unix like permission "Principal P is [Allowed/Denied] Operation O From Host H On Resource R"bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --allow-principal User:Alice --allow-host 198.51.100.0 --allow-host 198.51.100.1 --operation Read --operation Write --topic Test-topic

Encryption


Kafka Connect

Why ? Multiple tools for importing and exporting High engineering and operational overhead Some tools are poor fit for job


Kafka Connect

Kafka

Datasource

C

Datasource

B

Datasource

A

Datasink 3

Datasink 2

Datasink 1T1

T2

T3 T6

T5

T4


Kafka Connect

Kafka

Datasource

C

Datasource

B

Datasource

A

Datasink 3

Datasink 2

Datasink 1

Kaf

ka C

onne

ct

Kaf

ka C

onne

ct


Kafka Connect

Key Properties: Broad copying by default

Streaming and batch

Scales to application

Focus on copying data only

Parallel

Connector API


Kafka Connect

Advantages : Fault tolerance

Partitioning

Offset management

Delivery semantics

Operations

Monitoring


User defined quota

Why ? High reads High writes SLAs


User defined quota

Single large cluster Producer side (quota.producer.default) Consumer side (quota.consumer.default) Per client , Per broker Quota override ./bin/kafka-config.sh --alter --add-config ‘producer_byte_rate=1048576,consumer_byte_rate=1048576’ --entity-type clients --entity-name TestTopic

--zookeeper localhost:2181


New consumer

Motivation : Thin consumer client Central co-ordination Allow manual partition assignment Allow manual offset management Invocation of user specified callback on rebalance Non blocking consumer APIs


New consumer

Features: Group management protocol

Consumer


New consumer

State Diagram - consumer


New consumer


Consumer Co-ordinator


New consumer

State Diagram – Co-ordinator


New consumer


Consumer Co-ordinator

Failure detection protocol


New consumer

Interesting scenarios: Co-ordinator failover/connection loss Partition changes for subscribed topics Offset commit during rebalance Hearbeats during rebalance Slow consumers


Notable improvements and fixes

Automated replica lag tuning (replica.lag.time.max.ms) New purgatory design – low memory overhead Auto-assign node ids No data loss in Mirror Maker – unclean shutdown Log compaction for compressed topics Handling of corrupt index files


Upgrading from kafka 0.8

inter.broker.protocol.version=0.8.2.x Update code and restart inter.broker.protocol.version=0.9.0.0 Restart brokers again


Potential Breaking Changes: Java 1.6 and Scala 2.9 are not supported Broker IDs > 1000 ( reserved.broker.max.id and

broker.id.generation.enable ) replica.lag.max.messages removed replica.lag.time.max.ms No compaction for topics without key



Potential Breaking Changes contd…. Changes in default JVM options



Why not kafka 0.10 ?


Kafka 0.10 - Highlights

Kafka Streams Rack Awareness Timestamps in messages SASL improvements Kafka consumer max record Protocol version improvements


References

http://kafka.apache.org/090/documentation.html

http://www.confluent.io/blog/apache-kafka-0.9-is-released

http://www.confluent.io/blog/announcing-apache-kafka-0.10-and-confluent-platform-3.0

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design










kafka 0.9, things you should know

Technology