java zone 2015 how to make life with kafka easier
TRANSCRIPT
![Page 1: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/1.jpg)
PubSub++How to make your life with Kafka easier
Krzysztof Dębski
@DebskiChris
JavaZone 2015
![Page 2: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/2.jpg)
Who am I
@DebskiChris
http://hermes.allegro.tech
![Page 3: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/3.jpg)
Allegro Group
500+ people in IT
50+ independent teams
16 years on market
2 years after technical revolution
![Page 4: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/4.jpg)
Kafka as a backbone
![Page 5: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/5.jpg)
Kafka
![Page 6: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/6.jpg)
Hermes
![Page 7: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/7.jpg)
Kafka data
![Page 8: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/8.jpg)
Partitioning
Round robin partitioning (default)
Key based partitioning
![Page 9: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/9.jpg)
Performance issues
![Page 10: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/10.jpg)
Rebalancing leaders
Broker 1
P1 P0
Broker 2
P2 P1
Broker 3
P0 P2
Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000
Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3
![Page 11: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/11.jpg)
Rebalancing leaders
Broker 1
P1 P0
Broker 2
P2 P1
Broker 3
P0 P2
Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000
Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3
Brokers that should have partition copies
![Page 12: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/12.jpg)
Rebalancing leaders
Broker 1
P1 P0
Broker 2
P2 P1
Broker 3
P0 P2
Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000
Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3
In Sync Replicas
![Page 13: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/13.jpg)
Rebalancing leaders
Broker 1
P1 P0
Broker 2
P2 P1
Broker 3
P0 P2
Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000
Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3
Leader broker ID
![Page 14: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/14.jpg)
Rebalancing leaders
Broker 1
P1 P0
Broker 2
P2 P1
Broker 3
P0 P2
Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000
Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3
![Page 15: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/15.jpg)
Rebalancing leaders
Broker 1
P1 P0
Broker 2
P2 P1
Broker 3
P0 P2
Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000
Topic: test Partition: 0 Leader: 1 Replicas: 3, 1 ISR: 1Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2
![Page 16: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/16.jpg)
Rebalancing leaders
Broker 1
P1 P0
Broker 2
P2 P1
Broker 3
P0 P2
Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000
Topic: test Partition: 0 Leader: 1 Replicas: 3, 1 ISR: 1, 3Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3
![Page 17: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/17.jpg)
Lost events
![Page 18: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/18.jpg)
ACK levels
0 - don’t wait for response from the leader
1 - only the leader has to respond
-1 - all replicas must be in sync
Spee
d
Safe
ty
![Page 19: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/19.jpg)
Event identification
![Page 20: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/20.jpg)
Lost events
![Page 21: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/21.jpg)
Lost events
ERROR [Replica Manager on Broker 2]: Error when processing fetch request for partition [test,1] offset 10000 from consumer with correlation id 0. Possible cause:
Request for offset 10000 but we only have log segments in the range 8000 to 9000. (kafka.server.ReplicaManager)
![Page 22: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/22.jpg)
Lost events
Broker 1 Broker 2
Producer
ACK = 1
Replication factor = 1
replica.lag.max.messages = 2000
commited offset = 10000 commited offset = 9000
Zookeeper
![Page 23: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/23.jpg)
Lost events
Broker 1 Broker 2
Producer
ACK = 1
Replication factor = 1
replica.lag.max.messages = 2000
commited offset = 10000 commited offset = 9000
Zookeeper
![Page 24: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/24.jpg)
Lost events
Broker 1 Broker 2
Producer
ACK = 1
Replication factor = 1
replica.lag.max.messages = 2000
commited offset = 10000 commited offset = 9000
Zookeepercommited offset = 9000
![Page 25: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/25.jpg)
Slow responses
![Page 26: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/26.jpg)
Slow responses
75%
99%
99,9%
resp
onse
tim
e
![Page 27: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/27.jpg)
Slow responses
mes
sage
siz
e
75%
99%
99,9%
Is response time correlated to message size?
![Page 28: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/28.jpg)
Slow responsesre
spon
se t
ime
75%
99%
99,9%
Same distribution for fixed message size.
![Page 29: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/29.jpg)
Slow responses
resp
onse
tim
e
75%
99%
99,9%
Hermes overhead is just about 1 ms.
![Page 30: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/30.jpg)
Kafka
kernel 3.2.x
![Page 31: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/31.jpg)
Kafka
kernel 3.2.x
![Page 32: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/32.jpg)
Kafka
kernel 3.2.x kernel >= 3.8.x
![Page 33: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/33.jpg)
Normal operation
![Page 34: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/34.jpg)
Slow responses
![Page 35: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/35.jpg)
Message size
![Page 36: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/36.jpg)
Optimize message sizem
essa
ge s
ize
99,9%
all topics
99,9%
biggest topic
![Page 37: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/37.jpg)
Optimize message size
JSON human readablebig memory and network footprintpoor support for Hadoop
![Page 38: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/38.jpg)
Optimize message size
JSONSnappy
ERROR Error when sending message to topic t3 with key: 4 bytes, value: 100
bytes with error: The server experienced an unexpected error when
processing the request (org.apache.kafka.clients.producer.internals.
ErrorLoggingCallback)
java: target/snappy-1.1.1/snappy.cc:423: char* snappy::internal::
CompressFragment(const char*, size_t, char*, snappy::uint16*, int): Assertion
`0 == memcmp(base, candidate, matched)' failed.
errors on publishing large amount of messages
![Page 39: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/39.jpg)
Optimize message size
JSONSnappyLz4
failed on distributed data
com
pres
sion
rat
io
single
topic
multiple
topics
![Page 40: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/40.jpg)
Optimize message size
JSONSnappyLz4Avro
small network footprintHadoop friendlyeasy schema verification
![Page 41: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/41.jpg)
Improvements
![Page 42: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/42.jpg)
Multi data center
![Page 43: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/43.jpg)
Consumer backoff
You can’t have exactly one delivery
http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/
![Page 44: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/44.jpg)
Kafka offsets
<=0.8.1 - Zookeeper
>=0.8.2 - Zookeeper or Kafka
>=0.9(?) - Kafka
![Page 45: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/45.jpg)
Kafka Offset Monitor
![Page 46: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/46.jpg)
Manage your topics
![Page 47: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/47.jpg)
Improved security
Authentication and authorization interfaces provided
By Default:
You can create any topic in your group
You can publish everywhere (in progress)
Group owner defines subscriptions
![Page 48: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/48.jpg)
Improved offset management
![Page 49: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/49.jpg)
Improved offset management
![Page 50: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/50.jpg)
Improved offset management
![Page 51: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/51.jpg)
Improved offset management
![Page 52: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/52.jpg)
Turn back the time
PUT /groups/{group}/topics/{topic}/subscriptions/{subscription}/retransmission -8h
![Page 53: Java zone 2015 How to make life with kafka easier](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed9d6c1a28ab496e8b46c1/html5/thumbnails/53.jpg)
Blog: http://allegro.tech
Twitter: @allegrotechblog
Twitter: @debskichris