nyc kafka meetup 2015 - when bad things happen to good kafka clusters

16
When Bad Things Happen to Good Kafka Clusters True stories that actually happened to production Kafka clusters As told by Gwen Shapira, System Architect @gwenshap 1

Upload: gwen-chen-shapira

Post on 15-Apr-2017

2.315 views

Category:

Data & Analytics


0 download

TRANSCRIPT

Page 1: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

1

When Bad Things Happen toGood Kafka Clusters

True stories that actually happened to production Kafka clustersAs told by

Gwen Shapira, System Architect@gwenshap

Page 2: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

2

DisclaimerI am talking about other people’s systemsNot yours.I am sure you had perfectly good reasons to configure your system the way you did. This is not personal criticismJust some stories and few lessons we learned the hard way

Page 3: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

3

POCs are super easyIts time to go production

Page 4: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

4

We keep our data in/tmp/logs

What can possible go wrong?

Page 5: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

5

Replication-factor of 3 is way too much

Page 6: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

6

__consumer_offsets topic?

Never heard of it, so its probably ok to delete.

Page 7: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

7

Page 8: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

8

What’s wrong with running Kafka 0.7?

Page 9: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

9

Remember that time when…We accidentally lost all our data?

Page 10: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

10

We added new partitions…And immediately ran out of memory

Page 11: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

11

We wanted to lookup records by timeThe smaller the segments, the more accurate the lookups

So we created 10k segments.

Page 12: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

12

We need REALLY LARGE messages

Page 13: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

13

We just serialize JSON and throw it into a topic.It’s easy.The consumers will figure something out.

Page 14: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

14

Log4J is a great way to reliably send data to Kafka

Page 15: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

15

Keep your Kafka safe!“When it absolutely, positively has to be there:

Reliability guarantees in Apache Kafka”

Wednesday, 11:20am, Room 3D

Page 16: Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

16

Thank you

Visit Confluent in booth #929Books, Kafka t-shirts & stickers, and more…

Gwen Shapira | [email protected] | @gwenshap