No data loss pipeline with apache kafka

Download No data loss pipeline with apache kafka

Post on 16-Apr-2017

8.805 views

Category:

Technology

1 download

TRANSCRIPT

No Data Loss Pipeline with Apache KafkaJiangjie (Becket) Qin @ LinkedIn

Data lossproducer.send(record) is called but record did not end up in consumer as expectedMessage reorderingsend(record1) is called before send(record2)record2 shows in broker before record1 doesmatters in cases like DB replicationData loss and message reordering

Kafka based data pipelineKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror MakerTodays Agenda:No data lossNo message reorderingMirror maker enhancementCustomized consumer rebalance listenerMessage handler

Synchronous send is safe but slow...

producer.send(record).get()

ProducerKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

Using asynchronous send with callback can be trickyproducer.send(record,callback)

ProducerKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

Producer can cause data loss whenblock.on.buffer.full=falseretries are exhaustedsending message without using acks=allProducerKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

Is this good enough?producer.send(record,callback)block.on.buffer.full=TRUEretries=Long.MAX_VALUEacks=allresend in callback when message send failed

ProducerKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

Message reordering might happen if:max.in.flight.requests.per.connection > 1, AND retries are enabled

ProducerKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror MakerKafka BrokerProducermessage 0message 1message 0 failedretry message 0Timeline

Message reordering might also happen if:producer is closed carelesslyclose producer in user thread, orclose without using close(0)ProducerRecordAccumulatorSender ThreadKafka BrokerTimeline1.msg 02.callback(msg 0)ack expt.User Threadclose prod.3.msg 1notify

What if we just stop producing on send failure? No retry.

close producer in the callback on errorclose producer with close(0) to prevent further sending after previous message send failedProducerRecordAccumulatorSender ThreadKafka BrokerTimeline1.msg 02.callback(msg 0)ack expt.User Threadclose(0)notify

To prevent data loss:block.on.buffer.full=TRUEretries=Long.MAX_VALUE (for some use cases)acks=allTo prevent reordering:max.in.flight.requests.per.connection=1close producer in callback with close(0) on send failureProducerKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

Not a perfect solution:Producer needs to be closed to guarantee message order. E.g. In mirror maker, one message send failure to a topic should not affect the whole pipeline.When producer is down, message in buffer will still be lostProducerKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

Correct producer setting is not enoughacks=all still can lose data when unclean leader election happens.Two replicas are needed at any time to guarantee data persistence.

Kafka BrokersKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

replication factor >= 3min.isr = 2Replication factor > min.isrIf replication factor = min.isr, partition will be offline when one replica is down

Kafka BrokersKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

Settings we use:replication factor = 3min.isr = 2unclean leader election disabled

Kafka BrokersKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

Consumer might lose message when offsets are committed carelessly. E.g. commit offsets before processing messages completelyDisable auto.offset.commitCommit offsets only after the messages are processedConsumerKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

Kafka based data pipelineKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror MakerTodays Agenda:No data lossNo message reorderingMirror maker enhancementCustomized consumer rebalance listenerMessage handler

Consume-then-produce patternOnly commit consumer offsets of messages acked by target clusterDefault to no-data-loss and no-reordering settingsMirror Maker EnhancementKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

Customized Consumer Rebalance ListenerCan be used to propagate topic change from source cluster to target cluster. E.g. partition number change, new topic creation.Mirror Maker EnhancementKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

Customized Message Handler, useful forpartition-to-partition mirrorfiltering out messagesmessage format conversionother simple message processingMirror Maker EnhancementKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

startup/shutdown accelerationparallelized startup and shutdown26 nodes cluster with 4 consumer each takes about 1 min to startup and shutdownMirror Maker EnhancementKafka Cluster(Colo 1)ProducerKafka Cluster(Colo 2)ConsumerMirror Maker

Q&A