building a replicated logging system with apache kafka

78

Click here to load reader

Upload: guozhang-wang

Post on 14-Apr-2017

1.491 views

Category:

Software


1 download

TRANSCRIPT

Page 1: Building a Replicated Logging System with Apache Kafka

Building a Replicated Logging System with Apache Kafka

Guozhang Wang, Joel Koshy, Sriram Subramanian, Kartik Paramasivam Mammad Zadeh, Neha Narkhede, Jun Rao, Jay

Kreps, Joe Stein

Page 2: Building a Replicated Logging System with Apache Kafka

We All Love Logs!

Page 3: Building a Replicated Logging System with Apache Kafka
Page 4: Building a Replicated Logging System with Apache Kafka
Page 5: Building a Replicated Logging System with Apache Kafka
Page 6: Building a Replicated Logging System with Apache Kafka

Apache Kafka

• A distributed messaging system

..that store messages as a log!

Page 7: Building a Replicated Logging System with Apache Kafka

Example: LinkedIn back in 2010Point-to-Point Pipelines

What We Want: A Centralized Data Pipeline

Page 8: Building a Replicated Logging System with Apache Kafka

Log-centric Data Flow• Logical Ordering

• Persistent Buffering

• “Source-of-Truth”

Page 9: Building a Replicated Logging System with Apache Kafka

Store Messages as a Log

4 5 5 7 8 9 10 11 12...

Producer Write

Consumer1 Reads (offset 7)

Consumer2 Reads (offset 10)

Messages3

Page 10: Building a Replicated Logging System with Apache Kafka

Partition the Log across Machines

Topic 1

Topic 2

Partitions

Producers

Producers

Consumers

Consumers

Brokers

Page 11: Building a Replicated Logging System with Apache Kafka

Apache Kafka

Example: Kafka at LinkedIn

Page 12: Building a Replicated Logging System with Apache Kafka

“Source-of-Truth” should not be lost even when..

Page 13: Building a Replicated Logging System with Apache Kafka

Replicas and Layout

Logs

Broker-1

topic1-part1

topic1-part3

topic1-part2

Logs

topic1-part2

topic1-part1

topic1-part3

Logs

topic1-part3

topic1-part2

topic1-part1

Broker-2 Broker-3

Page 14: Building a Replicated Logging System with Apache Kafka

Consensus for Log Replication

Logs

Broker-1

Logs Logs

Broker-2 Broker-3

Write

Consensus Protocol

Consensus Protocol

Page 15: Building a Replicated Logging System with Apache Kafka

Key Idea

Separate membership configuration from data replication

Page 16: Building a Replicated Logging System with Apache Kafka

Primary-backup Replication

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write

Page 17: Building a Replicated Logging System with Apache Kafka

Conventional Quorum Commits

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write

Page 18: Building a Replicated Logging System with Apache Kafka

Conventional Quorum Commits

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write

Page 19: Building a Replicated Logging System with Apache Kafka

Conventional Quorum Commits

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Page 20: Building a Replicated Logging System with Apache Kafka

Conventional Quorum Commits

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Page 21: Building a Replicated Logging System with Apache Kafka

• Leader maintains in-sync-replicas (ISR)• Failed / slow follower => drop from ISR• Caught-up follower => re-join ISR

• Producer specifies required ACK based on ISR

Configurable ISR Commits

Page 22: Building a Replicated Logging System with Apache Kafka

Example: ACK with all ISRs

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2, 3}

Page 23: Building a Replicated Logging System with Apache Kafka

Example: ACK with all ISRs

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2, 3}

Page 24: Building a Replicated Logging System with Apache Kafka

Example: ACK with all ISRs

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2, 3}

Page 25: Building a Replicated Logging System with Apache Kafka

Example: ACK with all ISRs

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2, 3}

Page 26: Building a Replicated Logging System with Apache Kafka

Example: ACK with all ISRs

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2, 3}

Page 27: Building a Replicated Logging System with Apache Kafka

Example: ACK with all ISRs

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2, 3}

Page 28: Building a Replicated Logging System with Apache Kafka

Example: ACK with all ISRs

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2, 3}

Page 29: Building a Replicated Logging System with Apache Kafka

Example: ACK with Leader-only

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“leader”)

ISR {1, 2, 3}

Page 30: Building a Replicated Logging System with Apache Kafka

Example: ACK with Leader-only

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“leader”)

ISR {1, 2, 3}

Page 31: Building a Replicated Logging System with Apache Kafka

Example: ACK with Leader-only

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“leader”)

ISR {1, 2, 3}

Page 32: Building a Replicated Logging System with Apache Kafka

Example: ACK with Leader-only

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“leader”)

ISR {1, 2, 3}

Page 33: Building a Replicated Logging System with Apache Kafka

Example: ACK with Leader-only

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“leader”)

ISR {1, 2, 3}

Page 34: Building a Replicated Logging System with Apache Kafka

Example: ACK with Leader-only

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“leader”)

ISR {1, 2, 3}

Page 35: Building a Replicated Logging System with Apache Kafka

Example: Slow Follower

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2, 3}

Page 36: Building a Replicated Logging System with Apache Kafka

Example: Slow Follower

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2, 3}

Page 37: Building a Replicated Logging System with Apache Kafka

Example: Slow Follower

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2, 3}

Page 38: Building a Replicated Logging System with Apache Kafka

Example: Slow Follower

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2, 3}

Page 39: Building a Replicated Logging System with Apache Kafka

Example: Slow Follower

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2, 3}

Page 40: Building a Replicated Logging System with Apache Kafka

Example: Slow Follower

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2}

Page 41: Building a Replicated Logging System with Apache Kafka

Example: Slow Follower

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

Write (ack=“all”)

ISR {1, 2}

Page 42: Building a Replicated Logging System with Apache Kafka

Configurable ISR CommitsACK mode Latency On Failures

“no" no network delay some data loss

“leader" 1 network roundtrip a few data loss

“all" ~2 network roundtrips no data loss

Page 43: Building a Replicated Logging System with Apache Kafka

• Use an embedded controller• Detect broker failure via ZooKeeper• Leader failure => elect new leader from ISR

• Leader and ISR persisted in Zookeeper• For Controller fail-over

Membership Management

Page 44: Building a Replicated Logging System with Apache Kafka

Example: Broker Failure

Logs

Broker-1 *

Logs Logs

Broker-2 Broker-3

ISR {1, 2}

Page 45: Building a Replicated Logging System with Apache Kafka

Example: Broker Failure

Logs

Broker-1

Logs Logs

Broker-2 Broker-3

Page 46: Building a Replicated Logging System with Apache Kafka

Example: Broker Failure

Logs

Broker-1

Logs Logs

Broker-2 Broker-3

Page 47: Building a Replicated Logging System with Apache Kafka

Example: Broker Failure

Logs

Broker-1

Logs Logs

Broker-2 Broker-3

Page 48: Building a Replicated Logging System with Apache Kafka

Example: Broker Failure

Logs

Broker-1

Logs Logs

Broker-2 Broker-3ISR {2}

Page 49: Building a Replicated Logging System with Apache Kafka

Example: Broker Failure

Logs

Broker-1

Logs Logs

Broker-2 Broker-3ISR {2}

Page 50: Building a Replicated Logging System with Apache Kafka

Example: Broker Failure

Logs

Broker-1

Logs Logs

Broker-2 Broker-3ISR {2}

Page 51: Building a Replicated Logging System with Apache Kafka

Example: Broker Failure

Logs

Broker-1

Logs Logs

Broker-2 * Broker-3

ISR {2}

Page 52: Building a Replicated Logging System with Apache Kafka

Example: Broker Failure

Logs

Broker-1

Logs Logs

Broker-2 * Broker-3

ISR {2}

Page 53: Building a Replicated Logging System with Apache Kafka

Example: Broker Failure

Logs

Broker-1

Logs Logs

Broker-2 * Broker-3

ISR {2}

Page 54: Building a Replicated Logging System with Apache Kafka

Example: Broker Failure

Logs

Broker-1

Logs Logs

Broker-2 * Broker-3

ISR {2, 3}

Page 55: Building a Replicated Logging System with Apache Kafka

• Overview: Logs and Kafka

• Log Replication in Kafka

• Kafka Usage at LinkedIn

• Conclusion

Agenda

Page 56: Building a Replicated Logging System with Apache Kafka

Change Log Replication

Page 57: Building a Replicated Logging System with Apache Kafka

Apache Kafka

Example: Kafka at LinkedIn

Page 58: Building a Replicated Logging System with Apache Kafka

Example: Espresso

• A distributed document store

• Primary online data serving platform at LI

• Member profile, homepage, InMail, etc

[SIGMOD 2013]

Page 59: Building a Replicated Logging System with Apache Kafka

Old Espresso ReplicationData Center-1

Storage Node

Storage NodeMySQL

Replication

MySQL MySQL

Search Index

Hadoop …

…DatabusCross-DC Replicator

Data Center-1

Storage Node

Storage NodeMySQL

ReplicationMySQL MySQL

Search Index

Hadoop …

DatabusCross-DC Replicator

Page 60: Building a Replicated Logging System with Apache Kafka

Problems with MySQL Replication

Master Storage Node

P1

Slave Storage Node

P2 P3

P4 P5 P6

P1 P2 P3

P4 P5 P6

Binary Log Shipping

Page 61: Building a Replicated Logging System with Apache Kafka

Replicate Logs with KafkaStorage Node

Kafka Logs

P1

Storage Node

P2 P3

P4 P5 P6

P1 P2 P3

P4 P5 P6

Kafka Producer Kafka Consumer Kafka Consumer Kafka Producer

Page 62: Building a Replicated Logging System with Apache Kafka

Key-based Log Compaction

...

Partition Messages

Segment-3 Segment-4 Segment-6 *

Page 63: Building a Replicated Logging System with Apache Kafka

Key-based Log Compaction

d: 3 f: 8 b: 0 c: null...

Partition Messagesc: 3 a: 5 a: 6 a: 5 f: 9 ...

Segment-3 Segment-4

b: 2 d: 4a: 1

Page 64: Building a Replicated Logging System with Apache Kafka

Key-based Log Compaction

... d: 3 f: 8 b: 0 c: null a: 5 f: 9 ...

Segment-3 Segment-4

c: 3 a: 5 a: 6b: 2 d: 4a: 1 c: 3 a: 5 a: 6b: 2 d: 4a: 1 d: 3 f: 8 b: 0 a: 5 f: 9

New Segment

Partition Messages

Page 65: Building a Replicated Logging System with Apache Kafka

Key-based Log Compaction

... d: 3 f: 8 b: 0 c: null a: 5 f: 9 ...

Segment-3 Segment-4

c: 3 a: 5 a: 6b: 2 d: 4a: 1

c: 3 a: 6 d: 3 f: 8 b: 0

c: null a: 5 f: 9

New Segment

Partition Messages

Page 66: Building a Replicated Logging System with Apache Kafka

Key-based Log Compaction

... d: 3 f: 8 b: 0 c: null a: 5 f: 9 ...

Segment-3 Segment-4

c: 3 a: 5 a: 6b: 2 d: 4a: 1

d: 3 b: 0 a: 5 f: 9

New Segment

Partition Messages

Page 67: Building a Replicated Logging System with Apache Kafka

Key-based Log Compaction

... d: 3 f: 8 b: 0 c: null a: 5 f: 9 ...

Segment-3 Segment-4

c: 3 a: 5 a: 6b: 2 d: 4a: 1

d: 3 b: 0 a: 5 f: 9

New Segment

Partition Messages

Page 68: Building a Replicated Logging System with Apache Kafka

New Espresso ReplicationData Center-1

Storage Node

Storage Node

Storage Node

Kafka Logs

MySQL MySQL MySQL

Data Center-n

Storage Node

Storage Node

Storage Node

Kafka Logs

MySQL MySQL MySQL

Kafka MirrorMaker

Search Index

Hadoop …

Search Index

Hadoop …

* In Progress

Page 69: Building a Replicated Logging System with Apache Kafka

Stream Processing

Page 70: Building a Replicated Logging System with Apache Kafka

Apache Kafka

Example: Kafka at LinkedIn

Page 71: Building a Replicated Logging System with Apache Kafka

• Data flow streaming on Kafka and YARN• Stateful processing

• Re-processing

• Failure Recovery

Example: Samza [CIDR 2015]

Page 72: Building a Replicated Logging System with Apache Kafka

Kafka

Kafka

Samza

StateProcess

Protocol

StateProcess

Protocol

StateProcess

Protocol

Samza Processing

Page 73: Building a Replicated Logging System with Apache Kafka

Kafka

Kafka

Samza

StateProcess

Protocol

StateProcess

Protocol

StateProcess

Protocol

Samza Processing Kafka Changelog

Page 74: Building a Replicated Logging System with Apache Kafka

Kafka

Kafka

Samza

StateProcess

Protocol

StateProcess

Protocol

StateProcess

Protocol

Samza Processing Kafka Changlog

Page 75: Building a Replicated Logging System with Apache Kafka

Kafka

Kafka Samza

StateProcess

Protocol

StateProcess

Protocol

StateProcess

Protocol

Samza Processing Kafka Changlog

StateProcess

Protocol

Page 76: Building a Replicated Logging System with Apache Kafka

Take-aways• Log-centric data flow helps scaling your

systems

• Kafka: replicated log streams for real-time platforms

Page 77: Building a Replicated Logging System with Apache Kafka

We are Hiring

Page 78: Building a Replicated Logging System with Apache Kafka

Take-aways• Log-centric data flow helps scaling your

systems

• Kafka: replicated log streams for real-time platforms

THANKS!