apache kafka lightning talk

39
Q Apache Kafka Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Jeff Kunkle Nov 22, 2013 Tuesday, December 3, 13

Upload: jeff-kunkle

Post on 19-Jan-2015

701 views

Category:

Technology


3 download

DESCRIPTION

This short deck provides a high-level overview of how Apache Kafka works under the covers. It covers logs, topics partitions, consumer groups, and replication from a conceptual perspective.

TRANSCRIPT

Page 1: Apache Kafka Lightning Talk

Q

Apache KafkaQ

Q

QQ

Q

Q

Q

Q

Q

QQ

Q

Q

Q

Q

Q

QQ

Q

Q

Q

Q

Q

QQ

Q

QQ Q

Q

Q

Q

Q

QQ

Q

QQ

Q

QQ Q

Q

Q

Q

Jeff KunkleNov 22, 2013

Tuesday, December 3, 13

Page 2: Apache Kafka Lightning Talk

Q

What is Kafka?

Q

Q

QQ

Q

Q

Q

Q

Q

QQ

Q

Q

Q

Q

Q

QQ

QQ

Q

Q

Q

Q

QQ

Q

QQ Q

Q

Q

Q

Q

QQ

Q

QQ

Q

QQ Q

Q

Q

QTuesday, December 3, 13

Page 3: Apache Kafka Lightning Talk

Kafka is a distributed, partitioned, replicated commit log service.

Tuesday, December 3, 13

Page 4: Apache Kafka Lightning Talk

Tuesday, December 3, 13

Page 5: Apache Kafka Lightning Talk

?Tuesday, December 3, 13

Page 6: Apache Kafka Lightning Talk

... a non-JMS messaging system with a unique design.

Tuesday, December 3, 13

Page 7: Apache Kafka Lightning Talk

Key Concepts1 Topics, Logs, and Partitions2 Consumer Groups3 Replication

Tuesday, December 3, 13

Page 8: Apache Kafka Lightning Talk

Topics and Logs

Tuesday, December 3, 13

Page 9: Apache Kafka Lightning Talk

partition 0

partition 1

partition 2

Topics and Logs

Tuesday, December 3, 13

Page 10: Apache Kafka Lightning Talk

partition 0

partition 1

partition 2

old new

Topics and Logs

1 2 3 4 5 6 7log

1 2 3 4 5 6log

1 2 3 4 5log

Tuesday, December 3, 13

Page 11: Apache Kafka Lightning Talk

partition 0

partition 1

partition 2

writes

old new

Topics and Logs

1 2 3 4 5 6 7log

1 2 3 4 5 6log

1 2 3 4 5log

Tuesday, December 3, 13

Page 12: Apache Kafka Lightning Talk

1234567

partition 0

Pro

du

cers

123456

partition 1

12345

partition 2

P1

P2

Tuesday, December 3, 13

Page 13: Apache Kafka Lightning Talk

1 2

34567

partition 0

Pro

du

cers

1

23456

partition 1

1

2345

partition 2

P1

P2

Tuesday, December 3, 13

Page 14: Apache Kafka Lightning Talk

1 2

34567

partition 0

Pro

du

cers

1

23456

partition 1

1

2345

partition 2

P1

P2

bulk publishing

Tuesday, December 3, 13

Page 15: Apache Kafka Lightning Talk

1 2

34567

partition 0

Pro

du

cers

1

23456

partition 1

1 2 3 4

5

partition 2

P1

P2

bulk publishing

Tuesday, December 3, 13

Page 16: Apache Kafka Lightning Talk

1 2

34567

partition 0

Pro

du

cers

1

23456

partition 1

1 2 3 4

5

partition 2

P1

P2

parallel publishing

Tuesday, December 3, 13

Page 17: Apache Kafka Lightning Talk

1 2 3 4

567

partition 0

Pro

du

cers

1 2

3456

partition 1

1 2 3 4

5

partition 2

P1

P2

parallel publishing

Tuesday, December 3, 13

Page 18: Apache Kafka Lightning Talk

1 2 3 4 5 6 7partition 0

Pro

du

cers

1 2 3 4 5 6partition 1

1 2 3 4 5partition 2

P1

P2

Tuesday, December 3, 13

Page 19: Apache Kafka Lightning Talk

1 2 3 4 5

1 2 3 4 5 6

1 2 3 4 5 6 7 partition 0

partition 1

partition 2

Co

nsu

me

rs

1 2 3 4 5 6 7

1 2 3 4 5 6

1 2 3 4 5

consumer group 2

consumer group 1

C3

C2

C1

C4

Tuesday, December 3, 13

Page 20: Apache Kafka Lightning Talk

1 2 3 4 5

1 2 3 4 5 6

1

2 3 4 5 6 7 partition 0

partition 1

partition 2

Co

nsu

me

rs

1 2 3 4 5 6 7

1 2 3 4 5 6

1 2 3 4 5

consumer group 2

consumer group 1

C3

C2

C1

C4

Tuesday, December 3, 13

Page 21: Apache Kafka Lightning Talk

1 2 3 4 51

2 3 4 5 6

1

2 3 4 5 6 7 partition 0

partition 1

partition 2

Co

nsu

me

rs

1 2 3 4 5 6 7

12 3 4 5 6

1 2 3 4 5

consumer group 2

consumer group 1

C3

C2

C1

C4

Tuesday, December 3, 13

Page 22: Apache Kafka Lightning Talk

1 2 3 4 51

2 3 4 5 6

1

2 3 4 5 6 7 partition 0

partition 1

partition 2

Co

nsu

me

rs

1 2 3 4 5 6 7

12 3 4 5 6

1

2 3 4 5

consumer group 2

consumer group 1

C3

C2

C1

C4

Tuesday, December 3, 13

Page 23: Apache Kafka Lightning Talk

12 3 4 5123

4 5 6

12

3 4 5 6 7 partition 0

partition 1

partition 2

Co

nsu

me

rs

1

2

3 4 5 6 7

1

2

3

4 5 6

1

2

3 4 5

consumer group 2

consumer group 1

C3

C2

C1

C4

Tuesday, December 3, 13

Page 24: Apache Kafka Lightning Talk

12 3 4 5123

4 5 6

12

3 4 5 6 7 partition 0

partition 1

partition 2

Co

nsu

me

rs

1

2

3 4 5 6 7

1

2

3

4 5 6

1

2

3 4 5

consumer group 2

consumer group 1

C3

C2

C1

C4

bulk consumption

Tuesday, December 3, 13

Page 25: Apache Kafka Lightning Talk

12 3 4 5123

4 5 6

123456

7 partition 0

partition 1

partition 2

Co

nsu

me

rs

1

2

3456

7

1

2

3

4 5 6

1

2

3 4 5

consumer group 2

consumer group 1

C3

C2

C1

C4

bulk consumption

Tuesday, December 3, 13

Page 26: Apache Kafka Lightning Talk

12 3 4 5123

4 5 6

123456

7 partition 0

partition 1

partition 2

Co

nsu

me

rs

1

2

3456

7

1

2

3

4 5 6

1

2

3 4 5

consumer group 2

consumer group 1

C3

C2

C1

C4

parallel consumption

Tuesday, December 3, 13

Page 27: Apache Kafka Lightning Talk

123 4 512345

6

123456

7 partition 0

partition 1

partition 2

Co

nsu

me

rs

1

2

3456

7

1

2

345

6

1

23

4 5

consumer group 2

consumer group 1

C3

C2

C1

C4

parallel consumption

Tuesday, December 3, 13

Page 28: Apache Kafka Lightning Talk

123451234561234567

partition 0

partition 1

partition 2

Co

nsu

me

rs

1

2

3456

71

2

345

61

2345

consumer group 2

consumer group 1

C3

C2

C1

C4

Tuesday, December 3, 13

Page 29: Apache Kafka Lightning Talk

A1A2A3A4

A5

A6A7

server A

Replication

server Bserver C

B1B2B3B4

B5B6C1C2C3C4

C5

P2

P1

A master(all reads and writes)

B master(all reads and writes)

C master(all reads and writes)

Tuesday, December 3, 13

Page 30: Apache Kafka Lightning Talk

A1

A2A3A4

A5

A6A7

server A

Replication

server Bserver C

B1B2B3B4

B5B6C1C2C3C4

C5

P2

P1

A master(all reads and writes)

B master(all reads and writes)

C master(all reads and writes)

Tuesday, December 3, 13

Page 31: Apache Kafka Lightning Talk

A1

A2A3A4

A5

A6A7

server A

Replication

server Bserver C

B1B2B3B4

B5B6C1C2C3C4

C5

P2A1

P1

A master(all reads and writes)

B master(all reads and writes)

C master(all reads and writes)

Tuesday, December 3, 13

Page 32: Apache Kafka Lightning Talk

A1 A2

A3A4

A5

A6A7

server A

Replication

server Bserver C

B1B2B3B4

B5B6C1C2C3C4

C5

P2A1 A2

P1

A master(all reads and writes)

B master(all reads and writes)

C master(all reads and writes)

Tuesday, December 3, 13

Page 33: Apache Kafka Lightning Talk

A1 A2

A3A4

A5

A6A7

server A

Replication

server Bserver C

B1B2B3B4

B5B6

C1

C2C3C4

C5

P2A1 A2

P1C1

A master(all reads and writes)

B master(all reads and writes)

C master(all reads and writes)

Tuesday, December 3, 13

Page 34: Apache Kafka Lightning Talk

A1 A2 A3 A4

A5

A6A7

server A

Replication

server Bserver C

B1 B2 B3 B4 B5 B6

C1 C2 C3 C4 C5

P2A1 A2 A3 A4

P1C1 C2 C3 C4 C5

B1 B2 B3 B4 B5 B6

A master(all reads and writes)

B master(all reads and writes)

C master(all reads and writes)

Tuesday, December 3, 13

Page 35: Apache Kafka Lightning Talk

A1 A2 A3 A4 A5 A6 A7

server A

Replication

server Bserver C

B1 B2 B3 B4 B5 B6

C1 C2 C3 C4 C5

P2A1 A2 A3 A4

A5 A6 A7

P1C1 C2 C3 C4 C5

B1 B2 B3 B4 B5 B6

A master(all reads and writes)

B master(all reads and writes)

C master(all reads and writes)

Tuesday, December 3, 13

Page 36: Apache Kafka Lightning Talk

A1 A2 A3 A4 A5 A6 A7

server A

Replication

server Bserver C

B1 B2 B3 B4 B5 B6

C1 C2 C3 C4 C5

P2A1 A2 A3 A4 A5 A6 A7

P1C1 C2 C3 C4 C5

B1 B2 B3 B4 B5 B6

A master(all reads and writes)

B master(all reads and writes)

C master(all reads and writes)

Tuesday, December 3, 13

Page 37: Apache Kafka Lightning Talk

123 Messages in a partition are assigned a sequential id number, referred to as the offset.

Consumers keep track of their consumed offset in the partition. Kafka doesn’t maintain any metadata.

Kafka maintains all messages for a configurable time, regardless of whether they’ve been consumed.

Tuesday, December 3, 13

Page 38: Apache Kafka Lightning Talk

Alternativeshttp://queues.io

Tuesday, December 3, 13

Page 39: Apache Kafka Lightning Talk

Questions?

Tuesday, December 3, 13