apache kafka - messaging system overview

19
Dmitry Tolpeko, EPAM Systems – September 2014 Apache Kafka - Messaging System

Upload: dmitry-tolpeko

Post on 17-Dec-2014

265 views

Category:

Software


4 download

DESCRIPTION

Kafka is a real-time, fault-tolerant, scalable messaging system. It is a publish-subscribe system that connects various applications with the help of messages - producers and consumers of information.

TRANSCRIPT

Page 1: Apache Kafka - Messaging System Overview

Dmitry Tolpeko, EPAM Systems – September 2014

Apache Kafka - Messaging System

Page 2: Apache Kafka - Messaging System Overview

2

Kafka Overview

Kafka is a real-time, fault-tolerant, scalable messaging system.

It is a publish-subscribe system that connects various applications with the help of messages - producers and consumers of information.

Producers and consumers are independent, messages are queued, one producer can serve multiple consumers.

Was originally developed by LinkedIn.

Page 3: Apache Kafka - Messaging System Overview

3

CONCEPTSApache Kafka

SECTION

Page 4: Apache Kafka - Messaging System Overview

4

Kafka Architecture

• Brokers act as the server part of Kafka. Brokers are peers, there is no the master broker.

• Brokers can run on multiple nodes, but you can also run multiple brokers on each node. Each broker has own IP and port for client connections.

Producer(s) Broker(s) Consumers(s)

ZooKeeper

Client Server Client

Page 5: Apache Kafka - Messaging System Overview

5

Topic is a way to handle multiple data streams (different data feeds i.e.)

Each producer sends messages to, and consumers read the messages from the specified topic.  

New topics can be created automatically when a message with a new topic arrives, or you can use --create command to create a topic.

Topics

Broker

Topic 1

Topic 2

Producer 1

Producer 2

Producer 3

Consumer 1

Consumer 2

Page 6: Apache Kafka - Messaging System Overview

6

A topic can contain one or more partitions.

Each partition is stored on a single server, and multiple partitions allow the queue to scale and go beyond the limits of a single system.

Partitions also allow a single consumer to concurrently read messages in multiple concurrent threads. You can add new partitions dynamically.

Offset is uniquely identifies a message within partition.

Partitions

Broker 1

Topic 1

Partition 1

Topic 2

Partition 1

Broker 2

Topic 2

Partition 2

Partition 3

Page 7: Apache Kafka - Messaging System Overview

7

Each partition is replicated for fault-tolerance.

Partition has one server that acts a Leader, it handles all read-write requests.

Zero or more servers act as Followers, they replicate the leader and if it fails one of them becomes the new Leader.

Leader uses ZooKeeper heartbeat mechanism to indicate that it is alive.

A follower acts as a normal consumer, it pulls messages and updates own log. Only when all followers (ISR group) sync the message it can be send to consumers. When a follower rejoins after a downtime it can re-sync.

Replication

Partition 1 - Leader

Broker 1

Topic 1

Broker 2

Topic 1

Partition 1 - Follower

Page 8: Apache Kafka - Messaging System Overview

8

Consumers are organized to consumer groups.

To consume a single message by multiple consumers, they must belong to different consumer groups.

A consumer group is a single consumer abstraction, so consumers from single group read messages like from a queue there is no message broadcast within the group. This helps balance load among consumers of the same type (fault-tolerance, scalability).

The state of consumed messages are handled by consumers, not brokers. Consumers store the state in ZooKeeper - offset within each partition for each Consumer group, not consumer (!)

Consumer group name is unique within the Kafka cluster.

Consumer Groups

Partition 1

Topic 1

Partition 2

Partition 3

Group 1

Consumer

Consumer

Group 2

Consumer

Page 9: Apache Kafka - Messaging System Overview

9

Each partition can be consumed only one consumer within the consumer group.

Kafka only provides total order guarantee within a partition, not between different partitions in a topic.

If you need total order over messages you have to use one partition, and in this case you can use only one consumer process.

Kafka guarantees at-least-once delivery semantics by default where messages are never lost but may be redelivered (keys can be used to handle duplicates). Kafka offers options to disable retries (so messages can be lost) in case if the application can handle this, and needs a higher performance.

Kafka retains all published messages - no matter whether they are consumed or not - for the configured period of time (2 days by default).

Order Guarantees and Delivery Semantics

Page 10: Apache Kafka - Messaging System Overview

10

Producer can assign a key for a message that defines which partition to publish message to.

• Random (default, when no partition class or key specified)

• Round-robin for load balancing

• Partition function (hash by message key i.e.) - if key is a class type (Source ID i.e.) then all messages of the same type go to one partition.

Producer can optionally require an acknowledgment from the broker that the message was received (synced to Leader or all followers).

Kafka can group multiple messages and compress them.

Producers

Producer 1Partition 1

Partition 2

Partition 3

Topic 1

Producer 2

Page 11: Apache Kafka - Messaging System Overview

11

Consumers read the messages from the brokers leading the partitions (pull method). A consumer labels itself with a consumer group.

If the number of consumers of a specific consumer group is greater than the number of partitions, then some consumers will never see a message.

If there are more partitions than consumers of a specific consumer group, then a consumer can get messages from multiple partitions (no order guarantee). Then when you add consumers, Kafka re-balances partitions.

Consumers can get compressed message as a single message.

Consumers

Partition 1

Partition 2

Partition 3

Group 1

Consumer 1

Consumer 2

Consumer 3

Consumer 4

Page 12: Apache Kafka - Messaging System Overview

12

There are High Level and Simple Consumer API.

A High Level Consumer sets auto.commit.interval.ms option that defines how often offset is updated in ZooKeeper. If an error occurs between updates, the consumer will get replayed messages (!)

Simple Consumer is a low-level API that allows you to set any offset, explicitly read messages multiple times, or ensure that a message is processed only once.

Consumer Advanced Features

Page 13: Apache Kafka - Messaging System Overview

13

INTERNALSApache Kafka

SECTION

Page 14: Apache Kafka - Messaging System Overview

14

Kafka relies heavily on OS disk cache, not JVM heap even for caching messages. Data immediately written (appended) to a file. Consumed messages are not deleted.

Data files (called logs) are stored at log.dirs

A directory exists for each topic partition that contains log segments (files  0000000.log - named as offset of the 1st message in the log). log.segment.bytes and log.roll.hours define rotation policy.

log.flush.interval.xxx options define how often fsync performed on files.

All options can be specified either globally or per topic.

Persistence

Broker JVM App

OS page cache

/data/kafka-logs

TopicName-0

00000.log

Page 15: Apache Kafka - Messaging System Overview

15

Messages can be grouped together to minimize the number of network round-trips.

Multiple messages can be also compressed together (GZIP, Snappy) that helps achieve a good compression rate and reduce amount of data sent over network.

Producer can specify compression.codec and compressed.topics

Network I/O

Message3

Compressed

Network

Message2

Message1

Page 16: Apache Kafka - Messaging System Overview

16

There is no in-memory application level cache, data are in the OS pagecache.

Kafka uses sendfile Linux API calls that directly sends data from pagecache to a network socket, so there is no need to do read/write to application memory space.

Grouped messages are stored compressed in the log, and decompressed only by consumers.

Memory

Broker JVM App

OS page cache

Network

Page 17: Apache Kafka - Messaging System Overview

17

Without log compaction (time series data):

Log Compaction

Key1 Key2 Key3 Key1 Key2 Key1 Key3

A B C AA BB AAA CC

With log compaction only the last update is stored for each key:

Log compaction can be defined per topic. This can help increase performance of roll-forward operations, and reduce storage.

Key2 Key1 Key3

BB AAA CC

Page 18: Apache Kafka - Messaging System Overview

18

• Messaging - decouple processing or handle message buffer

• Monitoring and Tracking - collect activity, clickstream, status data and logs from various systems

• Stream Processing - aggregate, enrich, handle micro-batches etc.

• Commit Log - facilitate replication between systems

Kafka Use Cases