When One Data Center Is Not EnoughBuilding Large-scale Stream Infrastructures Across Multiple Data Centerswith Apache KafkaGwen Shapira

There’s a book on that!

Actually… a chapter

Outline

Kafka overviewCommon multi data center patterns Future stuff

What is Kafka?▪It’s like a message queue, right?-Actually, it’s a “distributed commit log”-Or “streaming data platform”

0 1 2 3 4 5 6 7 8

Data Source

Data Consumer

A

Data Consumer

B

Topics and Partitions▪Messages are organized into topics, and each topic is split into partitions.

- Each partition is an immutable, time-sequenced log of messages on disk.- Note that time ordering is guaranteed within, but not across, partitions.

0 1 2 3 4 5 6 7 8

0 1 2 3 4 5 6 7 8

0 1 2 3 4 5 6 7 8

Partition 0

Partition 1

Partition 2

Data SourceTopic

Scalable consumption model

Topic T1Partition 0Partition 1

Partition 2Partition 3

Consumer Group 1

Consumer 1

Topic T1

Partition 0Partition 1

Partition 2Partition 3

Consumer Group 1Consumer 1

Consumer 2

Consumer 3

Consumer 4

Kafka usage

Common use case

Large scale real time data integration

Other use cases

Scaling databasesMessagingStream processing…

Important things to remember:

1. Consumers offset commits2. Within a cluster – each partition has replicas3. Inter-cluster replication, producer and consumer defaults – all tuned for LAN

Why multiple data centers (DC)

Offload work from main clusterDisaster recoveryGeo-localization

• Saving cross-DC bandwidth• Better performance by being closer to users• Some activity is just local• Security / regulations

CloudSpecial case: Producers with network issues

Why is this difficult?

1. It isn’t, really – you consume data from one cluster and produce to another2. Network between two data centers can get tricky3. Consumers have state (offsets) – syncing this between clusters get tough

• And leads to some counter intuitive results

Pattern #1: stretched cluster

Typically done on AWS in a single region• Deploy Zookeeper and broker across 3 availability zones

Rely on intra-cluster replication to replica data across DCs

Kafka

producers

consumers

DC 1

DC 3

DC 2 produce

rsproduce

rs

consumers

consumers

On DC failure

Producer/consumer fail over to new DCs• Existing data preserved by intra-cluster replication• Consumer resumes from last committed offsets and will see same data

Kafka

producers

consumers

DC 1

DC 3

DC 2 produce

rs

consumers

When DC comes back

Intra cluster replication auto re-replicates all missing dataWhen re-replication completes, switch producer/consumer back

Kafka

producers

consumers

DC 1

DC 3

DC 2 produce

rsproduce

rs

consumers

consumers

Be careful with replica assignment

Don’t want all replicas in same AZRack-aware support in 0.10.0

• Configure brokers in same AZ with same broker.rack

Manual assignment pre 0.10.0

Stretched cluster NOT recommended across regions

Asymmetric network partitioning

Longer network latency => longer produce/consume timeCross region bandwidth: no read affinity in Kafka

region 1Kafk

a ZK

region 2Kafk

a ZK

region 3Kafk

a ZK

Pattern #2: active/passive

Producers in active DCConsumers in either active or passive DC

Kafka

producers

consumers

DC 1

Replication

DC 2

Kafka

consumers

Critical Apps

Nice Reports

Cross Datacenter Replication

Consumer & Producer: read from a source cluster and write to a target clusterPer-key ordering preservedAsynchronous: target always slightly behindOffsets not preserved

• Source and target may not have same # partitions• Retries for failed writes

Options:• Confluent Multi-Datacenter Replication• MirrorMaker

On active DC failure

Fail over producers/consumers to passive clusterChallenge: which offset to resume consumption

• Offsets not identical across clusters

Kafka

producers

consumers

DC 1

Replication

DC 2

Kafka

Solutions for switching consumers

Resume from smallest offset• Duplicates

Resume from largest offset• May miss some messages (likely acceptable for real time consumers)

Replicate offsets topic• May miss some messages, may get duplicates

Set offset based on timestamp• Old API hard to use and not precise• Better and more precise API in Apache Kafka 0.10.1 (Confluent 3.1)• Nice tool coming up!

Preserve offsets during replication• Harder to do

When DC comes back

Need to reverse replication• Same challenge: determining the offsets

Kafka

producers

consumers

DC 1

Replication

DC 2

Kafka

Limitations

Reconfiguration of replication after failoverResources in passive DC under utilized

Pattern #3: active/active

Local aggregate replication to avoid cyclesProducers/consumers in both DCs

• Producers only write to local clusters

Kafka local

Kafka aggrega

te

Kafka aggrega

te

producers

producers

consumers

consumers

ReplicationKafka local

DC 1

DC 2

consumers

consumers

On DC failure

Same challenge on moving consumers on aggregate cluster• Offsets in the 2 aggregate cluster not identical• Unless the consumers are continuously running in both clusters

Kafka local

Kafka aggrega

te

Kafka aggrega

te

producers

producers

consumers

consumers

ReplicationKafka local

DC 1

DC 2

consumers

consumers

SFKafka

Cluster

HoustonKafka

Cluster

Allapps

Allapps

West coastUsers

South CentralUsers

When DC comes back

No need to reconfigure replication

Kafka local

Kafka aggrega

te

Kafka aggrega

te

producers

producers

consumers

consumers

ReplicationKafka local

DC 1

DC 2

consumers

consumers

Alternative: avoid aggregate clusters

Prefix topic names with DC tagConfigure replication to replicate remote topics onlyConsumers need to subscribe to topics with both DC tags

Kafka

producers

consumers

DC 1

Replication

DC 2

Kafka

producers

consumers

Beyond 2 DCs

More DCs better resource utilization• With 2 DCs, each DC needs to provision 100% traffic• With 3 DCs, each DC only needs to provision 50% traffic

Setting up replication with many DCs can be daunting• Only set up aggregate clusters in 2-3

Comparison

Pros ConsStretched • Better utilization of

resources• Easy failover for

consumers

• Still need cross region story

Active/passive

• Needed for global ordering • Harder failover for consumers• Reconfiguration during failover• Resource under-utilization

Active/active • Better utilization of resources

• Can be used to avoid consumer failover

• Can be challenging to manage• More replication bandwidth

Multi-DC beyond Kafka

Kafka often used together with other data storesNeed to make sure multi-DC strategy is consistent

Example application

Consumer reads from Kafka and computes 1-min countCounts need to be stored in DB and available in every DC

Independent database per DC

Run same consumer concurrently in both DCs• No consumer failover needed

Kafka local

Kafka aggrega

te

Kafka aggrega

te

producers

producers

consumer

consumer

ReplicationKafka local

DC 1

DC 2

DB DB

Stretched database across DCs

Only run one consumer per DC at any given point of time

Kafka local

Kafka aggrega

te

Kafka aggrega

te

producers

producers

consumer

consumer

ReplicationKafka local

DC 1

DC 2

DB DB

on failover

Practical tips

• Consume remote, produce local• Unless you need encrypted data on the wire• Monitor!

• Burrow for replication lag• Confluent Control Center for end-to-end• JMX metrics for rates and “busy-ness”

• Tune!• Producer / Consumer tuning• Number of consumers, producers• TCP tuning for WAN

• Don’t forget to replicate configuration• Separate critical topics from nice-to-have topics

Future work

Offset reset toolOffset preservation“Remote Replicas”2-DC stretch cluster

Other cool Kafka future:• Exactly Once• Transactions• Headers

THANK YOU!Gwen Shapira| gwen@confluent.io | @gwenshap

Kafka Training with Confluent University• Kafka Developer and Operations Courses• Visit www.confluent.io/training

Want more Kafka?• Download Confluent Platform Enterprise at http://www.confluent.io/product• Apache Kafka 0.10.2 upgrade documentation at http://docs.confluent.io/3.2.0/upgrade.html • Kafka Summit recordings now available at http://kafka-summit.org/schedule/

Discount code: kafstrataSpecial Strata Attendee discount code = 25% off www.kafka-summit.orgKafka Summit New York: May 8Kafka Summit San Francisco: August 28

Presented by