building stream infrastructure across multiple data centers with apache kafka

When One Data Center is not Enough

Guozhang Wang Strata San Jose, 2016

Building large-scale stream infrastructure across multiple data centers with Apache Kafka

2

• Why across Data Centers?

• Design patterns for Multi-DC

• Kafka for Multi-DC

• Conclusion

Agenda

3

Why across Data Centers?

4

Why across Data Centers

• Catastrophic / expected failures

• Routine maintenance

• Geo-locality (Example: CDNs)

5

Why NOT across Data Centers

• Low bandwidth (10Mbps - 1Gbps)

• High latency (50ms - 450ms)

• Much More $$$

6

Why NOT across Data Centers

• … is hard and expensive

7

Why NOT across Data Centers

• … is hard and expensive

• … with real-time writes? Harder

8

Why NOT across Data Centers

• … is hard and expensive

• … with real-time writes? Harder

• … consistently? Oh My!

9

Consistency

• Weak

• Eventual

• StrongLatency Guarantee

10

Weak No Consistency

• Now you see my writes, now you don’t

• Best effort only, data can be stale

• Examples: think of “caches”, VoIP

11

Eventual Consistency

• You will see my writes, … eventually

• May need to resolve conflicts (manually)

• Examples: think of “emails”, SMTP

12

Strong Consistency

• You get what you write, for sure

• External > Sequential > Causal (Session)

• Examples: RDBMS, file systems

13

• LAN: consistency over latency

• WAN: latency over consistency

Latency vs. Consistency

14

• Why across Data Centers?

• Design patterns for Multi-DC

• Kafka for Multi-DC

• Conclusion

Agenda

15

Option I: Don’t do it

• Bunkerize the single data center

• Expect data loss at failures

• Examples: ??

16

Option II: Primary with Hot Standby

• Failover to hot standby (maybe inconsistent)

• Window of data loss at failures

• Examples: MySQL binlog

17

Option III: Active-Active

• Accepts writes in multi-DC

• Resolve conflicts (strong / week consistency)

• Examples: Amazon DynamoDB (vector clock) Google Spanner (2PC), Mesa (Paxos)

18

Ordering is the Key!

19

Ordering is Key

• Vector clocks: partial ordering

• Paxos, 2PC: global ordering

• Log shipping: logical ordering (per-partition)

21

Apache Kafka

• A distributed messaging system

..that store messages as a log!

22

Store Messages as a Log

4 5 5 7 8 9 10 11 12...

Producer Write

Consumer1 Reads (offset 7)

Consumer2 Reads (offset 10)

Messages

3

23

Partition the Log across Machines

Topic 1

Topic 2

Partitions

Producers

Producers

Consumers

Consumers

Brokers

24

ACK mode Latency On Failures

“no" no network delay some data loss

“leader" 1 network roundtrip a few data loss

“all" ~2 network roundtrips no data loss

Configurable ISR Commits

25

• Why across Data Centers?

• Design patterns for Multi-DC

• Kafka for Multi-DC

• Conclusion

Agenda

26

Option I: Active-Passive Replication

Kafka local

producers

consumer consumer

DC 1

MirrorMaker

DC 2

Kafka replica

27

Option I: Active-Passive Replication

• Async- replication across DC

• May lose data on failover

• Example: ETL to data warehouse / HDFS

Kafka local

producers

consumer consumer

DC 1

MirrorMaker

DC 2

Kafka replica

28

Option II: Active-Active Replication

Kafka local

Kafka aggregate

Kafka aggregate

producers producers

consumer consumer

MirrorMakerKafka local

on DC1 failure

DC 1 DC 2

29

Option II: Active-Active Replication

• Global view on agg. cluster

• Require offsets to resume

• Example: store materialization, index updates

Kafka local

Kafka agg

Kafka agg

producers producers

consumer consumer

MirrorMakerKafka local

on DC1 failure

DC 1 DC 2

30

• Offsets not identical between Kafka clusters• Duplicates during failover• Partition selection may be different

• Solutions• Resume from log end offset (suitable for real-time apps)• Resume from a timestamp (ListOffsets, offset index: KIP-33)

Caveats: offsets across DCs

31

Option III: Deploy across DCs

Kafka

producers producers

consumer consumer

DC 1 DC 2

32

Option III: Deploy across DCs

• Multi-tenancy support• Security (0.9)

• Quota Management (0.9)

• Latency optimization• Rack-aware partition assignment (0.10)

• Read affinity (future?)

Kafka

producers producers

consumer consumer

DC 1 DC 2

33

• Same region: essentially same network• asymmetric partitioning is rare, low latency• Need at least 3 DCs for Zookeeper

• Reserved instance to reduce churns• EIP for external clients, private IPs for internal communication• Reserved instance, local storage

Example: EC2 multi-AZ Deployment

34

Take-aways• Multi-DC: trade-off between latency and consistency

• Kafka: replicated log streams for multihoming

Thank youGuozhang | guozhang@confluent.io | @guozhangwang

Meet Confluent in booth #838

Confluent University ~ Kafka training ~ confluent.io/training

Join the Stream Data Hackathon Apr 25, SFkafka-summit.org/hackathon/

Download Apache Kafka & Confluent Platform

confluent.io/download