Change Data Capture Using Kafka

Akash Vacher2016/06/11

▪Akash VacherSite Reliability EngineerData Infrastructure Streaming (Bengaluru)Linkedin

Agenda▪ Story of a webapp

▪ Problems

▪ Answer

▪ Proposed solution

▪ Why Kafka?

▪ Refined solution

▪ Q and A

Story of a Webapp

Dual writes

Problems

Inconsistent data stores

Atomic writes are a pain

The answer?

The humble log!

Database Replication

Proposed solution

The log should be…

• Scalable

• Durable

• Low latency

• High throughput

• Allow bootstrapping – Have data persistence

Is Kafka a good fit?

Kafka has it all

▪ Horizontally Scalable

▪ Durable – Replication at Partition level

▪ Low latency

▪ High throughput

▪ Data is kept on disk

▪ Log compaction

Log Compaction

Revisiting proposed solution

Revised Solution

Change Data Capture

Questions?