©2015 Couchbase Inc. 6
Agenda
You might need Kafka if…
Kafka architecture
Background - Couchbase
Couchbase & Kafka
Behind the Scenes
Demo
An Example Producer and Consumer
©2015 Couchbase Inc. 8
You might need Kafka if…
Photo Credit: Cory Doctorow https://www.flickr.com/photos/doctorow/14638938
602
©2015 Couchbase Inc. 9
Different speeds for different
systems
NoSQL
RDBMS
Cache
Search
Apps
Metrics
Logs
Hadoop
Relational Data Warehouse
©2015 Couchbase Inc. 12
Kafka Architecture
Broker 1Consumer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Kafka Cluster
Broker 2
Broker 3
©2015 Couchbase Inc. 13
Kafka Architecture
Broker 1Consumer
Zookeeper
Producer
Producer
Producer
Consumer
Consumer
Consumer
Kafka Cluster
Broker 2
Broker 3
Topic 1 – Partition 1
Topic 2 – Partition 2
Topic 2 – Partition 1
Topic 3 – Partition 1
Topic 1 – Partition 2
Topic 3 – Partition 2
©2015 Couchbase Inc. 15
Couchbase Server 4.0 for modern applications
Combines the flexibility of JSON, the power of SQL, and the scale of
NoSQL
Develop with Agility Operate at Any Scale
Flexible JSON data model
Dynamic schema support
Powerful query language that extends SQL to JSON
Sub-millisecond latencies at scale
Elastic scaling on commodity servers
High availability
©2015 Couchbase Inc. 16
Couchbase Server Defined
The first NoSQL database that enables you to develop with agility and operate at any scale.
Managed Cache Key-Value Store Document
Database
Embedded
DatabaseSync Management
©2015 Couchbase Inc. 17
The Power Of The Flexible JSON Schema
Ability to store data in multiple ways• Denormalized single document, as opposed to normalizing data across multiple table
• Dynamic Schema to add new values when needed
©2015 Couchbase Inc. 18
Couchbase and Other Big Data Systems
data scientist / engineers up to 1010 application
users
NoSQL
Database
101- 102
Kafka Hadoop
Spark
Elasticsearch
EDW
©2015 Couchbase Inc. 21
Couchbase & Kafka Use Cases
Couchbase as the Master Database– Changes in the bucket update data elsewhere
Triggers / Event Handling– Handle events like deletions / expirations
externally
– E.g. expiration & replicated session tokens
Real-time Data Integration– Extract from Couchbase, transform and load
data in real-time
Real-time Data Processing– Extract from a bucket, process in real-time and
load back to another bucket
©2015 Couchbase Inc. 26
Database Change Protocol (DCP)
Couchbase Server’s internal data sync mechanism since Couchbase Server 3.x
Used for– Intra-Cluster Replication
– Indexing
– XDCR (Cross Datacenter Replication for HA/DR)
– Some connectors, including Kafka and Spark
• Use Couchbase 2.x Java SDK JVM Core IO DCP handling library
Sends mutations– Mutations = creation, update, or delete of an item
– Each mutation that occurs in a vBucket has a sequence number
Important: DCP not supported for external clients!
©2015 Couchbase Inc. 41
Couchbase Kafka Connector Roadmap
Available Now: 1.2 GA
Kafka Producer or Consumer
Stream events
Filters
Transform events
41
Code: https://github.com/couchbase/couchbase-kafka-connector/Issues: https://issues.couchbase.com/projects/KAFKACDocs: http://developer.couchbase.com/documentation/server/4.1/connectors/kafka-1.2/kafka-intro.html
Planned
Monthly maintenance releases
Under discussion
Merge code for Storm connector
Adopt Kafka Connect (Kafka 0.9)
???
©2015 Couchbase Inc. 42
Learn More - Couchbase Kafka Connector
Confluent’s Ewen Cheslack-Postava at Couchbase Connect 2015 Great high level intro to Kafka in ~20 minutes
https://youtu.be/fFPVwYKUTHs
Couchbase and Kafka - Up and Running in 10 Minutes Run through the sample code yourself
http://blog.couchbase.com/2015/november/kafka-and-couchbase-up-and-running-in-10-minutes
Product docs http://developer.couchbase.com/documentation/server/4.1/connectors/kafka-1.2/kafka-intro.html
Avalon Consulting blog and Github repo http://blogs.avalonconsult.com/blog/big-data/purchase-transaction-alerting-with-couchbase-and-
kafka/
https://github.com/Avalon-Consulting-LLC/couchbase-kafka
42