Transcript

Real Time Messages at Scale with Apache Kafka

Will GardellaProduct Manager

©2015 Couchbase Inc. 2

©2015 Couchbase Inc. 3

©2015 Couchbase Inc. 4

©2015 Couchbase Inc. 5

©2015 Couchbase Inc. 6

Agenda

You might need Kafka if…

Kafka architecture

Background - Couchbase

Couchbase & Kafka

Behind the Scenes

Demo

An Example Producer and Consumer

What’s Apache Kafka for?

You might need Kafka if…

©2015 Couchbase Inc. 8

You might need Kafka if…

Photo Credit: Cory Doctorow https://www.flickr.com/photos/doctorow/14638938

602

©2015 Couchbase Inc. 9

Different speeds for different

systems

NoSQL

RDBMS

Cache

Search

Apps

Metrics

Logs

Hadoop

Relational Data Warehouse

Source:

Confluen

t

Typical Kafka Use Cases

©2015 Couchbase Inc. 12

Kafka Architecture

Broker 1Consumer

Producer

Producer

Producer

Consumer

Consumer

Consumer

Kafka Cluster

Broker 2

Broker 3

©2015 Couchbase Inc. 13

Kafka Architecture

Broker 1Consumer

Zookeeper

Producer

Producer

Producer

Consumer

Consumer

Consumer

Kafka Cluster

Broker 2

Broker 3

Topic 1 – Partition 1

Topic 2 – Partition 2

Topic 2 – Partition 1

Topic 3 – Partition 1

Topic 1 – Partition 2

Topic 3 – Partition 2

Couchbase Server 4.0

A brief Introduction

©2015 Couchbase Inc. 15

Couchbase Server 4.0 for modern applications

Combines the flexibility of JSON, the power of SQL, and the scale of

NoSQL

Develop with Agility Operate at Any Scale

Flexible JSON data model

Dynamic schema support

Powerful query language that extends SQL to JSON

Sub-millisecond latencies at scale

Elastic scaling on commodity servers

High availability

©2015 Couchbase Inc. 16

Couchbase Server Defined

The first NoSQL database that enables you to develop with agility and operate at any scale.

Managed Cache Key-Value Store Document

Database

Embedded

DatabaseSync Management

©2015 Couchbase Inc. 17

The Power Of The Flexible JSON Schema

Ability to store data in multiple ways• Denormalized single document, as opposed to normalizing data across multiple table

• Dynamic Schema to add new values when needed

©2015 Couchbase Inc. 18

Couchbase and Other Big Data Systems

data scientist / engineers up to 1010 application

users

NoSQL

Database

101- 102

Kafka Hadoop

Spark

Elasticsearch

EDW

Kafka & Couchbase Use Cases

©2015 Couchbase Inc. 21

Couchbase & Kafka Use Cases

Couchbase as the Master Database– Changes in the bucket update data elsewhere

Triggers / Event Handling– Handle events like deletions / expirations

externally

– E.g. expiration & replicated session tokens

Real-time Data Integration– Extract from Couchbase, transform and load

data in real-time

Real-time Data Processing– Extract from a bucket, process in real-time and

load back to another bucket

The Couchbase Kafka Connector

How it works

©2015 Couchbase Inc. 26

Database Change Protocol (DCP)

Couchbase Server’s internal data sync mechanism since Couchbase Server 3.x

Used for– Intra-Cluster Replication

– Indexing

– XDCR (Cross Datacenter Replication for HA/DR)

– Some connectors, including Kafka and Spark

• Use Couchbase 2.x Java SDK JVM Core IO DCP handling library

Sends mutations– Mutations = creation, update, or delete of an item

– Each mutation that occurs in a vBucket has a sequence number

Important: DCP not supported for external clients!

An Example Producer and Consumer

Connecting Couchbase via Kafka to an Application

©2015 Couchbase Inc. 35

Kafka Generator Example

©2015 Couchbase Inc. 36

Kafka Producer Example

©2015 Couchbase Inc. 37

Kafka Producer Example

©2015 Couchbase Inc. 38

A Kafka Consumer Example

©2015 Couchbase Inc. 39

A Kafka Consumer Example

Demo

©2015 Couchbase Inc. 41

Couchbase Kafka Connector Roadmap

Available Now: 1.2 GA

Kafka Producer or Consumer

Stream events

Filters

Transform events

41

Code: https://github.com/couchbase/couchbase-kafka-connector/Issues: https://issues.couchbase.com/projects/KAFKACDocs: http://developer.couchbase.com/documentation/server/4.1/connectors/kafka-1.2/kafka-intro.html

Planned

Monthly maintenance releases

Under discussion

Merge code for Storm connector

Adopt Kafka Connect (Kafka 0.9)

???

©2015 Couchbase Inc. 42

Learn More - Couchbase Kafka Connector

Confluent’s Ewen Cheslack-Postava at Couchbase Connect 2015 Great high level intro to Kafka in ~20 minutes

https://youtu.be/fFPVwYKUTHs

Couchbase and Kafka - Up and Running in 10 Minutes Run through the sample code yourself

http://blog.couchbase.com/2015/november/kafka-and-couchbase-up-and-running-in-10-minutes

Product docs http://developer.couchbase.com/documentation/server/4.1/connectors/kafka-1.2/kafka-intro.html

Avalon Consulting blog and Github repo http://blogs.avalonconsult.com/blog/big-data/purchase-transaction-alerting-with-couchbase-and-

kafka/

https://github.com/Avalon-Consulting-LLC/couchbase-kafka

42

Thank you.

[email protected]: @WillGardella


Top Related