kafka & hadoop - for nyc kafka meetup

Kafka & Hadoop

Gwen Shapira / Software Engineer

2©2014 Cloudera, Inc. All rights reserved.

• 15 years of moving data around• Formerly consultant• Now Cloudera Engineer:– Flume– Sqoop– Kafka

About Me


There’s a book on that!


We are also blogging


Getting Data from Kafka to Hadoop

There are only bad options.

It's about finding the best one.


Camus


Camus

ZooKeeper

Setup

Topic Offsets

Pro

cess

es

HD

FSO

ther

Syst

em

s

TaskTask

Task

In process Avro Files

In process Avro Files

Audit Counts

Clean Up

Kakfa

B

A

C

D

F

G H

I

E


• Kafka has no MR layer– InputFormat, OutputFormat, Utils…

• Sqoop is a generic batch ingest framework– Why no Kafka?

Missing in Action


Flume + Kafka = Flafka

10

Sources Interceptors Selectors Channels Sinks

Flume Agent

How does work?Twitter, logs, webserver,

Kafka…

Mask, re-format,

validate…DR, critical

Memory, file

HDFS, Hbase,

Solr, Kafka

11

But I just want to get data from Kafka to Hbase / HDFS

©2014 Cloudera, Inc. All rights reserved.

12

Channels Sinks

Flume Agent

Kafka ChannelKafka! HDFS,

Hbase, Solr


SparkStreaming

Single Pass

SourceRawInputDStream

RDD


RDD

RDD

Filter Count Print


RDD

RDD

RDD

Single Pass

Filter Count Print

Pre-first Batch

First Batch

Second Batch


Storm

Spout

Source

Split wordsbolts

Split wordsbolts

Spout

Split wordsbolts

Split wordsbolts

Count

Count

Count

Spout Layer Fan out Layer 1 Shuffle Layer 2


Retro Thoughts


• Data often has schema• At least it should• Kafka is unaware – which is good• Need capability to figure out schema for

events• Without including it in every event

Schema


Kafka in Cloudera Manager

18

Visit us at

Booth #305

BOOK SIGNINGS THEATER SESSIONS

TECHNICAL DEMOS GIVEAWAYS