introducing kafka streams, the new stream processing library of apache kafka, berlin buzzwords 2016

82
1 Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Introducing Kafka Streams Michael Noll [email protected] Berlin Buzzwords, June 06, 2016 miguno

Upload: michael-noll

Post on 16-Apr-2017

6.713 views

Category:

Data & Analytics


1 download

TRANSCRIPT

Page 1: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

1Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Introducing Kafka Streams

Michael [email protected]

Berlin Buzzwords, June 06, 2016

miguno

Page 2: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

2Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Page 3: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

3Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Page 4: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

4Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Page 5: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

5Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Page 6: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

6Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Page 7: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

7Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Stream processing in Real Life™…before Kafka Streams

…somewhat exaggerated…but perhaps not that much

Page 8: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

8Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016Abandon all hope, ye who enter here.

Page 9: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

9Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

How did this… (#machines == 1)

scala> val input = (1 to 6).toSeq

// Stateless computationscala> val doubled = input.map(_ * 2)Vector(2, 4, 6, 8, 10, 12)

// Stateful computationscala> val sumOfOdds = input.filter(_ % 2 != 0).reduceLeft(_ + _)res2: Int = 9

Page 10: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

10Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

…turn into stuff like this (#machines > 1)

Page 11: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

11Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Page 12: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

12Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Page 13: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

13Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Page 14: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

14Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016Taken at a session at ApacheCon: Big Data, Hungary, September 2015

Page 15: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

15Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Kafka Streamsstream processing made simple

Page 16: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

16Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Kafka Streams• Powerful yet easy-to use Java library• Part of open source Apache Kafka, introduced in v0.10, May

2016• Source code:

https://github.com/apache/kafka/tree/trunk/streams• Build your own stream processing applications that are

• highly scalable• fault-tolerant• distributed• stateful• able to handle late-arriving, out-of-order data• <more on this later>

Page 17: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

17Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Kafka Streams

Page 18: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

18Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Kafka Streams

Page 19: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

19Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

What is Kafka Streams: Unix analogy

$ cat < in.txt | grep “apache” | tr a-z A-Z > out.txt

Kafka

Kafka Connect Kafka Streams

Page 20: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

20Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

What is Kafka Streams: Java analogy

Page 21: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

21Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

When to use Kafka Streams (as of Kafka 0.10)

Recommended use cases• Application Development• “Fast Data” apps (small or

big data) • Reactive and stateful

applications• Linear streams• Event-driven systems• Continuous transformations• Continuous queries• Microservices

Questionable use cases• Data Science / Data

Engineering• “Heavy lifting”• Data mining• Non-linear, branching

streams (graphs)• Machine learning, number

crunching• What you’d do in a data

warehouse

Page 22: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

22Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Alright, can you show me some code now?

KStream<Integer, Integer> input = builder.stream(“numbers-topic”);

// Stateless computationKStream<Integer, Integer> doubled = input.mapValues(v -> v * 2);

// Stateful computationKTable<Integer, Integer> sumOfOdds = input .filter((k,v) -> v % 2 != 0) .selectKey((k, v) -> 1) .reduceByKey((v1, v2) -> v1 + v2, ”sum-of-odds");

• API option 1: Kafka Streams DSL (declarative)

Page 23: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

23Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Alright, can you show me some code now?

Startup

Process a record

Periodic action

Shutdown

• API option 2: low-level Processor API (imperative)

Page 24: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

24Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

API != everythingAPI, coding

Operations, debugging, …

“Full stack” evaluation

Page 25: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

25Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

API != everythingAPI, coding

Operations, debugging, …

“Full stack” evaluation

Page 26: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

26Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Kafka Streams outsources hard problems to Kafka

Page 27: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

27Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

How do I install Kafka Streams?

<dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-streams</artifactId> <version>0.10.0.0</version></dependency>

• There is and there should be no “install”.• It’s a library. Add it to your app like any other library.

Page 28: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

28Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Page 29: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

29Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Do I need to install a CLUSTER to run my apps?• No, you don’t. Kafka Streams allows you to stay lean and

lightweight.• Unlearn bad habits: “do cool stuff with data != must have

cluster”

Ok. Ok. Ok. Ok.

Page 30: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

30Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

How do I package and deploy my apps? How do I …?

Page 31: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

31Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Page 32: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

32Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

How do I package and deploy my apps? How do I …?• Whatever works for you. Stick to what you/your company think

is the best way.• Why? Because an app that uses Kafka Streams is…a normal Java

app.• Your Ops/SRE/InfoSec teams may finally start to love not hate

you.

Page 33: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

33Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Kafkaconcepts

Page 34: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

34Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Kafka concepts

Page 35: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

35Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Kafka concepts

Page 36: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

36Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Kafka Streamsconcepts

Page 37: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

37Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Stream

Page 38: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

38Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Processor topology

Page 39: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

39Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Stream partitions and stream tasks

Page 40: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

40Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Streams meet Tables

http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simplehttp://docs.confluent.io/3.0.0/streams/concepts.html#duality-of-streams-and-tables

Page 41: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

41Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Streams meet Tables – in the Kafka Streams DSL

alice 2 bob 10 alice 3

time

“Alice clicked 2 times.”

“Alice clicked 2 times.”KTable= interprets data as changelog stream~ is a continuously updated materialized view

KStream

= interprets data as record stream

Page 42: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

42Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Streams meet Tables – in the Kafka Streams DSL

alice 2 bob 10 alice 3

time

“Alice clicked 2+3 = 5 times.”

“Alice clicked 2 3 times.”

KTable= interprets data as changelog stream~ is a continuously updated materialized view

KStream

= interprets data as record stream

Page 43: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

43Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Streams meet Tables – in the Kafka Streams DSL

alice eggs bob lettuce alice milk

alice paris bob zurich alice berlin

KStream

KTable

time

“Alice bought eggs.”

“Alice is currently in Paris.”

User purchase records

User profile/location information

Page 44: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

44Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Streams meet Tables – in the Kafka Streams DSL

alice eggs bob lettuce alice milk

alice paris bob zurich alice berlin

KStream

KTable

User purchase records

User profile/location information

time

“Alice bought eggs and milk.”

“Alice is currently in Paris Berlin.”

Show meALL values for a key

Show me the LATEST value for a key

Page 45: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

45Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Streams meet Tables – in the Kafka Streams DSL

Page 46: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

46Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Streams meet Tables – in the Kafka Streams DSL• JOIN example: compute user clicks by region via

KStream.leftJoin(KTable)

Page 47: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

47Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Streams meet Tables – in the Kafka Streams DSL• JOIN example: compute user clicks by region via

KStream.leftJoin(KTable)

Even simpler in Scala because, unlike Java, it natively supports tuples:

Page 48: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

48Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Streams meet Tables – in the Kafka Streams DSL• JOIN example: compute user clicks by region via

KStream.leftJoin(KTable)alice 13 bob 5Input KStream

alice (europe, 13) bob (europe,

5)leftJoin()w/ KTable

KStream

13europe 5europemap() KStream

KTablereduceByKey(_ + _) 13europe…………

18europe…………

Page 49: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

49Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Streams meet Tables – in the Kafka Streams DSL

Page 50: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

50Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Streams meet Tables – in the Kafka Streams DSL

Page 51: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

51Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Kafka Streamskey features

Page 52: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

52Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Key features in 0.10• Native, 100%-compatible Kafka integration

• Also inherits Kafka’s security model, e.g. to encrypt data-in-transit

• Uses Kafka as its internal messaging layer, too

Page 53: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

53Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Native Kafka integration• Reading data from Kafka

• Writing data to Kafka

Page 54: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

54Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Native Kafka integration• You can configure both Kafka Streams plus the underlying

Kafka clients

Page 55: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

55Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Key features in 0.10• Native, 100%-compatible Kafka integration

• Also inherits Kafka’s security model, e.g. to encrypt data-in-transit

• Uses Kafka as its internal messaging layer, too• Highly scalable• Fault-tolerant• Elastic

Page 56: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

56Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Execution model

Page 57: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

57Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Execution model

Page 58: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

58Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Execution model

Page 59: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

59Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Execution model

Page 60: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

60Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Kafka Streams outsources hard problems to Kafka

Page 61: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

61Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Key features in 0.10• Native, 100%-compatible Kafka integration

• Also inherits Kafka’s security model, e.g. to encrypt data-in-transit

• Uses Kafka as its internal messaging layer, too• Highly scalable• Fault-tolerant• Elastic• Stateful and stateless computations (e.g. joins,

aggregations)

Page 62: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

62Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Stateful computations• Stateful computations like aggregations or joins require state

• We already showed a join example in the previous slides.• Windowing a stream is stateful, too, but let’s ignore this for now.

• State stores in Kafka Streams• Typically: key-value stores• Pluggable implementation: RocksDB (default), in-memory, your

own …• State stores are per stream task for isolation (think: share-

nothing)• State stores are local for best performance• State stores are replicated to Kafka for elasticity and for

fault-tolerance

Page 63: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

63Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Execution model

State stores

(This is a bit simplified.)

Page 64: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

64Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Remember?

Page 65: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

65Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Execution model

State stores

(This is a bit simplified.)

charlie 3bob 1 alice 1alice 2

Page 66: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

66Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Execution model

State stores

(This is a bit simplified.)

Page 67: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

67Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Execution model

State stores

(This is a bit simplified.)

alice 1alice 2

Page 68: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

68Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Kafka Streams outsources hard problems to Kafka

Page 69: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

69Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Stateful computations• Kafka Streams DSL: abstracts state stores away from you

• Stateful operations include• count(), reduceByKey(), aggregate(), …

• Low-level Processor API: direct access to state stores• Very flexible but more manual work for you

Page 70: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

70Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Stateful computations• Use the low-level Processor API to interact directly with state

stores

Get the store

Use the store

Page 71: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

71Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Key features in 0.10• Native, 100%-compatible Kafka integration

• Also inherits Kafka’s security model, e.g. to encrypt data-in-transit

• Uses Kafka as its internal messaging layer, too• Highly scalable• Fault-tolerant• Elastic• Stateful and stateless computations• Time model

Page 72: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

72Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Time

Page 73: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

73Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Time

Page 74: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

74Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Time• You configure the desired time semantics through timestamp

extractors• Default extractor yields event-time semantics

• Extracts embedded timestamps of Kafka messages (introduced in v0.10)

Page 75: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

75Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Key features in 0.10• Native, 100%-compatible Kafka integration

• Also inherits Kafka’s security model, e.g. to encrypt data-in-transit

• Uses Kafka as its internal messaging layer, too• Highly scalable• Fault-tolerant• Elastic• Stateful and stateless computations• Time model• Windowing

Page 76: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

76Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

WindowingTimeWindows.of(3000)

TimeWindows.of(3000).advanceBy(1000)

Page 77: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

77Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Windowing use case: monitoring (1m/5m/15m averages)

Confluent Control Center for Kafka

Page 78: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

78Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Key features in 0.10• Native, 100%-compatible Kafka integration

• Also inherits Kafka’s security model, e.g. to encrypt data-in-transit• Uses Kafka as its internal messaging layer, too

• Highly scalable• Fault-tolerant• Elastic• Stateful and stateless computations• Time model• Windowing• Supports late-arriving and out-of-order data• Millisecond processing latency, no micro-batching• At-least-once processing guarantees (exactly-once is in

the works)

Page 79: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

79Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Wrapping up

Page 80: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

80Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Where to go from here?• Kafka Streams is available in Apache Kafka 0.10 and Confluent Platform

3.0• http://kafka.apache.org/• http://www.confluent.io/download (free + enterprise versions,

tar/zip/deb/rpm)

• Kafka Streams demos at https://github.com/confluentinc/examples • Java 7, Java 8+ with lambdas, and Scala• WordCount, Joins, Avro integration, Top-N computation, Windowing, …

• Apache Kafka documentation: http://kafka.apache.org/documentation.html

• Confluent documentation: http://docs.confluent.io/3.0.0/streams/• Quickstart, Concepts, Architecture, Developer Guide, FAQ

• Join our bi-weekly Ask Me Anything sessions on Kafka Streams• Contact me at [email protected] for details

Page 81: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

81Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Some of the things to come• Exactly-once semantics• Queriable state – tap into the state of your applications• SQL interface• Listen to and collaborate with the developer community

• Your feedback counts a lot! Share it via [email protected]

Tomorrow’s keynote (09:30 AM) by Neha Narkhede,co-founder and CTO of Confluent

“Application development and data in the emerging world of stream processing”

Page 82: Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

82Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016

Want to contribute to Kafka and open source?

Join the Kafka communityhttp://kafka.apache.org/

Questions, comments? Tweet with #bbuzz and /cc to @ConfluentInc

…in a great team with the creators of Kafka?Confluent is hiring http://confluent.io/