confluent building a real-time streaming platform using kafka streams and kafka...

57
Building a real-time streaming platform using Kafka Connect + Kafka Streams Jeremy Custenborder, Systems Engineer, Confluent

Upload: thomas-alex

Post on 16-Apr-2017

154 views

Category:

Software


5 download

TRANSCRIPT

Page 1: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version

Building a real-time streaming platform using Kafka Connect + Kafka StreamsJeremy Custenborder, Systems Engineer, Confluent

Page 2: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 3: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 4: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 5: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 6: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 7: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 8: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 9: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 10: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 11: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 12: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 13: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 14: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 15: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 16: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 17: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 18: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 19: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 20: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 21: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 22: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 23: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 24: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 25: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 26: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version

• Everything in the company is a real-time stream

• > 1.2 trillion messages written per day• > 3.4 trillion messages read per day• ~ 1 PB of stream data• Thousands of engineers• Tens of thousands of producer processes

Page 27: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 28: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 29: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 30: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 31: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 32: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 33: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 34: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 35: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 36: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 37: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 38: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 39: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 40: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 41: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 42: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 43: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 44: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 45: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 46: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 47: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 48: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 49: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 50: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 51: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 52: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 53: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 54: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version
Page 55: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version

Resources• Confluent

• Company website: http://www.confluent.io• Blog: http://www.confluent.io/blog• Free Ebook “Making Sense of Stream Processing” http

://www.confluent.io/making-sense-of-stream-processing-ebook• Apache Kafka

• http://kafka.apache.org• Kafka Connect

• http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines

• Kafka Streams• http://www.confluent.io/blog/introducing-kafka

-streams-stream-processing-made-simple

Page 56: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version

Thanks!Jeremy Custenborder | [email protected] |

Download Kafkaand Confluent Platform

www.confluent.io/download

Page 57: Confluent   building a real-time streaming platform using kafka streams and kafka connect-20min-version

Training!http://www.confluent.io/training

Discount Code: BELLEVUE10

Operations Training in Seattle December 10th.