kafka overview and use cases

Upload: indrajeet-kumar

Post on 26-Jan-2017

353 views

Category:

Technology

1 download

Report

Download

Embed Size (px):

TRANSCRIPT

Kafka - Overview

Indrajeet KumarSite Reliability Engineer at LinkedIn

So what is it?

It is a high-throughput, low-latency messaging system

And who uses it?

What for?•Messaging

•Website Activity Tracking

•Metrics

•Log Aggregation

•Stream Processing

•For fun ;)

So how does it work?

▪Components– Producer– Broker

▪Topic▪Partition

– Consumer

Broker

producer producer producer

B2B1P1 P2P1R P2

consumer consumer consumer

The consumer

consumer

B2B1P1 P2B3P3

C1 C2

P1R

The Producer

Producer

B2B1P1 P2B3P3

P1 P2

P1R

Attributes of a Kafka Cluster

▪Durable

▪Scalable

▪Low Latency

▪Finite Retention

▪No single point of failure

Kafka At LinkedIn

▪Multiple Datacenters, Multiple Clusters

▪Mirroring between clusters

▪Message Types– Metrics– Tracking– Queuing

▪Data transport from applications to Hadoop, and back

Some numbers!

▪1800+ Broker machines▪79K+ Topics▪1.1M+ Partitions

▪1.3 Trillion messages per day▪330 Terabytes in/day▪1.2 Petabytes out/day

▪Peak load for a single cluster– 2 million messages/sec– 4.7 Gigabits/sec inbound– 15 Gigabits/sec outbound

Questions

Kafka Reliability Guarantees ATL Kafka User Group

· Apache Kafka Introduction to Apache Kafka Apache Kafka Architecture explanation Practical Examples on Apache Kafka SCALA, PYTHON, SPARK Course Content

Kafka Audit - Kafka Meetup - January 27th, 2015

Spring for Apache Kafka · Spring for Apache Kafka 1.0.1.RELEASE Spring Kafka 2 2. Introduction This first part of the reference documentation is a high-level overview of Spring for

Overview of Zookeeper, Helix and Kafka (Oakjug)

Kafka Performances 1perfug.github.io/assets/files/PerfUG68.pdf · Apache Kafka. 34 Consuming From Kafka - Single Consumer C. 35 Consuming From Kafka - Grouped Consumers ... programming.”

Building Distributed Semantic Job Queue with Kafka · Apache Kafka Overview What is Apache Kafka ? Run as a cluster on one or more servers that can span multiple DC Apache Kafka®

User's Guide Apache Kafka Software Release 2 diagram shows the parts (green) of Apache Kafka: Core Apache Kafka (light green), including the Kafka client API and the Kafka broker

Kafka Tutorial: Kafka Security

Working with Kafka Advanced Consumers - Cloudurable · 2020-02-12 · Kafka / Cassandra Support in EC2/AWS. Kafka Training, Kafka Consulting, Kafka Tutorial Objectives Advanced Kafka

Kafka Messaging JavaCro2019 IBM pdfv · •Kafka basics •Kafka Use Cases •Java Connector Architecture •Distributed messaging challenges •Reliable Kafka Processing •IBM Event

Kafka Tutorial - Introduction to Apache Kafka (Part 1)

cases 1 AN OVERVIEW

Forcepoint Behavioral Analytics Installation Manual › content › support › library › ... · Kafka kafka 9092-9095 API, Rose Kafka Manager kafka 9000 Administrator Workstation

LPWAN - Overview & use cases

Topics Covered - eclass.uoa.gr · Topics Covered ¾ What is Kafka ¾ ¾ Why Kafka ¾ ¾ High level overview ¾ ¾ Use cases ¾ ¾ Key terminology ¾ ¾ Partitions distribution over

Metamorphosis by Kafka--An overview

MAURICE BLANCHOT, de Kafka a Kafka

20+ MILLION RECORDS A SECOND - dellemc.com · Apache Kafka Performance with Dell EMC Isilon F800 All-Flash NAS Overview Kafka is a distributed, horizontally-scalable, fault-tolerant,

PDF: Kafka Tutorial - Cloudurablecloudurable.com/ppt/cloudurable-kafka-tutorial-v1.pdf · Cassandra / Kafka Support in EC2/AWS. Kafka Training, Kafka Consulting ™ What is Kafka?

Formatted: Figure [PACKT] cm, Width: 21.59 cm, Height: 27 ... · Kafka 0.7.x Consumer Kafka 0.7.x Cluster Kafka Migration Kafka 0.8 Cluster Kafka 0.8 Producer Producer (Front End)

Kafka Streams: Hands-on Session - ce.uniroma2.it · Kafka Streams Kafka Streams: • Kafka Streams is a client library for processing and analyzing data stored in Kafka • Supports

101 ways to configure kafka - badly (Kafka Summit)

Kafka Connect & Streams - the ecosystem around Kafka

Kafka Low-Level Design discussion of Kafka Design Kafka …cloudurable.com/ppt/4-kafka-detailed-architecture.pdf · Cassandra / Kafka Support in EC2/AWS. Kafka Training, Kafka Consulting

Kafka Connect & Kafka Streams/KSQL - powerful ecosystem around Kafka core

Tungsten Replicator for Kafka, Elasticsearch, Cassandra to Kafka+Elastic... · •Full transaction support for Kafka •Support for Amazon Elasticsearch •Kafka Extraction –Parsing

Kafka blr-meetup-presentation - Kafka internals

Paris Kafka Meetup - How to develop with Kafka

Kafka & Hadoop - for NYC Kafka Meetup

Franz Kafka - Letter to his · PDF fileFranz Kafka Pictures Home Page Kafka and Judaism The Holocaust photographs Galleries Franz Kafka Biography Franz Kafka-Wax Museum Kafka &

Practical data science use cases in the 4th industrial ... · Kafka overview Apache Kafka is a distributed streaming platform. • Publish and subscribe to streams of records. •

Apache Kafka Overview€¦ · Apache Kafka is a high performance, highly available, and redundant streaming message platform. Kafka functions much like a publish/subscribe messaging

Security, Privacy and IPR/Data Protection … Flink Apache Kafka and Apache Spark Integration Aspects Appendix D: Overview of Message Bus Technology MQTT Apache Kafka Apache Flume

Apache Kafka Overview - docs.cloudera.com