Apache Kafka at LinkedIn
Post on 19-Aug-2014
244 views
DESCRIPTION
Jay Kreps is a Principal Staff Engineer at LinkedIn where he is the lead architect for online data infrastructure. He is among the original authors of several open source projects including a distributed key-value store called Project Voldemort, a messaging system called Kafka, and a stream processing system called Samza. This talk gives an introduction to Apache Kafka, a distributed messaging system. It will cover both how Kafka works, as well as how it is used at LinkedIn for log aggregation, messaging, ETL, and real-time stream processing.TRANSCRIPT
- Jay Kreps Introduction to Apache Kafka
- The Plan 1. What is Apache Kafka? 2. Kafka and Data Integration 3. Kafka and Stream Processing
- Apache Kafka
- A brief history of Apache Kafka
- Characteristics Scalability of a filesystem Hundreds of MB/sec/server throughput Many TB per server Guarantees of a database Messages strictly ordered All data persistent Distributed by default Replication Partitioning model
- Kafka is about logs
- What is a log?
- Logs: pub/sub done right
- Partitioning
- Nodes Host Many Partitions
- Producers Balance Load
- Consumers Divide Up Partitions
- End-to-End
- Kafka At LinkedIn 175 TB of in-flight log data per colo Replicated to each datacenter Tens of thousands of data producers Thousands of consumers 7 million messages written/sec 35 million messages read/sec Hadoop integration
- Performance Producer (3x replication): Async: 786,980 records/sec (75.1 MB/sec) Sync: 421,823 records/sec (40.2 MB/sec) Consumer: 940,521 records/sec (89.7 MB/sec) End-to-end latency: 2 ms (median) 14 ms (99.9th percentile)
- The Plan 1. What is Apache Kafka? 2. Kafka and Data Integration 3. Kafka and Stream Processing
- Data Integration
- Maslows Hierarchy
- For Data
- New Types of Data Database data Users, products, orders, etc Events Clicks, Impressions, Pageviews, etc Application metrics CPU usage, requests/sec Application logs Service calls, errors
- New Types of Systems Live Stores Voldemort Espresso Graph OLAP Search InGraphs Offline Hadoop Teradata
- Bad
- Good
- Example: User views job
- Comparing Data Transfer Mechanisms
- The Plan 1. What is Apache Kafka? 2. Kafka and Data Integration 3. Kafka and Stream Processing
- Stream Processing
- Stream processing is a generalization of batch processing
- Stream Processing = Logs + Jobs
- Examples Monitoring Security Content processing Recommendations Newsfeed ETL
- Frameworks Can Help
- Samza Architecture
- Log-centric Architecture
- Kafka http://kafka.apache.org Samza http://samza.incubator.apache.org Log Blog http://linkd.in/199iMwY Benchmark: http://t.co/40fkKJvanx Me http://www.linkedin.com/in/jaykreps @jaykreps
