Real time Messages at Scale with Apache Kafka and Couchbase

Download Real time Messages at Scale with Apache Kafka and Couchbase

Post on 21-Jan-2018

1.521 views

Category:

Software

1 download

TRANSCRIPT

  1. 1. RealTime Messages at Scale with Apache Kafka Will Gardella Product Manager
  2. 2. 2015 Couchbase Inc. 2
  3. 3. 2015 Couchbase Inc. 3
  4. 4. 2015 Couchbase Inc. 4
  5. 5. 2015 Couchbase Inc. 5
  6. 6. 2015 Couchbase Inc. 6 Agenda You might need Kafka if Kafka architecture Background - Couchbase Couchbase & Kafka Behind the Scenes Demo An Example Producer and Consumer
  7. 7. Whats Apache Kafka for? You might need Kafka if
  8. 8. 2015 Couchbase Inc. 8 You might need Kafka if Photo Credit: Cory Doctorow https://www.flickr.com/photos/doctorow/14638938
  9. 9. 2015 Couchbase Inc. 9 Different speeds for different systems NoSQL RDBMS Cache Search Apps Metrics Logs Hadoop Relational Data Warehouse
  10. 10. Source: Confluen t
  11. 11. Typical Kafka Use Cases
  12. 12. 2015 Couchbase Inc. 12 Kafka Architecture Broker 1 Consumer Producer Producer Producer Consumer Consumer Consumer Kafka Cluster Broker 2 Broker 3
  13. 13. 2015 Couchbase Inc. 13 Kafka Architecture Broker 1 Consumer Zookeeper Producer Producer Producer Consumer Consumer Consumer Kafka Cluster Broker 2 Broker 3 Topic 1 Partition 1 Topic 2 Partition 2 Topic 2 Partition 1 Topic 3 Partition 1 Topic 1 Partition 2 Topic 3 Partition 2
  14. 14. Couchbase Server 4.0 A brief Introduction
  15. 15. 2015 Couchbase Inc. 15 Couchbase Server 4.0 for modern applications Combines the flexibility of JSON, the power of SQL, and the scale of NoSQL Develop with Agility Operate at Any Scale Flexible JSON data model Dynamic schema support Powerful query language that extends SQL to JSON Sub-millisecond latencies at scale Elastic scaling on commodity servers High availability
  16. 16. 2015 Couchbase Inc. 16 Couchbase Server Defined The first NoSQL database that enables you to develop with agility and operate at any scale. Managed Cache Key-Value Store Document Database Embedded Database Sync Management
  17. 17. 2015 Couchbase Inc. 17 The Power Of The Flexible JSON Schema Ability to store data in multiple ways Denormalized single document, as opposed to normalizing data across multiple table Dynamic Schema to add new values when needed
  18. 18. 2015 Couchbase Inc. 18 Couchbase and Other Big Data Systems data scientist / engineersup to 1010 application users NoSQL Database 101- 102 Kafka Hadoop Spark Elasticsearch EDW
  19. 19. Kafka & Couchbase Use Cases
  20. 20. 2015 Couchbase Inc. 21 Couchbase & Kafka Use Cases Couchbase as the Master Database Changes in the bucket update data elsewhere Triggers / Event Handling Handle events like deletions / expirations externally E.g. expiration & replicated session tokens Real-time Data Integration Extract from Couchbase, transform and load data in real-time Real-time Data Processing Extract from a bucket, process in real-time and load back to another bucket
  21. 21. The Couchbase Kafka Connector How it works
  22. 22. 2015 Couchbase Inc. 26 Database Change Protocol (DCP) Couchbase Servers internal data sync mechanism since Couchbase Server 3.x Used for Intra-Cluster Replication Indexing XDCR (Cross Datacenter Replication for HA/DR) Some connectors, including Kafka and Spark Use Couchbase 2.x Java SDK JVM Core IO DCP handling library Sends mutations Mutations = creation, update, or delete of an item Each mutation that occurs in a vBucket has a sequence number Important: DCP not supported for external clients!
  23. 23. An Example Producer and Consumer ConnectingCouchbase via Kafka to anApplication
  24. 24. 2015 Couchbase Inc. 35 Kafka Generator Example
  25. 25. 2015 Couchbase Inc. 36 Kafka Producer Example
  26. 26. 2015 Couchbase Inc. 37 Kafka Producer Example
  27. 27. 2015 Couchbase Inc. 38 A Kafka Consumer Example
  28. 28. 2015 Couchbase Inc. 39 A Kafka Consumer Example
  29. 29. Demo
  30. 30. 2015 Couchbase Inc. 41 Couchbase Kafka Connector Roadmap Available Now: 1.2 GA Kafka Producer or Consumer Stream events Filters Transform events 41 Code: https://github.com/couchbase/couchbase-kafka-connector/ Issues: https://issues.couchbase.com/projects/KAFKAC Docs: http://developer.couchbase.com/documentation/server/4.1/connectors/kafka-1.2/kafka-intro.html Planned Monthly maintenance releases Under discussion Merge code for Storm connector Adopt Kafka Connect (Kafka 0.9) ???
  31. 31. 2015 Couchbase Inc. 42 Learn More - Couchbase Kafka Connector Confluents Ewen Cheslack-Postava at Couchbase Connect 2015 Great high level intro to Kafka in ~20 minutes https://youtu.be/fFPVwYKUTHs Couchbase and Kafka - Up and Running in 10 Minutes Run through the sample code yourself http://blog.couchbase.com/2015/november/kafka-and-couchbase-up-and-running-in-10-minutes Product docs http://developer.couchbase.com/documentation/server/4.1/connectors/kafka-1.2/kafka-intro.html Avalon Consulting blog and Github repo http://blogs.avalonconsult.com/blog/big-data/purchase-transaction-alerting-with-couchbase-and- kafka/ https://github.com/Avalon-Consulting-LLC/couchbase-kafka 42
  32. 32. Thank you. will.gardella@couchbase.com Twitter: @WillGardella