Event your life with Kafka

Mari Gutiérrez :: Madrid.rb :: Dec 2014

@valakirka

kafka /ˈkɑːfkɑː/

“Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a

messaging system, but with a unique design.”

zookeeper /ˈzuːˌkiːpə/

“ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed

synchronization, and providing group services.”

topic

partition

broker

producer

consumer

consumer group

kafka in Ruby: Poseidon /bpot/poseidon

require 'poseidon' !class MyProducer def initialize partitioner = lambda { |key, partition_count| key.to_i % partition_count } @producer = Poseidon::Producer.new( ['localhost:9092'], self.class.name, partitioner: partitioner) end ! def send_message(body) message = Poseidon::MessageToSend.new( 'mytopic', body, Time.now.to_i.to_s) ! @producer.send_messages([message]) end end

require 'poseidon' !class MyConsumer def initialize(offset) @consumer = Poseidon::PartitionConsumer. consumer_for_partition(self.class.name, ['localhost:9092'], 'mytopic', 0, offset) end ! def run loop do @consumer.fetch.each do |event| puts event.value end end end end

consumer = MyConsumer.new(0) consumer.run !producer = MyProducer.new producer.send_message('hola Madrid.rb')

require 'poseidon' require 'poseidon_cluster' !class MyConsumerGroup def initialize @group = Poseidon::ConsumerGroup.new( self.class.name, ['localhost:9092'], ['localhost:2181'], 'mytopic') end ! def run @group.fetch_loop do |partition, messages| messages.each { |m| puts m.value } end end end

why kafka? usage pattern

offset & retention

batch/offline consumers

ordered delivery

event sourcing /ɪˈvɛnt ˈsɔːsɪŋ/

“Capture all changes to an application state as a sequence of events.”

¡gracias!

sourceshttp://kafka.apache.org/documentation.html !http://zookeeper.apache.org/ !https://kafka.apache.org/08/ops.html !https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol !http://www.infoq.com/articles/apache-kafka !https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines !https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem !https://github.com/bpot/poseidon !https://github.com/bsm/poseidon_cluster !http://martinfowler.com/eaaDev/EventSourcing.html !http://harpers.org/wp-content/uploads/whykafka-final.jpg !https://github.com/valakirka/kafka_example