going event drive + kafka a rabbitmq

21
1 going event driven + kafka a rabbit-mq java group #13 bratislava @danielharcek

Upload: harcek

Post on 01-Nov-2014

90 views

Category:

Data & Analytics


2 download

DESCRIPTION

Kratko o event-driven architekture, zaklady messagingu, porovnanie mq systemov a scenarov pouzitia RabbitMQ a Kafka.

TRANSCRIPT

Page 1: Going event drive + Kafka a RabbitMQ

1

going event driven+ kafka a rabbit-mq

java group #13 bratislava @danielharcek

Page 2: Going event drive + Kafka a RabbitMQ

2

Agenda

1. Case study2. Retrospektiva3. Event-driven(ness)4. Messaging5. JMS6. RMQ7. Kafka8. Otazky?

Page 3: Going event drive + Kafka a RabbitMQ

3

Pred

lb

web

rtrtrtrtrt

rt

r loader

p loader

MySQLreq

multithreadeddaemons

MySQLprof

I/O

I/Oread

I/Omov

I/Omov

network

I/Owrite

network

I/Owrite

network

reportscampaign

s

MySQLad

MySQLsys

reportsaggregate

reportscustom

reportsvisiitors

web

sync

tab delimited files chunked per minutemostly cron-scheduled once per day

network I/O

write

MySQLreq

archive

Page 4: Going event drive + Kafka a RabbitMQ

4

Po

lb

web

rtrtrtrtrt

rt

rabbit

MySQLreqarch

MySQLprof

I/Oread

network

I/Owrite

network

I/Owrite

network

reportscampaign

s

MySQLad

MySQLsys

reportsaggregate

reportscustom

reportsvisiitors

web

sync

request sent, persisted and

forwarded over network

daemons & batch (cron jobs)

network I/O

write

Page 5: Going event drive + Kafka a RabbitMQ

5

#pred_a_po

rtrtrtrtrt

rt

r loader

p loader

MySQLreq

MySQLprof

reportscampaign

s

reportsaggregate

reportscustom

reportsvisiitors

rabbit

reportscampaign

s

reportsaggregate

reportscustom

reportsvisiitors

rtrtrtrtrt

rt

versus

Page 6: Going event drive + Kafka a RabbitMQ

6

Co sme ziskali• usetrili sme VELA disk i/o• usetrili load na DB a zredukovali mnozstvo

DB hostov (zostal vlastne uz iba “archiv” requestov)

• zlepsili distribuovatelnost• reporty sa stali menej previazane (netrebalo

zdielany diskovy caching)• moznost jednoduchsie pridavat nove typy

worker reportov• moznost signalovat potrebu recountu• skoro-real time countre (predtym

komplikovane koli loadu na DB)• ak nam padol tracker tak sme nestratili

(takmer) ziadne requesty• zjednodusila sa nam distribucia a mohli sme

sa viac sustredit na optimalizaciu reportingu• ak padol report spravy si nan pockaju (to ale

nebol problem predtym)

rabbit

reportscampaign

s

reportsaggregate

reportscustom

reportsvisiitors

rtrtrtrtrt

rt

Page 7: Going event drive + Kafka a RabbitMQ

7

“Reaktivita” je dnes proste

KAJŠMENTKE

asynchronous, non-blocking, real-time, highly-available, loosely

coupled, scalable, fault-tolerant, concurrent, reactive, event-driven, push instead of pull, distributed,

low latency, high throughput

Page 8: Going event drive + Kafka a RabbitMQ

8

Event-driven(ness)

But now a new architecture has evolved to let developers conceptualize and build applications that satisfy today’s demands.We call these Reactive Applications. This architecture allows developers to build systems that are event-driven, scalable, resilient and responsive.http://www.reactivemanifesto.org/

NIC NOVE!

Page 9: Going event drive + Kafka a RabbitMQ

9

Potreba

Komplexne systemy pracujuce s tokmi velkeho objemu dat s potrebou odozvy v takmer realnom case, “neustalym” uptimom nasadzovane do cloudoveho prostredia s flexibilnym modelom skalovania a spravy.

napr. IoT, mobilne appky, automaticke tradingove systemy

Page 10: Going event drive + Kafka a RabbitMQ

10

Messaging 101Push based (zvacsa)Producer (agent)Consumer (sink)“Event consumers subscribe to middleware which receives notification of an event from producer(s).”Vyuziva messaging middleware (backbone)• vacsinou menej alebo viac sofistikovany Broker (message manager)• alebo broker-less framework, jednoducho “Queue”* (p2p, napr. ZMQ)Durability vs. persistence (subscription vs. message)Garancia radenia (ziadna, FIFO)*Garancia dorucenia (at-most-once, at-least-once, exactly once)

Page 11: Going event drive + Kafka a RabbitMQ

11

Messaging broker / broker-less

O(n2) O(n)

http://www.eaipatterns.com/ramblings/03_hubandspoke.html

Page 12: Going event drive + Kafka a RabbitMQ

12

Messaging broker / broker-less

broker ako adresar distribuovany broker distribuovany adresar

http://zeromq.org/whitepapers:brokerless

Page 13: Going event drive + Kafka a RabbitMQ

13

Messaging - rozhodujúce kritériá• potrebna priepustnost (velkost spravy a pocet sprav), latencia• clustering / HA• aka topologia (broker, nebroker, aky workflow)• perzistencia (mat consumerov ktori nie su pern. online)• moznosti routingu, filtrovania, fransformacie• push a konzumovania batchov sprav• delivery a ordering garancie• “multiplatformovost” (klienti, drivers) a vyspelost• podpora protokolov• ttl, delayed / scheduled messages, prioritizacia messagov• acknowledgment / receipt notification

Page 14: Going event drive + Kafka a RabbitMQ

14

Messaging basic patterns: Queue

• point-to-point• by default FIFO• message je spracovany PRAVE

jednym consumerom

• logovanie udalosti / monitoring• load leveling (cakaju vo fronte

tak ako system stiha)• load balancing (pridanie

novych async workerov)

producer queue consumer

producer queue consumer

consumer

consumer

Page 15: Going event drive + Kafka a RabbitMQ

15

Messaging basic patterns: Publish-Subscribe • hub / broadcast

• sprava od publishera je forwardovana na vsetkych subscriberov

• * topic

• propagacia udalosti (napr. MMO, push notifikacie, live updaty skore zapasu a podobne)

• integracie

producer topic consumer

consumer

consumer

Page 16: Going event drive + Kafka a RabbitMQ

16

Messaging basic patterns: Request-reply

• ring • blizke RPC• asynchronne spracovanie

odpovede

• klientska aplikacia posle search query, backend searchne, vysledok sa vrati naspat

• aplikacia si vyziada stav inej aplikacie (napr. progress tasku)

producer

request q

consumer

reply q

Page 17: Going event drive + Kafka a RabbitMQ

17

JMS• Prva verzia 1.0.2b v 2001, 2.0 v 2013• Java Message Service API• MOM rozhranie (Message Oriented Middleware)• Nepopisuje protokol! (teda dve implementacie JMS nemusia vediet

komunikovat)• Provider, Client – Producer / Consumer, Message, Queue (per

Consumer), Topic (distribucny mechanizmus)• Nema garantovany ordering, dorucenie at-most-once alebo once-

and-only-once (ak je message persistentny)• Point-to-point a Publish-Subscribe• Implentacie: ActiveMQ, Qpid, Redis, …

Page 18: Going event drive + Kafka a RabbitMQ

18

RMQ• Vyspely broker (komplexne moznosti routovania, workflows), dokumentacia,

tooling• AMQP +plugins• Drivre do vsetkych relevantnych jazykov a kopec pluginov• Publisher, Exchange, Route, Queue, Consumer• Echanges: Direct, Fanout, Topic, Headers• Exchange su by default transientne, ich durabilitu a persistenciu je treba

explicitne deklarovat• Consumeri maju push aj pull API• ACK, a volba kedy poslat, nie je 100%, expiracia atd• pub-sub je Fanout• clustering, federation (plugin) a queues replication• Vhodny ak menej ako 20k+/sec* a ak potreba komplexnych routing

scenarov

Page 19: Going event drive + Kafka a RabbitMQ

19

Kafka• distribuovany pub-sub messaging system, skor klient-server ako broker (urceny pre

konkretny typ use-casov – spracovanie real time aktivity streamov, zber metrik, logovanie)• 0.8.x, meni sa pod rukami• klienti pre kazdy relevantny jazyk• messages su persistovane na servri, kafka zapise a potom zisti kto fetchuje,

konfigurovatelny “rolling window of time” (cas alebo miesto na hdd), jedna kopia stremu pre N consumerov

• dorucenie v poradi a at-least-once garancia• producer-centric (t.j. producer si strazi svoj stav – rmq ma metadata na strane brokera• pull-based (data fetchujeme a mozme aj specifikovat offset – robit rewind)• Cosumer, producer, topic, partition (ak su consumer groups)• Potrebuje Zookeeper – distribuovany register / adresar, je pouzity na koordinaciu clustra

producerov, consumerov a “brokera”• Od 0.8 replikacia partitions, partitioning definovany producerom, Kafka rozhoduje ako

rozhodi partitions na brokerov, aj ACK• Vhodny ak treba vysoku priepustnost (100k+/sec)*

Page 20: Going event drive + Kafka a RabbitMQ

20

Event-driven(ness)Umoznuje navrhovat systemy, kde volne previazane komponenty (asynchronne) komunikuju prostrednictvom sprav a takyto dizajn vediet k implementacii ktora zjedodusuje rozsirovanie a spravu systemu.

Pouzitie je teda na distribuciu uloh (routing), spravu (management) systemu, transformaciu* a kontrolu (monitoring).

Asynchronicita umoznuje efektivne zdielat prostriedky “jedneho HW vlakna” / vypoctovej jednotky resp. komunikacneho kanala. Non-blocking vedie k nizsej latencii a vyssej priepustnosti. Konkurentnost priamo v dizajne.

Page 21: Going event drive + Kafka a RabbitMQ

21

Otázky?

deh'-ku-yemza pozornost

a chlapcom z mrkvosoftu za to ze powerpoint NEMA autosave recovery-save sa nepocita!