devoxxfr 2016 - 3 degrees of mom
Post on 13-Jan-2017
143 Views
Preview:
TRANSCRIPT
#DevoxxFR
3 degrees of MoM kafka, rabbitmq, nsq
Guillaume Arnaud @guillarnaud
voyages-sncf.com
1
#DevoxxFR 2
P publisher
C consumer
A Message-oriented Middleware (MoM)just a queue ?
#DevoxxFR 3
ActiveMQ RabbitMQ Nats NSQ ZeroMQ Kafka AmazonSQS Disque Aeron Mosquitto
Beanstalk Apollo Nanomsg …
JMS Stomp MQTT AMQP XMPP …
Flume Redis Logstash Syslog …
Brokers Protocol/API Almost
https://github.com/mfornos/awesome-microservices#messaginghttp://queues.io/
#DevoxxFR 4
ActiveMQ RabbitMQNats NSQZeroMQ KafkaAmazonSQS Disque Aeron Mosquitto
Beanstalk Apollo Nanomsg …
JMS Stomp MQTT AMQP XMPP …
Flume Redis Logstash Syslog …
Brokers Protocol/API Almost
https://github.com/mfornos/awesome-microservices#messaginghttp://queues.io/
ROUTING
DECENTRALIZED
DISTRIBUTED
#DevoxxFR
RabbitMQ.id
5
parents: Pivotal age: v3.6.1 (march 2016) license: mozilla public license native languages: erlang/amqp fluently spoken: python, java, ruby, .Net, nodejs big brother: activemq
#DevoxxFR 6
P
C
decouplingexchange queue routing/binding keys
C
exchange
routing key: devoxx.2016.newuser
binding key: devoxx.2016.*
routing
queue: devoxx2016_service
#DevoxxFR 7
P
C C
exchange: fanout
routing key: devoxx.2016.newuser
binding key: devoxx.2016.*
message 1
fanoutbroadcast no routing rules
message 1 message 1
#DevoxxFR 8
P
C C
exchange: direct
routing key: devoxx.2016.newuser
binding key: devoxx.2016.newuser
message 1
directrouting key=binding key
message 1
queue: new_user_servicequeue: update_user_service
binding key: devoxx.2016.updateuser
#DevoxxFR 9
P
C C
exchange: topic
routing key: devoxx.2016.newuser
binding key: devoxx.2016.*
message 1
message 1 message 1
queue: devoxx_servicequeue: trigger_new_user
binding key: devoxx.2016.newuser
topicwildcard subscription
#DevoxxFR
RabbitMQ.model
10
exchange routing key
queue service
event type
binding key aggregator
business domain
#DevoxxFR
RabbitMQ.microservices
11
exchange'user' / fanout
cv.*
microservicesrouting between exchanges and services
specify_user_action
audit
check_fraud
exchange'cv' / topic
alert_recruiters
crop_photo
cv.photo
cv.employer.new
check_employer
exchange'mailer' / direct
generate_and_send_mail
#DevoxxFR
RabbitMQ.ops
12
• virtual hosts / security • plugins: management, ldap, mqtt, consistent hashing • cluster/federation • queue size/disk • cpu
#DevoxxFR
RabbitMQ.admin
13
$ rabbitmq-plugins enable rabbitmq-management
#DevoxxFR
Kafka.id
14
parents: LinkedIn/confluent age: v0.9.0.1 (feb. 2016) license: apache native language: java/scala family: hdfs, spark streaming, storm, flume
#DevoxxFR 15
P publisher
C consumer
5 4 3 2 1
offset
lag
log file(s)append only zero copy ~ k*100 000 msg/s
#DevoxxFR 16
P
C
5 4 3 2 1
Partitionsplit the topic partition key
4 3 2 1
keys 1, 3, 5… keys 2, 4, 6…
#DevoxxFR 17
P
C
distribution
5 4 3 2 1
4 3 2 1
4 3 2 1
4 3 2 1
node 1 node 2
#DevoxxFR 18
replication
5 4 3 2 1
4 3 2 1
4 3 2 1
P
replica lag
partition leader
partition follower
partition follower
#DevoxxFR 19
P
C
5 4 3 2 1
4 3 2 1
C
Scalingone consumer per partition add partitions for scaling
#DevoxxFR 20
P
C
5 4 3 2 1
4 3 2 1
C
Scalingone consumer per partition add partitions for scaling
C
#DevoxxFR 21
P
C C
Scalingone consumer per partition add partitions for scaling
C
#DevoxxFR 22
P
C C
Groupconsume same messages ~ JMS topic C
group 1
group 2
#DevoxxFR
Kafka.event sourcing
23
domain object
domain object
eventview 1
view 2event store
#DevoxxFR
Kafka.event sourcing
24
1 id:1234 my order (…)
2 id:abcd an other order
3 id:wxyz and again…
120 id:1234 my other order
…
log compaction
offset key payload
#DevoxxFR
Kafka.ops
25
• clients are part of the distributed system (heartbeat, tcp config, etc…)
• lag sender/consumer • replica lags • preferred replica election • repartition master/slave • don't forget Zookeeper • size and time retention • use kafka tools
~ equivalent to a NoSQL database
properties #
server 127
publisher 47
consumer 45
#DevoxxFR
Kafka.admin
26
https://github.com/yahoo/kafka-manager
#DevoxxFR
NSQ.id
27
parents: nsq.io age: v0.3.7 (feb. 2016) license: open source native language: go fluently spoken: go, python, javascript, http(!) zodiacal sign: syslog++
#DevoxxFR 28
P
C
topic: server_start_eventnsqd
channel: server_subscribe
topic: server_start_eventnsqd
channel: server_subscribe
P
consumer: server_subscribe
Cnsqlookupd
http http
tcptcp
#DevoxxFR 29
server
nsqd
server
nsqd
server
nsqd
server
nsqd
push
pull
decentralizedlocal push remote pull
#DevoxxFR 30
nsqd
nsqd
nsqd
haproxy slave
daemon
reload
admin
tomcat
new_ip_port1
2
3
5
6
7
use caseservice discovery
nsqd
tomcat
#DevoxxFR
NSQ.service discovery
31
nsqd
nsqd
nsqd
haproxy slave
daemon
reload
admin
tomcat
new_ip_port1
2
3
5
6
7
nsqd
tomcat
#DevoxxFR
NSQ.ops
32
$ tree bin/ bin/ ├── nsq_pubsub ├── nsq_stat ├── nsq_tail ├── nsq_to_file ├── nsq_to_http ├── nsq_to_nsq ├── nsqadmin ├── nsqd ├── nsqlookupd └── to_nsq
#DevoxxFR
NSQ.ops
33
$ nsqd —help -e2e-processing-latency-percentile value … -max-bytes-per-file int -max-heartbeat-interval duration -max-msg-size int -max-msg-timeout duration -max-output-buffer-size int -max-output-buffer-timeout duration -max-rdy-count int -max-req-timeout duration -mem-queue-size int -msg-timeout string -snappy …
#DevoxxFR
MoM.diversity
34
#DevoxxFR
NSQ.admin
35
http://nsq.io/components/nsqadmin.html
#DevoxxFR
your MoM is a part of your application
36
#DevoxxFR
your application is a part of your distributed system
37
#DevoxxFR 38
http://www.meetup.com/fr-FR/Meetup-Messaging-Paris/
#DevoxxFR
Questions ?
39
#DevoxxFR
Thanks
40
top related