a tale of queues — from activemq over hazelcast to disque - philipp krenn
TRANSCRIPT
A tale of queuesfrom ActiveMQ over Hazelcast to Disque
Philipp Krenn��� ���@xeraa
Vienna
Vienna
Vienna
ViennaDBPapers We Love Vienna
Electronic Data Interchange EDI
AutomotiveRetail
Transmission & message flowAS2, X.400, OFTP2, FTP, SMTP, IMAP,
web services,...
Document formatEDIFACT, X12, ODETTE, VDA,...
Architecture +--------+ | | +----+ | +--------+ | | | | | | | | | | +--------+ v + | | | | | | +------------> Processing +-|-|----+ | | + | | | | | +-|------+ | | | | <--------------+ +--------+ Connectors
Flow Camel component / protocol implementation + ^ | | | | v + Camel routing + ^ | | | | v + Queue + ^ | | | | v + Processing
QueuesDecoupling producer and consumer
Networked message queues like ActiveMQ, RabbitMQ, ZeroMQ, and a host of other Java inspired software
tumors are crutches of systems design.— Ted Dziuba, http://widgetsandshit.com/teddziuba/2011/02/the-case-
against-queues.html
"The Blocking Consumer"
"Collecting Data for Offline Processing"
"Everybody Loves System Complexity"
There are only two hard problems in distributed systems:
2. Exactly-once delivery1. Guaranteed order of messages
2. Exactly-once delivery— Mathias Verraes, https://twitter.com/mathiasverraes/status/
632260618599403520
OrderIt works like it would on a single
machinehttp://book.mixu.net/distsys/time.html
Exactly once
Removal / ACKBefore processing: at-most-onceA!er processing: at-least-once
At-most-onceRequires strict consistency with
communication overheadReplicated messages and liveness
Network split — majority
At-least-onceEasier to distribute
Idempotent consumptionTimeout tuning
CheatingOrder
Cheating1. Receive
2a. Persist in RDS (MySQL)2b. Camel routing + queue
3. Acknowledge
GoalsReliabilityLiveness
At-most-once
MachinesWaitRetry
Amazon Simple Queue Service (SQS)
Fast, reliable, scalable, and fully managed
But...At-least-once
64KB (now 256KB) payload
Master SlaveShared file system
JDBCReplicated LevelDB
+ ZooKeeper
<bean id="mysql-ds" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close"> <property name="driverClass" value="com.mysql.jdbc.Driver"/> <property name="jdbcUrl" value="jdbc:mysql://{{rds_hostname}}: 3306/{{rds.activemq.database}}?relaxAutoCommit=true"/> <property name="user" value="{{rds.activemq.user}}"/> <property name="password" value="{{rds.activemq.pass}}"/> <property name="acquireRetryAttempts" value="100"/> <property name="testConnectionOnCheckout" value="true"/> <property name="testConnectionOnCheckin" value="false"/> <property name="idleConnectionTestPeriod" value="3600"/> <property name="maxStatements" value="0"/> <property name="minPoolSize" value="2"/> <property name="maxPoolSize" value="20"/> <property name="maxIdleTimeExcessConnections" value="600"/> <property name="numHelperThreads" value="6"/> <property name="acquireRetryDelay" value="5000"/> <property name="maxConnectionAge" value="10800"/> <property name="preferredTestQuery" value="SELECT 1;"/></bean>
RDS down !
In-memory data gridSet, list, map, queue, topic, lock,
atomic long,...Query, aggregate, MapReduce
Hibernate 2nd level cache, session replication
Eierlegende Wollmilchsau
"egg-laying wool-milk-sow"
GreatDoes everything for Java apps
Multi-master HAFast
Bad feelingVery few queue users
Bugs with IP / DNS changesPainful upgrades
Currently our action is to create bug for these scenarios and try to
ensure exactly-once.— Enes Akar, https://groups.google.com/forum/#!msg/hazelcast/
u_KLHVnvT_U/Qx5Km8COk_oJ
KafkaReal-time focusAt-least-once
https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer
KestrelDead !
RabbitMQToo much middleware
In the event of network failure (or a node crashing), messages
can be duplicated, and consumers must be prepared to handle them.
— https://www.rabbitmq.com/reliability.html
[...] no one try to use N Redis independent nodes and the
offered primitives as a building block for a distributed system [...]
— Salvatore Sanfilippo, http://antirez.com/news/78
Disque, an in-memory, distributed job queue
— https://github.com/antirez/disque
GreatDocumentationDoes one thing
Multi-master HAConfigurable ACKSimilar to Redis
Driver: Spinach"A scalable Java Disque client" based
on lettuce, "A scalable Java Redis client"
! !Some teething problems
Current configurationPer message
Replication factor: 1Retry time: 0
No persistence
Current useBilling since 07/2015
Full ActiveMQ replacement soon?
WARNING: This is alpha code NOT suitable for production.
— https://github.com/antirez/disque
Nobody likes queues
Thank you!Questions?!
@xeraa
Image Credit· Schnitzel https://flic.kr/p/9m27wm
· Architecture https://flic.kr/p/6dwCAe· Conchita https://flic.kr/p/nBqSHT
· Paper https://flic.kr/p/83thLf· Airli! https://commons.wikimedia.org/wiki/File:C-54landingattemplehof.jpg
· Beer https://flic.kr/p/aBSmtY· Gum https://flic.kr/p/ALQ3b
· Eierlegende Wollmilchsau https://flic.kr/p/GzQTT· British https://flic.kr/p/hzxLvw