improvements in rabbitmq
TRANSCRIPT
Improvements in RabbitMQ Performance and Future Directions
Alvaro Videla - RabbitMQ
@old_sound
github.com/videlalvaro
What is RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ• Multi Protocol Messaging Server
• Multi Protocol Messaging Server• Open Source (MPL)
RabbitMQ
• Multi Protocol Messaging Server• Open Source (MPL)• Polyglot
RabbitMQ
• Multi Protocol Messaging Server• Open Source (MPL)• Polyglot• Written in Erlang/OTP
RabbitMQ
Multi Protocol
http://bit.ly/rmq-protocols
http://www.rabbitmq.com/community-plugins.html
Community Plugins
Polyglot
Polyglot• Java• node.js• Erlang• PHP• Ruby• .Net• Haskell
Polyglot
Even COBOL!!!11
Some users of RabbitMQ
• Indeed.com
• Telefonica• Mercado Libre
• NHS• Mozilla
Some users of RabbitMQ
The New York Times on
This architecture - Fabrik - has dozens of RabbitMQ instances spread across 6 AWS zones in Oregon and Dublin.
Upon launch today, the system autoscaled to ~500,000 users. Connection times remained flat at ~200ms.
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2014-January/032943.html
http://www.rabbitmq.com/download.html
Unix - Mac - Windows
Messaging with RabbitMQ
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator/RabbitMQSimulator
RabbitMQ Simulator
Performance Improvements
The Problem
The Problem
./runjava.sh com.rabbitmq.examples.PerfTest \-e exchange_name -t topic -u queue_name \-f persistent -r 5000 -s 1000 -x1 -y0
The Problem
Publish 5000 1kb msgs/sec without consumers
Be able to sustain publish rate during one hour
The Problem
Big throughput drop around 1 minute into the benchmark
The Problem
As soon as Queue Paging kicked inPerformance Dropped
Culprits
• queue_index flush journal
• messages published into the queue index when paging
• memory leaks related to refc binaries
Flushing the QI Journal
• Sparse Array by Message SeqIds
• Publishes, Delivers and Acks
• Queue Index is written in Append Only Segments
• RabbitMQ keeps an in memory Journal of ops to prevent going to disk all the time
Flushing the QI Journal
queue_index_max_journal_entries = 65536
Flushing the QI Journal
entry_to_segment/3 called 65536 times
Flushing the QI Journal
entry_to_segment/3 called 65536 times
Appending data to a binary and then writing that to disk
Solution
• Keep a Sparse Array as a cache of “entry to segments”
• Flush said array to disk whenever the Journal needs to be flushed
Solution
array:to_list(EntriesToSegment).
Solution
array:to_list(EntriesToSegment).
What happens with blank elements?
Solution
array:new([{default, []}, fixed, {size, ?SEGMENT_ENTRY_COUNT}]).
Messages Publishes into the Queue Index
queue_index_embed_msgs_below = 4096
Messages Publishes into the Queue Index
Messages Publishes into the Queue Index
• Implemented Batch Publishing into the Queue Index
• Batch Handling of Delivers and Acks
• Flush Publishes, Delivers and Acks cache every 20.000 messages or when paging finished.
Queue Paging
Alphas -> Betas -> Deltas
binary memory leak• Related to Paging
• discovered with recon
• forced queue process garbage collection after paging
http://ferd.github.io/recon/index.html
The Problem II
The Problem II
Mirror Queue Synchronisation
“is too slow”
The Problem II
Initial tests showed that it took 60 seconds to sync 1MM messages
The Problem II
Original implementation was synching one message at a time
Solution II
Why not implement batch publishing at the queue level?
Batch Publishing• Implemented for Backing Queue Implementations
• rabbit_variable_queue
• rabbit_priority_queue
• rabbit_mirror_queue_master
• rabbit_mirror_queue_slave
Solution II
Initial tests showed that it took 6 seconds to sync 1MM messages
The Problem III
Queues that support offline consumers for 24 hours or more
Solution III• Lazy-Queue concept
• Disable in-memory message cache
• Only load messages in RAM if consumers are present
• Implement Lazy Queues for all Backing Queue implementations
Lazy Queues
https://github.com/rabbitmq/rabbitmq-server/issues/351
Initial tests show stable throughput comparable to “default” queues with persistent messages
Lazy Queues
https://github.com/rabbitmq/rabbitmq-server/issues/351
10MM of 1KB use 1.5MB of RAM vs
1.2GB of RAM for default queues
Lazy Queues
https://github.com/rabbitmq/rabbitmq-server/issues/351
40MB of RAM when a consumer is active
Other Improvements
• moved to erlang.mk
• moving to Ranch/Cowboy and related libraries
• improved management UI performance
• improved rabbitmqctl performance
Other Improvements
• pluggable hashing algorithm for passwords
• queue master location strategies
Future
• add Raft based queue mirroring
• add Raft based cluster formation
• Re-thing rabbitmqctl to allow for extensions
• Tools to recover data from disk
Thanks
Alvaro Videla - RabbitMQ