message queues

Message Queues (RabbitMQ, CloudAMQP, AmazonSNS,

Stormmq, ActiveMQ, …)By Ahmad Karawash

7/4/2016 1

List of Some Message Queues (MQ)

I. RabbitMQ

II. CloudAMQP (RabbitMQ as a service)

III. Amazon SNS

IV. Stormmq (MQ as a service)

V. Apache ActiveMQ

VI. SwiftMQ

VII. Sparrow

VIII.Starling

IX. Kestrel

X. Kafka (used by linkedin) 7/4/2016 2

I- RabbitMQ patterns

7/4/2016 4

II- CloudAMQP (RabbitMQ as a service)

• CloudAMQP are managed RabbitMQ servers in the cloud

• Hosted message queues that lets you pass messages between processes and other systems.

• Messages are published to a queue by a producer, the consumers can then get the messages off the queue when the consumer wants to handle the messages. In-between, it can route, buffer, and persist the messages according to rules you give it.

• Messages can be sent cross languages, platforms and OS, this way of handling messages decouple your processes and creates a highly scalable system.

7/4/2016 5

II- CloudAMQP instances

7/4/2016 6

II- CloudAMQP Console

7/4/2016 7

II- RabbitMQ Management Interface

7/4/2016 8

III- Amazon Simple Notification Service (Amazon SNS)

• Send push messages, Deliver messages instantly to applications or users and eliminate polling in your apps.

• Scale as your needs grow, Start fast with three simple APIs, and deliver an unlimited number of messages.

• Use direct-addressing to send messages to individual devices or broadcast to multiple destinations at-once.

• Use your choice of platforms, Directly access SNS using iOS, Android, Java, Python, PHP, Node.js, or .NET SDKs from AWS.

• Deliver worldwide and across multiple protocols, Send notifications to Apple iOS, Android and other mobile devices, and destinations like Amazon SQS queues, Lambda functions, email addresses, SMS, and HTTP endpoints.

• Easily connect with other AWS services, Use SNS as a message bus to send messages, alarms, and notifications from your AWS services such as Amazon RDS, CloudWatch, and S3 to other AWS services such as SQS and Lambda.

• Get message delivery analytics, Get delivery status information via Amazon CloudWatch on success rates, failure rates, and dwell times for mobile push messages as well as deliveries to SMS, SQS, HTTP, and Lambda destinations.

• Usage-based pricing, You pay only for what you use with SNS. $0.50 per million publishes.

7/4/2016 9

III- How SNS works

There are two types of clients: publishers and subscribers.

• Publishers communicate asynchronously with subscribers by producing and sending a message to a topic, which is a logical access point and communication channel.

• Subscribers (i.e., web servers, email addresses, Amazon SQS queues, AWS Lambda functions) consume or receive the message or notification over one of the supported protocols (i.e., Amazon SQS, HTTP/S, email, SMS, Lambda) when they are subscribed to the topic.

7/4/2016 10

IV- Stormmq (MQ as a service)

• Free. As In Beer: Send and receive messages absolutely free for up to 5 connected clients. No sign up charges. No subscriptions. Unless you want more.

• Unlimited Monthly Messages: Send. Store. Receive. Any size. Any number: 1 or 1,000,000,000. Just pay for the storage you need. Consume messages quickly? Then storage is free.

• Internet Scale: broker scales well beyond the C10K limit of traditional MQ technology, with no sync locks, and up to 850,000 messages / sec for ultimate users. Designed for today’s 48 core hydras.

• PostgreSQL Admin Interface

• Store & Forward

• Never lose a message. Acknowledgments, partial re-delivery, the works.

• Follows Publish & Subscribe

• Secure by Default7/4/2016 11

V- Apache ActiveMQ (Apollo)

• Multi-protocol broker and supports STOMP, AMQP, MQTT, Openwire, SSL, and WebSockets.

• Queue Browsers

• Durable Subscriptions on Topics

• Mirrored Queues

• Reliable Messaging

• Message Expiration, Message Swapping, Message Selectors, Message Groups

• JAAS Authentication

• ACL based Authorization

• SSL/TLS Support and Certificate based Authentication

• REST Management API

7/4/2016 12

VI- SwiftMQ

• High Availability JMS Router with ACTIVE/STANDBY deployment and active replication.

• Transparent JMS and JNDI failover, no extra reconnect code required.

• Automatic and transparent resolve of all open transactions in transit, including XA transactions, as well as any non-transacted acknowledge mode.

• Full inbound and outbound duplicate message detection. Messages are never delivered twice.

• No persistent messages lost during failover. Message sequence is guaranteed.

• Choice of 3 different store types: Replicated File Store, Shared File Store, Shared JDBC Store.

• Transparent reconnect from SwiftMQ Explorer, CLI, CLI Admin API.

• Easy conversion to High Availability by HA Wizard. Turns any existing configuration into High Availability.

• High Availability test suite included. Test your JMS applications under a continuous failover scenario.

• Additional Swiftlets included at no extra cost: JMS XA/ASF Swiftlet, Routing/Unlimted Swiftlet, Standard File Store Swiftlet, JDBC Store Swiftlet.

7/4/2016 13

VII- Sparrow

• fast lightweight queue written in Ruby that speaks memcache. That means you can use Sparrow with any memcached client library (Ruby or otherwise).

• keeps messages in memory, but persists them to disk, using Sqlite, when the queue is shutdown.

• Basic tests shows that Sparrow processes messages at a rate of around 1660 per second. The load Sparrow can cope with increases exponentially as you add to the cluster.

• Sparrow's speed greatly increases when using the included client.

• comes with built in support for daemonization and clustering.

• If the Sparrow servers aren't available, queuing can fallback to Amazon SQS

7/4/2016 14

VIII- Starling

• Starling is a powerful but simple messaging server that enables reliable distributed queuing with an absolutely minimal overhead.

• It speaks the MemCache protocol for maximum cross-platform compatibility.

• Any language that speaks MemCache can take advantage of Starling's queue facilities.

• Only tested with Memcache client 1.7.x and with SystemTimer (systemtimer.rubyforge.org/)

• Starling is “slow” as far as messaging systems are concerned. In practice, it's fast enough.

7/4/2016 15

http://systemtimer.rubyforge.org/

IX- Kestrel

• Fast, It runs on the JVM so it can take advantage of the hard work people have put into java performance.

• Small, currently about 2500 lines of scala, because it relies on Netty (a rough equivalent of Danger's ziggurat or Ruby's EventMachine) -- and because Scala is extremely expressive.

• Durable, Queues are stored in memory for speed, but logged into a journal on disk so that servers can be shutdown or moved without losing any data.

• Reliable, A client can ask to "tentatively" fetch an item from a queue, and if that client disconnects from kestrel before confirming ownership of the item, the item is handed to another client. In this way, crashing clients don't cause lost messages.

• It is not strongly ordered, While each queue is strongly ordered on each machine, a cluster will appear "loosely ordered" because clients pick a machine at random for each operation. The end result should be "mostly fair".

• It is not Transactional, This is not a database. Item ownership is transferred with acknowledgement, but kestrel does not support grouping multiple operations into an atomic unit.

7/4/2016 16

X- Kafka

• It is a distributed publish-subscribe messaging system.

• Current usage of Kafka at LinkedIn includes:

• Collecting log data (e.g., frontend activity events and service metrics) and loading them into Hadoop/DWH for offline reporting and ad hoc analysis.

• Feeding log data directly to online applications, such as security, firehose (feeding Signal and LinkedIn Today), wotc, and realtime service dashboard.

• Deploying derived data (e.g., customized emails and connection strength scores for Cloud) from Hadoop to production.

• Automatic load balancing

• Asynchronous send

• Semantic partitioning7/4/2016 17

?@:

[email protected]

7/4/2016 18

message queues

Technology