eugene shakhsuvarov - improving enterprise store scalability using amqp and asynchronous operations
TRANSCRIPT
© 2016 Magento, Inc. Page | 1
© 2016 Magento, Inc. Page | 2
Improving Enterprise Store Scalability with Queues
© 2016 Magento, Inc. Page | 3
Message Queue Framework in Magento EEOverview of capabilities
© 2016 Magento, Inc. Page | 4
What are Queue advantages?• Execute tasks in background process• Asynchronously, with multiple workers• Take advantage of RabbitMQ benefits:
– High Availability– Scalability– Load Balancing– Postponed execution– Platform Agnostic– Open Source
© 2016 Magento, Inc. Page | 5
What does Magento EE add to this?• Provides support for queuing with the
simple API• Transparent workflow• Takes care of topology with just xml
configurations• Great way to eliminate blocking
requests• Fallback queue solution based solely
on MqSQL for testing and specific applications
© 2016 Magento, Inc. Page | 6
Basic Concepts• Message is an atomic unit of processing• Message exists in queue until explicitly deleted• Never sent more then once
• Always sent with specific topic• Consumers get appropriate messages by
subscribing to the desired topic
© 2016 Magento, Inc. Page | 7
Basic Workflow
© 2016 Magento, Inc. Page | 8
Publisher• Publishes message data to the specific exchange• Data may be represented as an arbitrary array or DTO• Configured with xml configuration• Simple interface
© 2016 Magento, Inc. Page | 9
Consumer• Consumes message from the queue• Passes it to the configured handler class for processing• Again, simple xml configuration
© 2016 Magento, Inc. Page | 10
Batch Consumer• For the cases where lots of messages are involved• Consumes a batch of multiple messages of configured size• Allows handler to process whole batch in a loop
© 2016 Magento, Inc. Page | 11
Configuration• Queues configuration happens in a few files inside the module’s
“etc” directory.• “communication.xml” is responsible for aspects of the message
queue system that all communication types have in common. AMQP and Database connections. Values may be overridden for deployment in “env.php”.
• “queue.xml” is responsible for defining the broker that processes the topic.
• “queue_consumer.xml” responsible for configuring a consumer, allowing to set it’s type and handler for messages.
• “queue_publisher.xml” configures publisher connection to the message broker.
© 2016 Magento, Inc. Page | 12
Are there any disadvantages?• Concurrent execution of similar actions
may lead to locks in specific scenarios• No guarantee on the timeframe, when
the task will be processed
• Generally, using a single worker decreases overall performance:– Using Multiple workers improves
performance, but only up to a certain grade.
© 2016 Magento, Inc. Page | 13
Asynchronous Operations ModuleYes, it gets even better.
© 2016 Magento, Inc. Page | 14
Asynchronous Operations in Magento EE• Significantly improves “Look and Feel” for the merchant• Allows Store Admin to send entity updates to the queue for
background processing eliminating long blocking requests• Provides a UI for tracking status of the execution and logging• Allows to retry certain failed operations• Built on top of the Message Queue Framework
© 2016 Magento, Inc. Page | 15
Asynchronous Operations
© 2016 Magento, Inc. Page | 16
Magento Asynchronous Operations Workflow• Create Operation per entity to be processed• Form bulk of operations• Post messages to the Queue• Handle operations and their statuses from the Queue
Operation
Operation
Operation
Queue Handler UI
© 2016 Magento, Inc. Page | 17
Operation• Operation is a single unit of work to be
processed• Framework interprets it as a simple DTO
containing:– Operation UUID– Name of the destination topic in Queue– Serialized data or DTO for processing– Operation Status and Status Message
• Recoverably Failed operations may be retried in a separate bulk
© 2016 Magento, Inc. Page | 18
Operation Statuses• Operations can be in one of the 4 possible statuses:
– Open: still in queue– Complete: processed successfully without errors– Failed Recoverably: based on exception type it is possible to determine
if operation is worth retrying. For example a connection, wait or lock exceptions, which, in some cases, may happen when dealing with the large number of similar items
– Failed Non Recoverably: for cases when operation is failed due to reasons which require manual interaction, for example validation errors
• Operation status is accompanied by Result Message, visible to the store admin, which may contain more detailed information on the problem
© 2016 Magento, Inc. Page | 19
Bulk Management• Bulk is an array of operations,
identified by Unique ID• Visible as a separate entry to the
admin user who created it, along with the count and status of operations within it
• Every Operation is published as a separate message under a desired topic name
• Bulk Management provides a simple API to publish Operations into the Queue
© 2016 Magento, Inc. Page | 20
How does it look for a store admin?
© 2016 Magento, Inc. Page | 21
Quick Recap
© 2016 Magento, Inc. Page | 22
Quick recap• Queue is good for performing long operations in background• Great way to avoid blocking requests• Scales well with the number of workers, however has its limits• Simple API on Magento's side
© 2016 Magento, Inc. Page | 23
Example Module• You may take a look at the example module, which using some of
the discussed features updates Catalog Product Stock Quantities
• https://tinyurl.com/mmit17-queues• https://github.com/ishakhsuvarov/mmit17-queue-example
© 2016 Magento, Inc. Page | 24
Q&AFor more questions: @ishakhsuvarov(twitter, github or @magento.com email)