PLEASE HOLD: YOUR CALL IS IN A QUEUE

An introduction to queueing in Drupalby Tom Phethean

Sunday, 3 March 13

QUEUING IN 30 MINUTES...

• I hate waiting in line...

•Drupal Queue API

• Beyond Core

•Queue jumpers and other gotchas

•Questions

Sunday, 3 March 13

I HATE WAITING IN LINE...

Sunday, 3 March 13

Sunday, 3 March 13

Sunday, 3 March 13

Sunday, 3 March 13

Sunday, 3 March 13

QUEUES WILL FORM WHEN PROCESSES WITH VARIABILITY

ARE LOADED TO HIGH LEVELS OF UTILISATION

WITHOUT CONSTRAINT

Sunday, 3 March 13

SOMETHING WILL QUEUE SOMEWHERE

Sunday, 3 March 13

SOMETHING WILL QUEUE SOMEWHERE

Whether you like it or not

Sunday, 3 March 13

DO YOU WANT TO BE IN CONTROL?

Sunday, 3 March 13

DO YOU WANT TO BE IN CONTROL?

</silly_question>

<silly_question>

Sunday, 3 March 13

DRUPAL TO THE RESCUE!

Sunday, 3 March 13

CORE QUEUE API

• Two kinds of queue: reliable and non-reliable

• Provides interface to common queue functions

•MemoryQueue implements QueueInterface

• SystemQueue implements ReliableQueueInterface

• In Drupal 7 (and 8), Batch API extends SystemQueue

Sunday, 3 March 13

WHERE CAN I FIND QUEUE API?

•Drupal 7:

• ./modules/system/system.queue.inc

•Drupal 8:

• ./core/lib/Drupal/Core/Queue

Sunday, 3 March 13

QUEUE API METHODS

• createQueue()

• createItem()

• claimItem()

• releaseItem()

• numberOfItems()

• deleteItem()

• deleteQueue()

Sunday, 3 March 13

Code time.

Sunday, 3 March 13

/** * Create a queue. */function queues_create_queue() { $queue = DrupalQueue::get('my_queue'); $queue->createQueue();}

/** * Add an item to a queue. */function queues_create_queueitem() { $queue = DrupalQueue::get('my_queue');

$queue_data = array ( 'stuff' => 'test', );

return $queue->createItem($queue_data);}

/** * Check number of items on a queue. */function queues_count_queueitems() { $queue = DrupalQueue::get('my_queue');

return $queue->numberOfItems();}

/** * Process a queue item */function queues_process_queueitem() { $queue = DrupalQueue::get('my_queue');

$item = $queue->claimItem(30);

if ($item && $item->data['stuff'] == 'test') { // Success - we don't need this item anymore. $queue->deleteItem($item); } else { // Something unexpected happened, we still need this item. $queue->releaseItem($item); }}

/** * Implements hook_cron_queue_info(). */function queues_cron_queue_info() { $queues['my_queue'] = array( 'worker callback' => 'queues_process_queueitem', 'time' => 60, // Time is NOT the lease item. // Time is the amount of time cron will spend processing this queue. ); return $queues;}

/** * Implements hook_queue_info() from queue_ui module. */function queues_queue_info() { return array( 'my_queue' => array( 'title' => t('My test queue'), 'batch' => array( 'operations' => array(array('queues_batch_process', array())), 'finished' => 'queues_batch_finished', 'title' => t('Processing my test queue'), ), 'cron' => array( 'callback' =>'queues_process_queueitem', ), ), );}

Sunday, 3 March 13

/** * Create a queue. */function queues_create_queue() { $queue = DrupalQueue::get('my_queue'); $queue->createQueue();}

/** * Add an item to a queue. */function queues_create_queueitem() { $queue = DrupalQueue::get('my_queue');

$queue_data = array ( 'stuff' => 'test', );

return $queue->createItem($queue_data);}

/** * Check number of items on a queue. */function queues_count_queueitems() { $queue = DrupalQueue::get('my_queue');

return $queue->numberOfItems();}

/** * Process a queue item */function queues_process_queueitem() { $queue = DrupalQueue::get('my_queue');

$item = $queue->claimItem(30);

if ($item && $item->data['stuff'] == 'test') { // Success - we don't need this item anymore. $queue->deleteItem($item); } else { // Something unexpected happened, we still need this item. $queue->releaseItem($item); }}

/** * Implements hook_cron_queue_info(). */function queues_cron_queue_info() { $queues['my_queue'] = array( 'worker callback' => 'queues_process_queueitem', 'time' => 60, // Time is NOT the lease item. // Time is the amount of time cron will spend processing this queue. ); return $queues;}

/** * Implements hook_queue_info() from queue_ui module. */function queues_queue_info() { return array( 'my_queue' => array( 'title' => t('My test queue'), 'batch' => array( 'operations' => array(array('queues_batch_process', array())), 'finished' => 'queues_batch_finished', 'title' => t('Processing my test queue'), ), 'cron' => array( 'callback' =>'queues_process_queueitem', ), ), );}

Sunday, 3 March 13

/** * Create a queue. */function queues_create_queue() { $queue = DrupalQueue::get('my_queue'); $queue->createQueue();}

/** * Add an item to a queue. */function queues_create_queueitem() { $queue = DrupalQueue::get('my_queue');

$queue_data = array ( 'stuff' => 'test', );

return $queue->createItem($queue_data);}

/** * Check number of items on a queue. */function queues_count_queueitems() { $queue = DrupalQueue::get('my_queue');

return $queue->numberOfItems();}

/** * Process a queue item */function queues_process_queueitem() { $queue = DrupalQueue::get('my_queue');

$item = $queue->claimItem(30);

if ($item && $item->data['stuff'] == 'test') { // Success - we don't need this item anymore. $queue->deleteItem($item); } else { // Something unexpected happened, we still need this item. $queue->releaseItem($item); }}

/** * Implements hook_cron_queue_info(). */function queues_cron_queue_info() { $queues['my_queue'] = array( 'worker callback' => 'queues_process_queueitem', 'time' => 60, // Time is NOT the lease item. // Time is the amount of time cron will spend processing this queue. ); return $queues;}

/** * Implements hook_queue_info() from queue_ui module. */function queues_queue_info() { return array( 'my_queue' => array( 'title' => t('My test queue'), 'batch' => array( 'operations' => array(array('queues_batch_process', array())), 'finished' => 'queues_batch_finished', 'title' => t('Processing my test queue'), ), 'cron' => array( 'callback' =>'queues_process_queueitem', ), ), );}

Sunday, 3 March 13

/** * Create a queue. */function queues_create_queue() { $queue = DrupalQueue::get('my_queue'); $queue->createQueue();}

/** * Add an item to a queue. */function queues_create_queueitem() { $queue = DrupalQueue::get('my_queue');

$queue_data = array ( 'stuff' => 'test', );

return $queue->createItem($queue_data);}

/** * Check number of items on a queue. */function queues_count_queueitems() { $queue = DrupalQueue::get('my_queue');

return $queue->numberOfItems();}

/** * Process a queue item */function queues_process_queueitem() { $queue = DrupalQueue::get('my_queue');

$item = $queue->claimItem(30);

if ($item && $item->data['stuff'] == 'test') { // Success - we don't need this item anymore. $queue->deleteItem($item); } else { // Something unexpected happened, we still need this item. $queue->releaseItem($item); }}

/** * Implements hook_cron_queue_info(). */function queues_cron_queue_info() { $queues['my_queue'] = array( 'worker callback' => 'queues_process_queueitem', 'time' => 60, // Time is NOT the lease item. // Time is the amount of time cron will spend processing this queue. ); return $queues;}

/** * Implements hook_queue_info() from queue_ui module. */function queues_queue_info() { return array( 'my_queue' => array( 'title' => t('My test queue'), 'batch' => array( 'operations' => array(array('queues_batch_process', array())), 'finished' => 'queues_batch_finished', 'title' => t('Processing my test queue'), ), 'cron' => array( 'callback' =>'queues_process_queueitem', ), ), );}

Sunday, 3 March 13

/** * Create a queue. */function queues_create_queue() { $queue = DrupalQueue::get('my_queue'); $queue->createQueue();}

/** * Add an item to a queue. */function queues_create_queueitem() { $queue = DrupalQueue::get('my_queue');

$queue_data = array ( 'stuff' => 'test', );

return $queue->createItem($queue_data);}

/** * Check number of items on a queue. */function queues_count_queueitems() { $queue = DrupalQueue::get('my_queue');

return $queue->numberOfItems();}

/** * Process a queue item */function queues_process_queueitem() { $queue = DrupalQueue::get('my_queue');

$item = $queue->claimItem(30);

if ($item && $item->data['stuff'] == 'test') { // Success - we don't need this item anymore. $queue->deleteItem($item); } else { // Something unexpected happened, we still need this item. $queue->releaseItem($item); }}

/** * Implements hook_cron_queue_info(). */function queues_cron_queue_info() { $queues['my_queue'] = array( 'worker callback' => 'queues_process_queueitem', 'time' => 60, // Time is NOT the lease item. // Time is the amount of time cron will spend processing this queue. ); return $queues;}

/** * Implements hook_queue_info() from queue_ui module. */function queues_queue_info() { return array( 'my_queue' => array( 'title' => t('My test queue'), 'batch' => array( 'operations' => array(array('queues_batch_process', array())), 'finished' => 'queues_batch_finished', 'title' => t('Processing my test queue'), ), 'cron' => array( 'callback' =>'queues_process_queueitem', ), ), );}

Sunday, 3 March 13

That’s it!

Sunday, 3 March 13

BEYOND CORE

• Drupal Queue (backport to D6) - http://drupal.org/project/drupal_queue

• Queue UI - http://drupal.org/project/queue_ui

• Examples - http://drupal.org/project/examples

• Queue backends:

• Beanstalkd - http://drupal.org/project/beanstalkd

• STOMP - http://drupal.org/project/stomp

• Redis - http://drupal.org/project/redis_queue

Sunday, 3 March 13

CUSTOM BACKENDS

•Default queue class:

• $conf['queue_default_class'] = 'SystemQueue';

• $conf[‘queue_default_reliable_class’] = ‘SystemQueue’;

•Override queue classes:

• $conf['queue_class_{queue name}'] = 'StompQueue';

Sunday, 3 March 13

CUSTOM BACKENDS

•Default queue class:

• $conf['queue_default_class'] = 'SystemQueue';

• $conf[‘queue_default_reliable_class’] = ‘SystemQueue’;

•Override queue classes:

• $conf['queue_class_{queue name}'] = 'RedisQueue';

Sunday, 3 March 13

QUEUES AND DRUSH

• drush queue-list

• drush queue-run [queue name]

• ...use with caution: queue-run will always delete your queue item

Sunday, 3 March 13

WHEN TO QUEUE

• Remote service call triggered by user interface

• Complex business rules around data processing

•Need guaranteed success

Sunday, 3 March 13

SOME EXAMPLES

• Publishing user information to a remote CRM

• Backend order fulfilment

• Email sending (see notifications module)

Sunday, 3 March 13

GOTCHAS

Sunday, 3 March 13

MULTIPLE CONSUMERS

Sunday, 3 March 13

MULTIPLE CONSUMERS

• Identify what it is you’re processing

• Use Lock API to secure a lock on it:

• lock_acquire(‘my_id’)

• lock_release(‘my_id’)

• Check the “thing” is in a state you expect it to be

• Release or Delete from queue as appropriate

Sunday, 3 March 13

DEBUGGING YOUR QUEUE

Sunday, 3 March 13

DEBUGGING YOUR QUEUE

• watchdog() is your friend

• Log EVERYTHING

• Abstract it so you can toggle verbose logging

• Use Graylog, LogStash etc to save your database from death by logs

Sunday, 3 March 13

THINGS CAN GO WRONG...

Sunday, 3 March 13

THINGS CAN GO WRONG...

•Monitor your queue depths with Munin or Nagios

• Alert if they get too deep

• Know what the alert means (and listen to it)!

Sunday, 3 March 13

THINGS BREAK. BUT WHEN THEY DO, KNOW THAT EVERY ITEM IN YOUR QUEUE

IS A REQUEST WHICH WOULD OTHERWISE HAVE BEEN LOST.

Sunday, 3 March 13

IMPLEMENTING QUEUEING IN YOUR APPLICATION SMOOTHS THE PEAKS,

Sunday, 3 March 13

IMPLEMENTING QUEUEING IN YOUR APPLICATION SMOOTHS THE PEAKS,

MAKE THROUGHPUT PREDICTABLE,

Sunday, 3 March 13

IMPLEMENTING QUEUEING IN YOUR APPLICATION SMOOTHS THE PEAKS,

MAKE THROUGHPUT PREDICTABLE, REMOVES PROCESSING TIME FROM

THE USER EXPERIENCE

Sunday, 3 March 13

IMPLEMENTING QUEUEING IN YOUR APPLICATION SMOOTHS THE PEAKS,

MAKE THROUGHPUT PREDICTABLE, REMOVES PROCESSING TIME FROM

THE USER EXPERIENCE AND ENSURES THAT PROBLEMS ARE

HANDLED GRACEFULLY

Sunday, 3 March 13

IN OTHER WORDS...

Sunday, 3 March 13

Sunday, 3 March 13

QUESTIONS?

Tweet: @tsphethean Drupal.org: tsphethean

IRC: tsphetheanSunday, 3 March 13

IMAGE ATTRIBUTIONS

• Image of supermarket queue - http://petoneponderings.blogspot.co.uk/2012_12_01_archive.html

• Post Office queue - http://www.flickr.com/photos/welshkaren/5367517662/

• Apple Store queueing http://www.telegraph.co.uk/technology/apple/7854982/Apple-iPhone-4-more-than-a-quarter-of-users-think-video-calling-is-a-gimmick.html

• Supermarket crush - http://www.dailymail.co.uk/news/article-2078597/Boxing-Day-sales-Record-numbers-shops-open-80-push-grab-shoppers.html

• Drupal Superhero - http://www.origineight.net/blog-post/web-sites-you-manage

• Lots of Shoppers - http://www.dailymail.co.uk/news/article-2253523/United-Nations-Bargain-Hunters-Britons-the-queue-China-Middle-East-lead-rush-designer-brands.html

• Confused baby - http://thementalpausechronicles.blogspot.co.uk/2008/08/head-scratching.html

• Mark Sonnabaum - http://www.whatwouldmarksonnabaumdo.com

Sunday, 3 March 13