scaling webappswithrabbitmq
DESCRIPTION
Talk given at the ECUG Conference in Beijing, October 17th. 2010TRANSCRIPT
![Page 1: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/1.jpg)
Scaling Web Apps With RabbitMQ
Alvaro Videla - ECUG Con 2010
Thursday, October 21, 2010
![Page 2: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/2.jpg)
About Me
• Development Manager at TheNetCircle.com
• Twitter: @old_sound
• Blog: http://videlalvaro.github.com/
• 老外
Thursday, October 21, 2010
![Page 3: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/3.jpg)
Agenda
• RabbitMQ
• AMQP
• Scaling Web Apps
• High Availability
Thursday, October 21, 2010
![Page 4: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/4.jpg)
RabbitMQ
Thursday, October 21, 2010
![Page 5: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/5.jpg)
RabbitMQ
• Enterprise Messaging System
• Open Source MPL
• Written in Erlang/OTP
• Commercial Support
Thursday, October 21, 2010
![Page 6: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/6.jpg)
Features
• Reliable and High Scalable
• Easy To install
• Easy To Cluster
• Runs on: Windows, Solaris, Linux, OSX
• AMQP 0.8 - 0.9.1
Thursday, October 21, 2010
![Page 7: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/7.jpg)
Client Libraries
• Java
• .NET/C#
• Erlang
• Ruby, Python, PHP, Perl, AS3, Lisp, Scala, Clojure, Haskell
Thursday, October 21, 2010
![Page 8: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/8.jpg)
Docs/Support
• http://www.rabbitmq.com/documentation.html
• http://dev.rabbitmq.com/wiki/
• #rabbitmq at irc.freenode.net
• http://www.rabbitmq.com/email-archive.html
Thursday, October 21, 2010
![Page 9: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/9.jpg)
AMQP
Thursday, October 21, 2010
![Page 10: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/10.jpg)
AMQP
• Advanced Message Queuing Protocol
• Suits Interoperability
• Completely Open Protocol
• Binary Protocol
• AMQP Model
• AMQP Wire Format
Thursday, October 21, 2010
![Page 11: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/11.jpg)
AMQP Model
• Exchanges
• Message Queues
• Bindings
• Rules for binding them
Thursday, October 21, 2010
![Page 12: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/12.jpg)
AMQP Wire Protocol
• Functional Layer
• Transport Layer
Thursday, October 21, 2010
![Page 13: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/13.jpg)
Message Flow
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/chap-Messaging_Tutorial-Initial_Concepts.html
Thursday, October 21, 2010
![Page 14: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/14.jpg)
Exchange Types
• Fanout
• Direct
• Topic
Thursday, October 21, 2010
![Page 15: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/15.jpg)
Default Exchanges
• amqp.fanout
• amqp.direct
• amqp.topic
Thursday, October 21, 2010
![Page 16: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/16.jpg)
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts-Fanout_Exchange.html
Thursday, October 21, 2010
![Page 17: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/17.jpg)
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts-Direct_Exchange.html
Thursday, October 21, 2010
![Page 18: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/18.jpg)
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts-Topic_Exchange.html
Thursday, October 21, 2010
![Page 19: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/19.jpg)
Scaling Web Apps
Thursday, October 21, 2010
![Page 20: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/20.jpg)
What means “to Scale”
• Scale Up
• Scale Down
• Scale to New Product Requirements
Thursday, October 21, 2010
![Page 21: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/21.jpg)
Scenario I
Batch Processing
Thursday, October 21, 2010
![Page 22: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/22.jpg)
Requirements
Thursday, October 21, 2010
![Page 23: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/23.jpg)
Requirements
• Generate XML
Thursday, October 21, 2010
![Page 24: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/24.jpg)
Requirements
• Generate XML
• Distribution Over a Cluster
Thursday, October 21, 2010
![Page 25: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/25.jpg)
Requirements
• Generate XML
• Distribution Over a Cluster
• Elasticity - Add/Remove new workers
Thursday, October 21, 2010
![Page 26: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/26.jpg)
Design
Thursday, October 21, 2010
![Page 27: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/27.jpg)
Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false, true, false);
$msg = new AMQPMessage($video_info, array('content_type' => 'text/plain',
'delivery_mode' => 2));
$channel->basic_publish($msg, 'video-desc-ex');
$channel->close();$conn->close();
Thursday, October 21, 2010
![Page 28: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/28.jpg)
Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();
$channel->exchange_declare('video-desc-ex', 'direct', false, true, false);
$channel->queue_declare('video-desc-queue', false, true, false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
$channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer);
while(count($channel->callbacks)) { $channel->wait();}
Thursday, October 21, 2010
![Page 29: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/29.jpg)
Scenario II
Upload Pictures
Thursday, October 21, 2010
![Page 30: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/30.jpg)
Requirements
Thursday, October 21, 2010
![Page 31: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/31.jpg)
Requirements
• Upload Picture
Thursday, October 21, 2010
![Page 32: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/32.jpg)
Requirements
• Upload Picture
• Reward User
Thursday, October 21, 2010
![Page 33: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/33.jpg)
Requirements
• Upload Picture
• Reward User
• Notify User Friends
Thursday, October 21, 2010
![Page 34: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/34.jpg)
Requirements
• Upload Picture
• Reward User
• Notify User Friends
• Resize Picture
Thursday, October 21, 2010
![Page 35: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/35.jpg)
Design
Thursday, October 21, 2010
![Page 36: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/36.jpg)
Publisher Code$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();
$channel->exchange_declare('upload-pictures', 'fanout', false, true, false);
$metadata = json_encode(array('image_id' => $image_id,'user_id' => $user_id,‘image_path' => $image_path));
$msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2));
$channel->basic_publish($msg, 'upload-pictures');
$channel->close();$conn->close();
Thursday, October 21, 2010
![Page 37: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/37.jpg)
Consumer Code$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);
$channel->queue_declare('resize-picture', false, true, false, false);
$channel->queue_bind('resize-picture', 'upload-pictures');
$channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer);
while(count($channel->callbacks)) {$channel->wait();
}
Thursday, October 21, 2010
![Page 38: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/38.jpg)
Consumer Code
$consumer = function($msg){$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->
basic_ack($msg->delivery_info['delivery_tag']);
if($msg->body == 'quit'){$msg->delivery_info['channel']->
basic_cancel($msg->delivery_info['consumer_tag']);}
};
Thursday, October 21, 2010
![Page 39: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/39.jpg)
Scenario III
Distributed Logging
Thursday, October 21, 2010
![Page 40: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/40.jpg)
Requirements
Thursday, October 21, 2010
![Page 41: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/41.jpg)
Requirements
• Several Web Servers
Thursday, October 21, 2010
![Page 42: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/42.jpg)
Requirements
• Several Web Servers
• Logic Separated by Module/Action
Thursday, October 21, 2010
![Page 43: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/43.jpg)
Requirements
• Several Web Servers
• Logic Separated by Module/Action
• Several Log Levels:
Thursday, October 21, 2010
![Page 44: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/44.jpg)
Requirements
• Several Web Servers
• Logic Separated by Module/Action
• Several Log Levels:
• Info
• Warning
• Error
Thursday, October 21, 2010
![Page 45: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/45.jpg)
Design
Thursday, October 21, 2010
![Page 46: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/46.jpg)
Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();
$channel->exchange_declare('logs', 'topic', false, true, false);
$msg = new AMQPMessage('some log message',array('content_type' => 'text/plain'));
$channel->basic_publish($msg, 'logs', server1.user.profile.info');
$channel->close();$conn->close();
Thursday, October 21, 2010
![Page 47: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/47.jpg)
Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();
$channel->exchange_declare('logs', 'topic', false, true, false);
$channel->queue_declare('server1-logs', false, true, false, false);
$channel->queue_bind('server1-logs', 'logs', 'server1.#');
Thursday, October 21, 2010
![Page 48: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/48.jpg)
Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();
$channel->exchange_declare('logs', 'topic', false, true, false);
$channel->queue_declare('error-logs', false, true, false, false);
$channel->queue_bind('error-logs', 'logs', '#.error');
Thursday, October 21, 2010
![Page 49: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/49.jpg)
One Setup for HA
Thursday, October 21, 2010
![Page 50: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/50.jpg)
Questions?
Thursday, October 21, 2010
![Page 51: Scaling webappswithrabbitmq](https://reader036.vdocuments.us/reader036/viewer/2022062418/554fae75b4c905ad218b50c4/html5/thumbnails/51.jpg)
Thanks!Alvaro Videla
http://twitter.com/old_sound
http://github.com/videlalvaro
http://github.com/tnc
http://www.slideshare.net/old_sound
Thursday, October 21, 2010