resque - files.meetup.com awesome.pdf · tangent alert: redis! scalability the gem load balances...

ResqueHigh-performance asynchronous task queuing and

processing

ResqueHigh-performance asynchronous task queuing and

processing

Experts‘Cause you might build the next GitHub

Three Things

1. BackgroundWhy run background processes at all?

2. Delayed::JobBecause it’s the foundation for Resque

3. ResqueProcess different.

1. Background

Why?

1. Background

It’s all about the ms• Resize an image

• Send an e-mail

• Communicate with a web service

• Geocode a location

1. Background

Do something later than now

1. Background

Webpages.Rails serves webpages.

Memories. You're talkin about memories.

1. Background

1. Receive requests2. Send responses

1. Background

Everything else can go elsewhere

1. Background

Where?• CRON

• Beanstalkd

• Workling

• BackgroundRB

1. Background

• Silent failure

• No status checks

• Unstable processes

WTF my tasks r fail?!1

2. Delayed::Job

To the rescue!

2. Delayed::Job

ActiveRecord::Base’d

• Persistence! At last!

• Ease of use! At last!

2. Delayed::Job

How easy?

2. Delayed::Job

# without delayed_jobNotifier.deliver_signup(@user)

# with delayed_jobNotifier.send_later :deliver_signup, @user

Super easy.

2. Delayed::Job

Where do they go?

2. Delayed::Job

mysql> SELECT * FROM delayed_jobs;*************************** 1. row *************************** id: 1 priority: 0 attempts: 0 handler: --- !ruby/object:WorkerBee <-- Notice the pretty Marshal dumpn: 18

last_error: NULL run_at: 2010-03-10 22:44:30 locked_at: NULL failed_at: NULL locked_by: NULLcreated_at: 2010-03-10 22:44:30updated_at: 2010-03-10 22:44:301 row in set (0.00 sec)

2. Delayed::Job

What do you do with them?

$ script/delayed_job start

2. Delayed::Job

Workers.

• Live in their own thread

• Query the database for jobs

• Perform the long-running taskaway from request/response

The Old Way

The New Way

Delayed::Job

2. Delayed::Job

Totally rad.

3. Resque

Isn’t Delayed::Job enough?

3. Resque

Apparently not.

3. Resque

Resque is some Enterprise Grade shit.

3. Resque

Runs on Redis

Just like Memcached

Redis is a Key-Value Store

Tangent Alert: Redis!

Tangent Alert: Redis!

PersistenceUnlike Memcached

Tangent Alert: Redis!

ScalabilityThe Gem load balances for you

Resque.redis = DistRedis.new( :hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})

Tangent Alert: Redis!

Basically:Fast & Distributed

Also, Rails-independentNo ActiveRecord/ActiveSupport means you can run Resque (on Redis) in Sinatra or any other Ruby app

Tangent Alert: Redis!

3. Resque

• Forked, sandboxed threads

• Failing tasks are tracked

• Kills timeouts

Better Workers

3. Resque

Queueing

fileserver$ QUEUE=zip_files rake resque:work webserver$ QUEUE=cache_images rake resque:work

Separate queues for separate workers

3. Resque

No MarshalingBecause it should be functional

Resque.enqueue(SignupNotifier, @user.id)

3. Resque

No Marshaling

class SignupNotifier def self.perform(user_id) Notifier.deliver_signup(User.find(user_id)) endend

3. Resque

UIThat’s right. It has its own interface.

3. Resque

• Many background tasks

• Distributed task servers

• Stable workers

So... why?

3. Resque

• Complex and expensive

• No scheduling or priority like in D::J

• Asynchronous Redis writes don’t guarantee persistence

Trouble Points

3. Resque

Live coding FTW!Here goes nothing...

ThnaksThanks for taking the time to listen to me blather

about Resque!

Me!

Github:Homepage:

Twitter:E-mail:

http://github.com/flipsasserhttp://x451.com@flipsasserflip@intridea.com

Bmore on Rails

http://www.meetup.com/bmore-on-rails/

Meetups - 2nd Tuesday#OSHN - 4th Tuesday