job_queues

Job Queues

Web is more than request and response

Who am I

• Co Founder / CTO Practo.com• Practo.com - simple software for doctors• 2000+ clinics, 10,000+ doctors, 1M+ patients • Cooking PHP for 5 years

Job Queues

Any long running work which does not require user feedback should be done in background.

Order Processing

Order Processing

Jobs to run in background

• Sending email• Fetching RSS feed periodically• Interacting with other application API• Complex report generation• Log analysis• cleaning file system• periodically or after certain action.

Options out there

Beanstalkd

• Open source C daemon• Fast• Persistence – bin log file• Distributed• PHP library – pheanstalk, beanstalk• Simple protocol

Beanstalkd ~ Memcached

• Beanstalkd design based on Memcache Philosophy• Simple protocol• In Memory Queue• Distributed servers handled by client

Beanstalkd Architecture

Beanstalkd client

Beanstalkd Worker

Beanstalkd Goodness

• Named Tubes• Priority• Delayed job• Bury• Kick

How we use it

• SMS System• Image Re sizing• PDF Generation• Real Time Analytics• And a lot more

ONE PROCESS TO RULE THEM ALL

Supervisord

• Python Application• Manages foreground process• Easy to configure - .ini format• Robust and Reliable

The configuration

Load Balance your workers

• Measure average time taken to complete a job• Define threshold of average time• Change num_process in supervisor config• Reload to launch new workers

Practo is Hiring

•  Hacker• Love building stuff• Solve difficult problems• Change the world

abhinav@practo.com

Questions

@abhinavlalabhinavlal.inSildeshare.net/abhinavlal