Delayed_jobs vs resque vs beanstalkd?

Question

Here are my needs:

• Enqueue_in (10.hours, ...) (DJ syntax is perfect.)
• Multiply workers at the same time. (Resque or beanstalkd are good for this, but not a DJ).
• Must handle push and pop from 100 jobs per second. (I will need to run a test to make sure, but I think the DJ cannot handle this many tasks)

Resque and beanstalkd do not enqueue_in.

There is a plugin (resque_scheduler) that does this, but I'm not sure how stable it is.

Our environment is on Amazon, and they rolled out beanstalkd for those who have Amazons for free, this is a plus for us, but I'm still not sure if this is the best option.

We run rails 2.3, but soon bring it to rails 3.0.3.

But what is my best choice? I missed another stone that does the job better?

I feel that my only option that actually works now is resque_scheduler.

Edit:

Sidekiq ( https://github.com/mperham/sidekiq ) is another option you should check.

Answer 1

For my projects, I will feel very comfortable using collectiveidea / delayed_job in rails2 and 3. I do not know beanstalkd, but I will try soon :-). I followed the suggestions in the resque docs. I will report this.

Resque vs DelayedJob 

How does Resque compare to DelayedJob, and why do you choose one of them?

• Resque supports multiple queues
• DelayedJob supports lower priorities
• Resque workers are memory leak / bloat resistant
• DelayedJob workers are extremely simple and easy to modify.
• Resque requires Redis
• DelayedJob requires ActiveRecord
• Resque can queue JSONable Ruby objects as arguments
• DelayedJob can put any Ruby object in turn as arguments
• Resque includes a Sinatra app to monitor what is happening.
• DelayedJob can be requested from your Rails application if you want to add an interface

If you are developing Rails, you already have a database and ActiveRecord. DelayedJob is very easy to set up and works great. GitHub has used it for many months to handle nearly 200 million jobs.

Choose Resque if:

• You need several queues
• You don't need / don't like numerical priorities
• You do not need to persist every Ruby object
• You have potentially huge lines
• You want to see what happens.
• You expect a lot of setbacks / chaos
• You can customize redis
• You do not have enough RAM.

Choose DelayedJob if:

• Do you like numerical priorities
• You do not complete a gigantic amount of tasks every day.
• Your turn remains small and agile.
• Not many glitches / chaos
• You want to easily throw everything in line
• You do not want to install Redis

Choose Beanstalkd if:

• Do you like numerical priorities
• You need a very fast line
• You do not want to waste RAM.
• You want to complete a large number of tasks
• Everything is fine with JSONable Ruby objects in the queue as arguments
• You need several queues

In no way is Resque "better" than DelayedJob, so make sure you choose the tool that works best for your application.

Good comparison of queue speed :

  enqueue work ------------------------------------------------- delayed job | 200 jobs/sec 120 jobs/sec resque | 3800 jobs/sec 300 jobs/sec rabbitmq | 2500 jobs/sec 1300 jobs/sec beanstalk | 9000 jobs/sec 5200 jobs/sec 

Have a nice day!

PS There is a RailsCast about resque , deferred assignment ( modified version ) and Beanstakld . Take a look!

PPS My favorite choice now is Sidekiq (very simple, fast and efficient for simple tasks), see this page for comparison.

Answer 2

Amazon Beanstalk is not Beanstalkd.

Beanstalkd - queue - has delayed tasks that will not be reserved from the queue until the specified number of seconds has passed. If this is what Enqueue_in(10.hours, ... ) means Enqueue_in(10.hours, ... ) , then it's just syntactic sugar to calculate the number of seconds and not do the work until then.

Answer 3

A quick note: delayed_job 3.0+ supports named queues

 object.delay(:queue => 'tracking').method Delayed::Job.enqueue job, :queue => 'tracking' handle_asynchronously :tweet_later, :queue => 'tweets'