Resque: one worker in line

Question

I currently have a Rails 3.0 project with Ruby 1.9.2 and Resque.

My application has several working classes and several queues that are dynamically created (at runtime). In addition, there are several workers who can work in any queues, because there are no existing queues during startup, and they cannot be predicted:

$ COUNT=3 QUEUE=* rake resque:workers 

Queues created based on the project identifier:

 @queue = "project_#{project.id}".to_sym 

For this queue, their tasks must be processed in order and one at a time. My problem is that having several employees, several tasks are processed in parallel.

Is there a way to set the maximum number of workers in a queue (up to 1)? Is there a way to block the queue during job processing?

Thanks!

Answer 1

Finally, I came up with a fairly simple solution using resque-retry and the locks stored in redis (I do this for users, just do it for projects): https://stackoverflow.com/questions/168308/how-to-create-a-string-in-javascript/232632#232632

Answer 2

The first solution I was thinking about is to check if there is any worker working in a given queue when another worker polls the same queue. This can be done by overriding Resque::Job.reserve(queue) :

 module Resque class Job def self.reserve(queue) Resque::Worker.working.each do |worker| # if already working in the same queue if worker.job['queue'] == queue return end end return unless payload = Resque.pop(queue) new(queue, payload) end end end 

A possible problem may be the state of the race. Thoughts?

Answer 3

Resque-pool can help you specify the number of workers in the queue.

https://github.com/nevans/resque-pool