Geek Answers Handbook

Realization of “single” and “normal” priority queues for long-term tasks in celery

I am trying to implement the following scenario with Celery: two queues of (identical) long-running tasks, one for "normal" and one for "idle" priority.

I would force the workers to control both queues and first perform tasks from the "normal" priority queue, and if it is empty, it will perform tasks with an "unoccupied" priority.

My question is: is it possible to guarantee the order in which workers will check the queues of their tasks? Also, is this the right approach for implementing priorities?

Reference Information. In tasks, ffmpeg transcoding jobs are performed. The "usual" priorities will be new videos (which should be transcoded as ASAP), and the priority of "inaction" will be the tasks of transcoding the old (40.000+ videos) archives into updated formatting options. I do not have several servers available for dispatching multiserver tasks.

+8
django celery rabbitmq django-celery
source share
2 answers

In accordance with the theory that idle workers do not consume a lot of system resources, I implemented two priorities with a second exchange, a second set of queues and a second set of workers. I do not have such intensive workflows, so I do not have my second set of workers throttle themselves when the first set is active, but it is possible that the UNIX renice command will work for this.

Perhaps you can also do something with the control queue so that employees do not have to receive a work request, and also need to get a token from the management queue. Then you spit out a small number of tokens in the queue management queue, but a larger number of tokens are in the normal management queue.

+1
source share

Docs:

also:

  • Workaround for Celery Task Priority on RabbitMQ?
-one
source share

More articles:


All Articles