How to automatically kill celery workers when the line they are listening is empty?

Question

Is there any “clean” way to kill workflows after the queue they listened to becomes empty?

The idea is that I do not need workers to continuously listen to the line if they have no work. I tried the autoscale option, but from my experience, even if I set the lower threshold to 0, the worker will not die.

In my case, I need to download 1000 files from somewhere using celery. So I put 1000 tasks in the queue, but that’s all I want this group of workers to do. It would be ideal if they could somehow die after everyone has finished downloading all the files, and not being idle there until I forget to kill the processes manually.

I could not find any suggestions on how to do this from inside the celery. Any ideas? Thank!

Answer 1

With a very nice Rq simple job queuing system, you started it in batch mode. The worker will start, start processing all the jobs in the queues and die. To handle more tasks, we run a worker every time in a minute using Cron.

For celery, we can make the equivalent using two separate Cron tasks:

* / 2 * * * * celery -A proj worker -l info

It starts celery every minute.

1-59 / 2 * * * * ps auxww | grep 'celery worker' | awk '{print $ 2}' | xargs kill

stop . : " shutdown, , ".

, 50% . 5% .

. , , . .