I run celeryd as a demon, but sometimes it’s hard for me to stop its grace. When I send a TERM signal and there are elements in the queue (in this case, service celeryd stop ), celeryd will stop accepting new jobs and turn off all work processes. However, the parent process will not be closed.
I just came across a script when I had celeryd running on two separate working machines: A and B. With about 1000 messages on the RabbitMQ server, I shut down A and experienced the situation described above. B continued to work, but then stopped with 40 messages left on the server. However, I was able to stop B correctly.
I restarted B to check if 40 would be taken from the queue, but that is not the case. Then I severely killed A, after which B grabbed and completed tasks.
My conclusions are that the parent process has reserved 40 elements of our RabbitMQ server for its children. It will work correctly with children, but will not return the elements back to RabbitMQ, unless I manually drive it.
Has anyone experienced something similar?
I am using Celery 2.2.2
django celery django-celery celeryd
Zach
source share