Using Celery as a Failover Planner

Question

I would like to use celery w / rabbitmq as a fault tolerant scheduler in a distributed environment. Due to tolerance, I mean that if a task is given to an employee, and this employee is following the path for some reason, the celery should be able to transfer it to another server. How can this be achieved in an environment where there are several work nodes?

Answer 1

You probably only need to set CELERY_ACKS_LATE

Late ack means that job messages will be acknowledged after the task is completed, and not just earlier, which is the default behavior. That is, if the working crash rabbit MQ still has a message.

More info here.

Retry Lost or Failed Tasks (Celery, Django and RabbitMQ)

Answer 2

Ask each of the workers to consume the same queue, and Rabbit will twist messages for workers (consumers). If any of them does not work when processing the task and before he can send his confirmation, the message will be automatically placed in the queue, and the next employee will pick it up. This is a sample of at least once delivery.

This link from the RabbitMQ website explains the template and includes sample Python code.