Celery randomly losing tasks

Question

I see that tasks seemingly “disappear” in the celery, work with two nodes. It seems to be happening by chance. The task is created as follows:

task = perform_advance.apply_async(...) logger.info('Task created, id: {}'.format(task.task_id)) 

When this works, I will see something like:

[2016-04-21 01:13:02,470: INFO/Worker-8] foo.tasks.some_task[e52615da-de7a-49de-88d6-b3ca43a3383f]: Task created, id: eaaeb427-a167-4a78-ba39-4803e20cc753

[2016-04-29 21:18:40,667: DEBUG/MainProcess] Task accepted: foo.tasks.some_task[eaaeb427-a167-4a78-ba39-4803e20cc753] pid:1104

But when he fails, I never see the task accepted, only it was created. There are no errors in the logs.

celery version: 3.1.23

rabbitmq version: 3.3.3

Answer 1

Worked on this.

I thought I would be so kind to share a solution to this problem here.

It turned out to be the Amabon ELB internal load balancer for RabbitMQ, which confused us. Connecting to RabbitMQ directly, not ELB, solved this.