Docker / Celery: Cannot Get Celery to Work

Question

I can't get my celery workers to constantly listen to the default queue. Celery is constantly coming out.

$: docker-compose up Starting tasker_rabbitmq_1 Starting tasker_celery_1 Attaching to tasker_rabbitmq_1, tasker_celery_1 tasker_celery_1 exited with code 1 rabbitmq_1 | rabbitmq_1 | RabbitMQ 3.6.1. Copyright (C) 2007-2016 Pivotal Software, Inc. rabbitmq_1 | ## ## Licensed under the MPL. See http://www.rabbitmq.com/ rabbitmq_1 | ## ## rabbitmq_1 | ########## Logs: /var/log/rabbitmq/ [e-mail] rabbitmq_1 | ###### ## /var/log/rabbitmq/ [e-mail] rabbitmq_1 | ########## rabbitmq_1 | Starting broker... completed with 6 plugins. 

I am trying to create an application with a separate task layer as a container of separately defined tasks. So, architecture:

• Web / App Layer ( Django) on EBS
• Employment Level: Celery + RabbitMQ as a Docker Container

This is what I have:

Folder structure:

 -tasker -tasker -tasks.py -celeryconfig.py - __init__.py -Dockerfile -docker-compose.yml -requirements.txt 

tasks.py:

 from celery import Celery from celery import task celery = Celery('tasks', broker='amqp:// guest@localhost //') import os @celery.task def add(x, y): return x + y 

Dockerfile:

 FROM python:3.4 ENV PYTHONBUFFERED 1 WORKDIR /tasker ADD requirements.txt /tasker/ RUN pip install -r requirements.txt ADD . /tasker/ 

docker-compose.yml:

 rabbitmq: image: tutum/rabbitmq environment: - RABBITMQ_PASS=mypass ports: - "5672:5672" - "15672:15672" celery: build: . command: celery worker --app=tasker.tasks volumes: - .:/tasker links: - rabbitmq:rabbit 

Is there something I am missing? Why is celery output with code 1?

Answer 1

1) install celery on a Pockon Docker image or use https://hub.docker.com/r/library/celery/

2) add the full address for rabbitmq with the password in the line:

celery = Celery('tasks', broker='amqp://guest: mypass@rabbit //')

Answer 2

The RabbitMQ container is not available when you try to open the container for celery.

A quick and dirty way to handle this is to manually enable rabbitmq in the background using the -d flag:

 docker-compose up -d rabbitmq 

Wait a few seconds and:

 docker-compose up celery 

This describes the correct way to control the startup order -> https://docs.docker.com/compose/startup-order/

---

EDIT

I am expanding my answer a bit on how to do this with 'wait it it

 celery: build: . command: ./wait-for-it.sh rabbitmq:5672 --timeout=2 --strict -- celery worker -A tasker.tasks volumes: - .:/tasker links: - rabbitmq 

---

EDIT 2

I also noticed that in your build file, the container name is incorrect according to the link directive. It should just be rabbitmq . It also affects the order in which your containers are included.

EDIT 3

You probably need to add the hostname to the rabbitmq container:

 rabbitmq: image: tutum/rabbitmq hostname: rabbitmq