Can celery celery use a database glider without Django?

Question

I have a small infrastructure plan that doesn't include Django. But, due to my experience with Django, I really like Celery. All I really need is Redis + Celery to do my project. Instead of using the local file system, I would like to save everything to Redis. My current architecture uses Redis for everything until it is ready to dump results on AWS S3. Admittedly, I have no great reason to use Redis instead of the file system. I just invested so much in architecture with Docker and scalability in mind, it feels wrong.

Answer 1

I searched too long for a Django database planner, but it seems like nothing more. So I accepted the Django scheduler code and modified it to use SQLAlchemy. It should be even easier to get him to use Redis.

Answer 2

Turns out you can!

First I created this little project from a tutorial on celeryproject.org .

This is great, so I built a Dockerized demo as a proof of concept.

What I learned from this project

• Docker

  • using --link to create network connections between containers
  • executable commands inside containers

• Dockerfile

  • using FROM to create images iteratively
  • using official images
  • using CMD for images that "just work"

• Celery

  • using celery without Django
  • Using Celerybeat without Django
  • using Redis as a queue broker
  • project layout
  • task naming requirements

• Python

  • correct project layout for setuptools / setup.py
  • project installation via pip
  • using entry_points to access console scripts
  • using setuid and setgid to de-escalate privileges for a celery taster