How to implement a long-term, event-driven python program?

Question

I have a number of maintenance tasks for a WSGI python application that are too complicated for crontab (work tasks must be performed at frequencies derived from the size of the job queue, manage the connection pool to the EC2 instance group, etc.).

How do I implement a long-running, event-driven python program? I have never needed this feature before, so I'm not even sure what to do with Google.

Answer 1

Most major modern python sites use Celery for this type of work. This is a distributed task queue that also supports task scheduling.

Although it may be a bit heavy for a small site, it will grow with you. I am going to implement it myself (without Rabbit) in the near future.

I recently found another choice for django users, django-tasks , which focuses on smaller, longer, and batch jobs. There is also django-ztask using zeromq.

Application: just came across a gearman that has python bindings .