Is it possible to use celery with an existing MySQL table instead of a broker?

Question

The task at hand:

There is a MySQL table in which user activity is exposed as a row. This action needs to be processed later.

id | activity_type | activity_data | creation_time | status 23 EMAIL {....... } 2013-02-01 UNPROCESSED 

Processing without celery:

• Use a script that collects N records for processing, processes them, and then updates the PROCESSED status for the processed records.

How can I achieve the same result using Celery, I would like to use the same functionality to mark tasks instead of using a broker? those. the add → task accepts the 10 oldest rows added to the mysql table with UNPROCESSED status, which also has data associated with the task.

at the end of the task → mark the lines as PROCESSED.

How can celery say that instead of pushing a task to the broker, he should retrieve the tasks from the MySQL table?

I start at Celery, so I don’t realize all its functionality. Using MySQL as a broker is not recommended, but I would like to know about the possibility.

Answer 1

I think you misunderstand how celery works. You cannot just replace your broker with a MySQL table of your own design - well, without making significant changes to the source code.

The broker is an internal part of celery, which it uses to track its tasks using its own internal format, so there is no advantage to using the MySQL broker solely because the way you currently store the necessary information to carry out your tasks.

You can still use celery if you want, but you will need to write the code necessary to translate your user action table into celery tasks.

However, I would recommend experimenting with celery first ...

• Get a basic understanding of how to use it.
• determine if this is a good decision for what you are trying to achieve.

Start with a textbook on celery and see how it goes from there.