I use RabbitMQ so that workflows encode video files. I would like to know when all files are completed, that is, when all workflows are completed.
The only way I can do this is to use a database. When the video finishes coding:
UPDATE videos SET status = 'complete' WHERE filename = 'foo.wmv'
And then, to check if all the videos have been encoded:
SELECT count(*) FROM videos WHERE status != 'complete'
But if I am going to do this, then I feel that I am losing the benefit of RabbitMQ as a mechanism for several distributed workflows, as I still have to manually maintain the database queue.
Is there a standard mechanism for RabbitMQ dependencies? That is, we can say: "Wait until these 5 tasks are over, and as soon as they are finished, then complete a new task?"
I don’t want the parent process to add these tasks to the queue and then “expect” each to return a “completed” status. Then I have to maintain a separate process for each group of videos, after which I lost the advantage of decoupled workflows compared to one ThreadPool concept.
Am I asking for something that is impossible? Or are there standard, widespread solutions for managing the overall status of tasks in a queue that I missed?
Edit: after searching, I found this similar question: Getting the result of a long-term task using RabbitMQ
Are there any special thoughts that people have about this?
source
share