How do I know when the RabbitMQ task set is complete?

I use RabbitMQ so that workflows encode video files. I would like to know when all files are completed, that is, when all workflows are completed.

The only way I can do this is to use a database. When the video finishes coding:

UPDATE videos SET status = 'complete' WHERE filename = 'foo.wmv'
-- etc etc etc as each worker finishes --

And then, to check if all the videos have been encoded:

SELECT count(*) FROM videos WHERE status != 'complete'

But if I am going to do this, then I feel that I am losing the benefit of RabbitMQ as a mechanism for several distributed workflows, as I still have to manually maintain the database queue.

Is there a standard mechanism for RabbitMQ dependencies? That is, we can say: "Wait until these 5 tasks are over, and as soon as they are finished, then complete a new task?"

I don’t want the parent process to add these tasks to the queue and then “expect” each to return a “completed” status. Then I have to maintain a separate process for each group of videos, after which I lost the advantage of decoupled workflows compared to one ThreadPool concept.

Am I asking for something that is impossible? Or are there standard, widespread solutions for managing the overall status of tasks in a queue that I missed?

Edit: after searching, I found this similar question: Getting the result of a long-term task using RabbitMQ

Are there any special thoughts that people have about this?

+8
source share
3 answers

. - RabbitMQ, :

  • , .
  • ( )
  • , - ( - ),
  • , numSent == numResponded,

-, , - - , - ? , :

  • -.
  • -
  • - , .

Request Reply Pattern.

+10

, . , . ..

, ( -) , , , , .

+1

enter image description here

Based on Brendan's extremely helpful answer that needs to be accepted, I designed this quick chart that will be useful to some.

0
source

All Articles