I have requests for different samples (s1, s2, ..) that need to be processed in a linear way (i.e. only one can say that the s1 request can be processed at the moment). I have an N-number of work services that can handle these requests. How can I implement the rpc-queue template so that the samples are consumed one at a time and still allow me to distribute the calculations between different samples?
I would like to implement this with rabbit-mq because of its simplicity, clustering capabilities, but I also want to consider other solutions.
Here is a picture illustrating the problem (with two workers)
worker 1
+-----------+
| |
input queue +---->| |-------+
+--------------+ | | | |
| | | +-----------+ |
| s1,s2,s1,s1 |------+ |
| | | worker 2 |
+--------------+ | +-----------+ |
| | | |
output queue +---->| |-------+
+--------------+ | | |
| | +-----------+ |
|(s1,s2,s1,s1) |<-+ |
| | +-----------------------------+
+--------------+