Blocked and batch messages with RabbitMq

Question

I am trying to use RabbitMq in a more unconventional way (although at this point I can choose any other implementation of the message queue, if necessary). Instead of leaving Rabbit messages for my consumers, the consumer connects to the queue and receives a packet of N messages (during which he consumes some and possible deviations from some), after which he switches to another queue and so on. This is done for redundancy. If some consumers harm, all messages are guaranteed to be consumed by another consumer.

The problem is that I have several consumers, and I do not want them to compete for the same line. Is there a way to guarantee queue blocking? If not, can I at least make sure that if 2 consumers are connected to the same queue, they do not read the same message? Transactions can help me to some extent, but I heard that they will be deleted from RabbitMQ.

Other architectural suggestions are also welcome.

Thanks!

EDIT: As pointed out in the comment, there is a peculiarity in how I need to process the messages. They only make sense in groups, and there is a high probability that related messages are grouped together in a queue. If, for example, I print a package of 100 messages, there is a high probability that I can do something with messages 1-3, 4-5.6-10, etc. If I can’t find the group for some messages, I’ll re-send them to the queue. WorkQueue will not work, because it will distribute messages from one group to several workers who would not know what to do with them.

Answer 1

Have you looked at this free online book on Enterprise Integration Templates ?

It sounds like you really need a workflow in which you have a dispenser component before messages reach your employees. There are two ways to do this with RabbitMQ. Either use an exchange type (and message format) that can do doping for you, or have one queue, and a worker who sorts the batches and puts each batch in turn. The dispenser should probably also send a batch ready message to the management queue so that the worker can detect the existence of a new packet queue. After processing the batch, the worker can delete the batch queue.

If you have control over the message format, you can force RabbitMQ to perform batch processing implicitly in several ways. When exchanging topics, you can make sure that the routing key in each message is in the format work.batchid.something, and then an employee who finds out about the existence of the xxyzz package will use a binding key, such as # .xxyzz. # Only for consume these messages. No republishing is required.

Another way is to include the packet identifier in the header and use a new type of header exchange. Of course, you can also implement your own types of exchange if you want to write a small amount of Erlang code.

I really recommend checking out the book because it provides a better overview of the messaging architecture than the typical work queue concept with which most people start.

Answer 2

Ask your customers to pull out of the same queue. They will be guaranteed that they will not send messages (Rabbit will twist messages among users currently connected), and it is highly optimized for this exact usage pattern.

It is ready to use, out of the box. In RabbitMQ docs, it's called Work Queue . One turn, several consumers, none of which shares anything. This is similar to what you need.

Answer 3

You can set a channel / consumer prefetch counter to consume messages in batches. To resend messages, you must use the basic.reject AMQP method, and these messages can be selected in order to be sent or sent to the dead letter queue. Multiple consumers trying to pull messages from the same queue are not a problem because the AMQP basic.get method will be synchronized to handle concurrent consumers.

https://groups.google.com/forum/#!topic/rabbitmq-users/hJ8f5du-GCA