Behavior of RabbitMQ Clusters and Mirror Queues Backstage

Question

Can someone explain what happens behind the scenes in a RabbitMQ cluster with multiple nodes and queues in mirror mode when published to a sub node?

From what I read, it seems that all actions other than those published are sent only to the owner, and the master then transfers the effect of the actions to subordinates (this is from the documentation). Formulate my understanding, this means that the consumer will always consume the message from the main queue. In addition, if I send a request to a slave to consume a message, this slave will perform an additional jump, going to the wizard to receive this message.

But what happens when I publish a sub node? Will this node do the same thing that sends a message to the host first?

It seems that there are a lot of extra jumps when working with subordinates, so it seems you could improve performance if you only know the wizards. But how do you deal with a master failure? Then one of the slaves will be elected as a master, so you need to know what to connect to?

Ask about it because we are using the RabbitMQ cluster with HAProxy in front, so we can separate the cluster structure from our applications. Thus, whenever a node is executed, HAProxy is redirected to live nodes. But we have problems when we kill one of the nodes of the rabbit. The connection with the rabbit is permanent, so if this fails, you must recreate it. In addition, you must resend the messages in this case, otherwise you will lose them.

Even so, all messages can be lost, because they can be in the way when I kill the node (in some buffers, somewhere on the network, etc.). Therefore, you need to use transactions or publisher confirmations that guarantee delivery after all the mirrors have been filled with the message. But here is another problem. Perhaps you have duplicate messages because the broker could send a confirmation that never reached the manufacturer (due to network outages, etc.). Therefore, consumer applications will need to deduplicate or process incoming messages using an idempotent method.

Is there any way to avoid this? Or should I decide if I can lose a couple of messages against duplicating some messages?

Answer 1

Can someone explain what happens behind the scenes in a RabbitMQ cluster with multiple nodes and queues in mirror mode when published to a sub node?

This blog describes what is happening.

But what happens when I publish a sub node? Will this node do the same thing that sends a message to the host first?

The message will be redirected to the main queue - that is, the node on which the queue was created.

But how do you deal with a master failure? Then one of the slaves will be elected as a master, so you need to know what to connect to?

Again, described here. Essentially, you need a separate service that polls RabbitMQ and determines if the nodes are waiting or not. For this, RabbitMQ provides a management API . Your publications and consuming applications must access this service either directly or through a reciprocal data store to determine which node is the correct one to publish or use.

The connection with the rabbit is permanent, so if it fails, you must recreate it. In addition, you must resend the messages in this case, otherwise you will lose them.

You need to subscribe to disconnected events in order to respond to disconnected connections. You will need to create some level of redundancy on the client to ensure that messages are not lost. I propose, as above, to introduce a service specifically designed for the RabbitMQ survey. You can try to post a message with the last active connection known, and if that fails, the client can request a monitor service to update the list of the RabbitMQ cluster. Assuming that there is at least one active node, the client can establish a connection with it and publish the message successfully.

Even so, all messages can be lost, because they can be in the way when I kill node

There are certain edge cases that you cannot cover with redundant capabilities, and none of them can use RabbitMQ. For example, when a message is queued and the HA policy invokes a background process to copy the message to a node backup. During this process, it is likely that the message will be lost before it is saved to the node backup. If the active node terminates immediately, the message will be lost forever. There is nothing that could be done about this. Unfortunately, when we get to the level of actual bytes moving along the wire, there is a limit on the number of guarantees that we can build.

therefore, consumer applications will need to deduplicate or process incoming messages using an idempotent method.

You can handle this in several ways. For example, setting message-ttl to a relatively low value ensures that duplicate messages will not remain in the queue for extended periods of time. You can also mark each message with a unique link and check this link at the consumer level. Of course, for this you need to store a cache of processed messages to compare incoming messages; the idea is that if a previously processed message arrives, its tag will be cached by the consumer, and the message can be ignored.

One thing I would like to emphasize in AMQP and Queue in general is that your infrastructure provides tools, but not an entire solution. You must address these gaps depending on the needs of your business. Often the best solution is through trial and error. Hope my suggestions are helpful. Here I will talk about several RabbitMQ design solutions, including the problems you mentioned, here if you are interested.