AMQP: confirmation and prefetch

Question

I am trying to understand some aspects of the AMQP protocol. I currently have a project with RabbitMQ and I am using the Python Pika library. So the question is about confirming and preloading messages.

1. Suppose we have a single consumer queue (this queue was probably declared exclusive). So, I understand correctly: does it matter if I consume ack with or without a flag? In any case, I should not be able to process several messages at the same time, and there are no other consumers who could receive some other messages in the queue. Even better, do not include confirmation, because it can reduce the load on the AMQP server.

2. The prefetch counter means nothing if there is no confirmation. Correct?

3. I'm not sure how preloading works. I have a callback for a new message, and in its final approval I confirm or reject the message. This is the only function, and no matter how large the number of prefetchings is, in any case, another message will not be processed until the current message is completed. So why should I change the value of prefetch_count?

thanks in advance.

Answer 1

If the autoack flag autoack disabled, if your application failed during message processing, all received messages will be lost. If this situation is quite rare, and message loss is a suitable option in your application (for example, without restrictions, processing logs), you can disable autorun.

And yes, if autoack unset requires simpler broker logic, so it uses less resources.

As for the prefetch count (and prefetch size ), these parameters tell the broker how big the payload can be sent to the client in advance for one message. It is usually used to save time on network operations in order to wait for new messages. When the prefetch size client will receive one or more messages that have a total size equal to or smaller in order to preset the size of the clean (and / or an amount that is smaller).

Both prefetch count and prefetch size rules apply. When one of them is set to zero (not set), it will not be applied.

Most importantly, prefetching defines the behavior for sending messages until more messages are received that the client is not packed .

Combined, these two settings produce something like this:

Prefetching with message limit and sending messages in advance:

Conditions:

• Queue: N posts x 1kb
• prefetch-size=5kb, prefetch-count=4 : prefetch-size=5kb, prefetch-count=4
• Autoack: off

The working process:

• The broker sends 4 messages to the client (limited by prefetch-count=4 ). 4 messages will be marked as inactive and taken out of the queue (therefore, they will not be delivered to other clients).
• Client ack 1 message.
• The broker has -1 unacked message (delete this message) and send another 1 message to the client (+1 un-aked, -1 from the queue, and the client already has 3 unanswered messages).
• The client receives the remaining 3 messages + new delivered.
• The broker has -4 unanswered messages and again sends 4 messages, +4 un-aked, -4 from the queue.
• The client receives message 1 and does not work.
• The broker will be inactive, and then move the remaining un-acked to the queue, so -3 queues without a queue and +3, so they can be delivered again to this or another client.

Prefetch with large messages:

Conditions:

• Queue: 1 message x 5Kb, N messages x 1kb
• prefetch-size=5kb, prefetch-count=2 : prefetch-size=5kb, prefetch-count=2
• Autoack: off

The working process:

• The broker sends 1 message to the client (limited prefetch-size=5kb ), and this message is marked as unaccumulated.
• Client ack 1 message.
• The broker has a -1 un-acked message, sends 2 messages again (limited by prefetch-count=2 , note that only the first message was 5kb, the remaining 1kb), and these messages are marked as un-acked.
• The client receives message 1 and does not work.
• The broker will move the selected message from the message queue, and the remaining inactive messages will be moved again to the queue to which they belong, so they can be delivered to this or another client again.

With auto

Conditions:

• Queue: N posts x 1kb
• prefetch-size=5kb, prefetch-count=10 : prefetch-size=5kb, prefetch-count=10
• Autoack: on

The working process:

• While both prefetch-size and prefetch-count ignored when no-ack set to true (such as the automatic access function called in RabbitMQ and in AMQP documents), messages will be sent to the client one by one and removed from the queue after successful shipment.

Note that AMQP has an asynchronous architecture, so under certain conditions two clients MAY receive one message at a time. Also, an inactive message MAY be delivered to the same client back (especially if you have one client).

Also look at the prefetch-size and prefetch-count official documentation and experiment a bit with these options.

PS: autoack basically no-ack AMQP flag set to true .