Makes several attempts to play the Azure role role in the same queue as Dead Lock or Poison message

Question

Scenario:

if I disconnect multiple worker roles or the ONE Worker role with multiple threads that poll new messages in Azure Queue.

Can someone please confirm if this approach is correct? The reason I want to have many working roles is to speed up the PROCESSJOB process. Our application should be practically in real time, i.e. As soon as the messages we need to receive appear, apply complex business rules and commit DB AZURE. We expect 11,000 messages in 3 minutes.

Thanks.

Answer 1

You can have as many lineup readers as you want. This is very common for scaling worker role instances, as they can all read from the same queue, giving you much greater throughput.

When you read a queue message, it is marked as “invisible” for a certain period of time so that others do not read or do the same job. The owner of the message must delete it before the time period expires, otherwise the message will become visible again, and an exception will be thrown when the original reader tries to delete it. This means that your operations must be idempotent.

There is no direct processing of poisonous messages, but it is easy to implement, since each message has a detection account. Just check it and delete poisonous messages after reading 3-4 times. You can also dynamically adjust the timeout period based on the amount of decompression, as processing may have failed due to a too short time window.

Here's the MSDN documentation for DequeueCount .

EDIT: to process 11,000 messages in 3 minutes: the scaling goal for the queues is 500,2000 TPS or up to 360,000 transactions in 3 minutes (much more than you need 11,000 messages), you can speed up the process by combining the messages into one message about the queue, as well as reading several messages at the same time, which will also reduce the number of transactions. You can also look at the ApproximateMessageCount property of the queue to check if your queue is reserved (and then the queue elements consume additional items help help consume).