Best practice and scale of creating azure bursts

Question

Im studying the architecture for a high-performance web performance application (theoretical at the moment) on Windows Azure and wanted to pick your brains about "Windows Azure Queues (not SB)" and how best to scale / create them.

I mainly look at the MVC interface (web role), Windows Azure queues (decoupling of asynchronous messages), the worker role, and the blackened SQL database.

I understand that we receive a message in the role of the Web, and then pass it to the queue, the worker role will poll the queues {do the work ... for example, SQL DB CRUD operation} and send a completion message.

What is the best way to handle the Windows Azure queue queue for scaling, as well as forwarding messages back and forth through the web and worker roles? It is best to have one queue for sending work, for example. orders, and then the next queue for notification, for example. status message

I saw that many posts say that you should create your queues outside of your application code, it makes sense, but how do you scale this with a queue restriction? "The scaling target for a single Windows Azure queue is" limited "to 500 transactions / sec. ??

MSDN has some useful resources for queuing scaling.

• Scaling role instances refers to adding and removing additional instances of web pages or work roles to handle just-in-time workloads. This often involves changing the number of instances in the service configuration. Increasing the number of instances will cause Windows Azure to start to run new instances, while decreasing the number of instances will in turn disable running instances.

• Scaling a process (thread) refers to maintaining sufficient capacity in terms of processing threads in a given instance of a role by adjusting the number of threads up and down depending on the current workload.

In short, I am looking for answers to the following questions:

• Best practice for creating a queue?
• How a web role (MVC application) tracks messages, i.e. Does she poll the notification queue after she sends the message, and what is the best way to handle message correlation in the Web role to send back the client (web consumer)?
• Are the scaling options above the best approach, or should I look at dynamically scaling queues on the fly (if so?), I think SB queues can be better in this case) creating new queues to bypass 500 transactions / sec limit?

As already noted, my questions are more theoretical at this point in time, I want the architect and the future to prove any solution in the future.

thanks

Answer 1

In the question of how you track responses, I usually do the following:

• The web role writes a message to the queue containing the job ID.
• The worker role does the work and writes the result to a table with the partition key.
• The web role polls this row of the table. Once it exists, the answer is returned.

Again, if this is an interactive interactive thing in which the user is really waiting for an open HTTP connection to get the result, you might be better off just using synchronous communication and skipping the queue.

If I were you, I would at least take a look at using something like 0MQ (ZeroMQ or its new fork Crossroads I / O), which gives you good abstractions on top of raw sockets that might be useful for this. Such things. For example, web servers send messages through PUSH sockets, worker roles receive through PULL sockets, worker roles publish responses through PUB sockets, and web roles receive responses through SUB sockets. PUSH / PULL performs load balancing, and PUB / SUB processes the message back to the pending web role. FYI is precisely the architecture and messaging technology used by Mongrel2.

In terms of getting over 500 operations per second, you can simply create multiple queues and spray messages randomly. (Each employee typically interviewed them all.) But keep in mind that one account has a limit of 5000 operations per second, so after you create 10 queues, you need to create a new vault account to get more guaranteed scalability.

Answer 2

I would recommend that you use this template -IF- you need a quick, synchronous (and scalable) solution.

The reason for using this template is that the queue allows you to independently scale web pages and work roles. This means quite a few queues, but it will mean that you could, for example, have 2 web worker roles and increase from 6 to 8 work roles if you need to.

I guess this could be faster than SQL querying, and it will be more robust than not using the queue.

More details here: https://msdn.microsoft.com/en-us/library/dn589781.aspx