Connecting an HTTP request / response model with an asynchronous queue

Question

What is a good way to connect a synchronous http request / response model with a queue based asynchronous model?

When a custom HTTP request arrives, it generates a job request that goes into the queue ( beanstalkd in this case). One of the workers picks up the request, does the work and prepares the answer.

The queue model is not a request / response — there are only requests, not answers. So the question is, what is the best way to get the answer back to the HTTP world and back to the user?

Ideas:

• Beanstalkd supports light threads or queues (they call them pipes). We could create a handset for each request, so that the worker creates a message on that handset, and the http process sits and waits for a response to the handset. I don't particularly like this one, as it has apache processes sitting around memory.

• Survey the http client for a response. A custom HTTP request starts the job in the queue and returns immediately. The client (user’s browser) periodically checks the response. On the backend, the worker places his answer in memcached, and we connect nginx to memcached so that the survey is easy.

• Use Comet . Like the second option, but with a more attractive http message to avoid polling.

I am inclined to 2 because it is easily and well known (I have not used a comet yet). I guess there is probably also a much more obvious model that I did not think about. What do you think?

Answer 1

Here's how to efficiently implement a JMS request-response , which can be useful (although Java / JMS-oriented). The general idea is to create a temporary queue for each client / thread, and then use correIDs to correlate response requests, etc.

Answer 2

Survey is a simple solution; the comet is a more effective solution. Do you have a nail :)

I personally love the comet (although I am biased since I helped write WebSync ), it beautifully allows your customers to subscribe to the channel and receive a message when your server process is ready. Works like a champion.

Answer 3

I am looking to implement a Beanstalkd and memcached system to start several processes following a request - in this case, searching for information when a user logs in (the number of messages waiting for the user, for example). Information is stored in Memcached and then read on the next page load.

Without knowing a little more about what tasks you are doing, it is not so easy to say what needs to be done, or how. Option number 2, however, is the easiest, and that may be all you need - depending on what you push back into the workers.