Running multiple concurrent HTTP requests using node.js

Question

I am tasked with dividing the process into several small processes running in parallel, distributed across several subordinate machines. The request arrives through HTTP, the server breaks it down into a series of child processes issued to the slave machines, waits until all the slave requests return responses, and then compares the results of the aggregate into one data object, the top-level request returned as a result. I thought node.js would be useful, but since it is single-threaded, I cannot say if this is possible, or if it will block waiting for each request to return before moving on to the next. Is this possible with node.js? If so, can someone point me in the right direction? that is, a node module to use or describe how this will be done?

Thanks for any help.

Answer 1

You can create child processes manually.

var util = require('util'), spawn = require('child_process').spawn, node = spawn('node', ['slave.js']); node.stdout.on('data', function (data) { console.log('stdout: ' + data); }); node.stderr.on('data', function (data) { console.log('stderr: ' + data); }); node.on('exit', function (code) { console.log('child process exited with code ' + code); }); node.stdin.write("do my work!"); 

We expect the Web Workers API to be implemented to handle child processes with higher abstraction.

Answer 2

You want to read about WebWorkers. In particular, you should read http://developer.yahoo.com/blogs/ydn/posts/2010/07/multicore_http_server_with_nodejs/

Answer 3

Try looking at the node.js cluster module, which is a multi-core server manager.

Answer 4

if even one of the smaller processes had a synchronous interface, which is single-threaded, would be a problem.

Fortunately, node is designed around asynchronous interfaces.

e.g. child process http://nodejs.org/docs/v0.4.6/api/child_processes.html#child_process.exec

or http requests http://nodejs.org/docs/v0.4.6/api/http.html#http.request

Answer 5

I think this blog post may seem interesting in terms of how async helps with parallel execution and how to execute parallel execution with Node.

http://blog.mixu.net/2011/02/01/understanding-the-node-js-event-loop/