Having problems streaming a response to a client using expressjs

Question

It is very difficult for me to wrap my head around how to transfer data back to my client when using Nodejs / Expressjs.

I grab a lot of data from my database, and I do it in chunks, I would like to pass this back to the client as I receive the data, so I don't need to store the entire data set in memory, like a json object before sending back.

I would like the data to be transferred back to the file, i.e. I want the browser to ask my users what to do with the file at startup. Previously, I created a file system write stream and transferred the contents of my data to the file system, and then, when this was done, I sent the file back to the client. I would like to exclude the average person (creating a tmp file in the file system) and just transfer the data to the client.

app.get( '/api/export', function(req, res, next) { var notDone = true; while (notDone) { var partialData = // grab partial data from database (maybe first 1000 records); // stream this partial data as a string to res??? if (checkIfDone) notDone = false; } } ); 

I can call res.write ("some string data") and then call res.end () when done. However, I am not 100% sure that this is actually a streaming response to the client when I write. It seems that expressjs stores all the data until I name the end and then send the response. It's true?

What is the correct way to stream data rows in response using expressjs?

Answer 1

You can do this by setting the appropriate headers and then just writing down the response object. Example:

 res.writeHead(200, { 'Content-Type': 'text/plain', 'Content-Disposition': contentDisposition('foo.data') }); var c = 0; var interval = setInterval(function() { res.write(JSON.stringify({ foo: Math.random() * 100, count: ++c }) + '\n'); if (c === 10) { clearInterval(interval); res.end(); } }, 1000); // extracted from Express, used by `res.download()` function contentDisposition(filename) { var ret = 'attachment'; if (filename) { filename = basename(filename); // if filename contains non-ascii characters, add a utf-8 version ala RFC 5987 ret = /[^\040-\176]/.test(filename) ? 'attachment; filename="' + encodeURI(filename) + '"; filename*=UTF-8\'\'' + encodeURI(filename) : 'attachment; filename="' + filename + '"'; } return ret; } 

In addition, Express / node does not write data written to the socket if the socket is not suspended (explicitly or implicitly due to backpressure). Data buffered with node, when in this paused state, may or may not be combined with other pieces of data that are already buffered. You can check the return value of res.write() to determine whether to continue writing to the socket. If it returns false, then listen for the leak event, and then continue recording again.

Answer 2

The response object is already a writable stream. Express processes automatically transmitted data, so you do not need to do anything except:

response.send(data)

You can also check the built-in pipe method, http://nodejs.org/api/stream.html#stream_event_pipe .