What are the pros and cons of fs.createReadStream vs fs.readFile in node.js?

Question

I trick node.js and found two ways to read the file and send it by cable as soon as I found that it exists and sent the correct MIME type using writeHead:

// read the entire file into memory and then spit it out fs.readFile(filename, function(err, data){ if (err) throw err; response.write(data, 'utf8'); response.end(); }); // read and pass the file as a stream of chunks fs.createReadStream(filename, { 'flags': 'r', 'encoding': 'binary', 'mode': 0666, 'bufferSize': 4 * 1024 }).addListener( "data", function(chunk) { response.write(chunk, 'binary'); }).addListener( "close",function() { response.end(); }); 

Am I right in thinking that fs.createReadStream can provide a better user interface if this file was something big, like a video? It looks like it might be less blocky; It's true? Are there any other pros, cons, reservations, or gotchas that I need to know?

Answer 1

A better approach is if you just connect the "data" to "write ()" and the "close" to "end ()":

 // 0.3.x style fs.createReadStream(filename, { 'bufferSize': 4 * 1024 }).pipe(response) // 0.2.x style sys.pump(fs.createReadStream(filename, { 'bufferSize': 4 * 1024 }), response) 

The read.pipe(write) or sys.pump(read, write) approach also has the advantage of flow control. That way, if the write stream cannot receive data as fast, it will say that the read stream will be turned off to minimize the amount of data that is buffered in memory.

flags:"r" and mode:0666 implied by the fact that this is a FileReadStream. The binary confirmation is outdated - if no encoding is specified, it will only work with raw data buffers.

In addition, you can add some other goodies that will make your file more complete:

• Sniff for req.headers.range and see if the line matches the line /bytes=([0-9]+)-([0-9]+)/ . If so, you just want to go from this beginning to the end. (Invalid number means 0 or "end".)
• The hash index and creation time from the stat () call to the ETag header. If you get a request header with "if-none-match" matching that header, send 304 Not Modified .
• Check the if-modified-since header for the mtime date of the mtime object. 304 if it has not been modified since the date of submission.

Also, in general, if you can, send a Content-Length header. (You put the file, so you should have this.)

Answer 2

fs.readFile load the entire file into memory, as you indicated, and as fs.createReadStream will read the file in pieces of the size you specified.

The client will also start receiving data faster using fs.createReadStream , since it is sent in chunks as it is read, and as fs.readFile will read the entire file and only then send it to the client. This may be minor, but it can make a difference if the file is very large and the drives are slow.

Think about it, although if you run these two functions in a 100 MB file, the former will use 100 MB of memory to download the file, while the latter will use no more than 4 KB.

Edit: I really see no reason why you would use fs.readFile , especially since you said that you would open large files.

Answer 3

If it is a large file, then "readFile" will be memory dependent, as it fills the entire contents of the file in memory and may hang your system. Although ReadStream is readable in chunks.

Run this code and note the memory usage on the performance tab of the task manager.

  var fs = require('fs'); const file = fs.createWriteStream('./big_file'); for(let i=0; i<= 1000000000; i++) { file.write('Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\n'); } file.end(); //.............. fs.readFile('./big_file', (err, data) => { if (err) throw err; console.log("done !!"); }); 

Infact, you will not see "done !!". message. "readFile" cannot read the contents of the file because the buffer is not large enough to hold the contents of the file.

Now instead of "readFile" use readStream and track memory usage.

Note: The code is taken from Samer-Bune Node Pluralsight course

Answer 4

Another, perhaps not so well-known fact, is that I believe that Node cleans up unused memory better after using fs.readFile compared to fs.createReadStream . You should check this to see what works best. In addition, I know that in each new version of Node, this has become better (i.e. the Garbage Collector has become more intelligent with these types of situations).