Geek Answers Handbook

NodeJS: how to free buffers that are allocated outside of the V8 memory heap

I have an application where I sequentially download mp3 files from the server, temporarily storing them on my server and then transferring them directly to clients, for example:

function downloadNextTrack(){ var request = http.get('http://mp3server.com', function(response){ response.on('data', function(data) { fs.appendFile('sometrack.mp3', data, function (err) {}); }); response.on('end', function(){ streamTrack('sometrack.mp3'); } }); }; var clients = []; // client response objects are pushed to this array when they request the stream through a route like /stream.mp3 var stream; function streamTrack(track){ stream = fs.createReadStream(track); stream.on('data', function(data){ clients.forEach(function(client) { client.write(data); }); }); stream.on('end', function(){ downloadNextTrack(); // redoes the same thing with another track } };

Apparently, this code creates a lot of buffers that are not freed up by the OS when I run the “free -M” command, this is what I get (about 4 hours after the application starts):

  total used free shared buffers cached Mem: 750 675 75 0 12 180 -/+ buffers/cache: 481 269 Swap: 255 112 143

The number under the “buffers” is constantly growing (as well as cached memory), and the OS doesn’t seem to return this 180mb back until my application runs out of memory and crashes when I try to create a small process to check the bitrate of the track , sample rate, id3 information, etc.

I have been diagnosed with many different tools (for example, memwatch and nodetime) to find out if it was an internal memory leak, but it isn’t, the heap of V8 memory and also Node RSS changes +/- 10 Mb, but remain constant in most cases , while the free memory of the OS is getting lower and lower (when I start the Node process, I have about 350 MB of free memory).

I read somewhere that the Buffer instances allocated by Node have direct access to raw memory, and so the V8 has no power over them (which checks that I am not getting a memory leak from the V8 heap) The thing is, I need a way get rid of these old buffers. Is it possible? Or will I have to restart the application every 5 hours or so (or, even worse, buy more RAM!)?

PS. I am running Node v0.8.16 on Ubuntu 10.04.

+6
source share
2 answers

I agree with Thiago, I think this is due to the recursive nature of your code. I don’t think that threads are what swallows your heap because, as you said, the stream variable is reassigned to a new ReadStream with each iteration. However, the http.get request and response (and any buffers they use) on line 2 are never issued before invoking the next iteration; they are covered by the downloadNextTrack function. You get a recursive stack trace that has a set of request and response objects (and some basic buffers) for each file.

In general, if this code needs to run many, many times, why not give up recursion and do it all iteratively? endless recursion will always gobble up more and more memory until the program crashes, even if there are no memory leaks in your memory.

+2
source

Read this: http://www.linuxatemyram.com

The buffer cache is the cache for inodes and dentries (file system structure). This memory is still available for processes. You do not have to worry about that.

0
source

More articles:


All Articles