JavaScript memory leak error (Node.js / Restify / MongoDB)

Question

Update 4 : by creating an instance of the restify client (see the /messages.js controllers) outside the function and calling global.gc () after each request, it seems that the memory growth rate was reduced by a lot (~ 500KB in 10 seconds). However, memory usage is still growing.

Update3 : came out through this post: https://journal.paul.querna.org/articles/2011/04/05/openssl-memory-use/

It may be worth noting that I am using HTTPS with Restify.

Update 2 . Updated code below current state. I tried reinstalling Restify with Express. Unfortunately, this did not matter. It seems that calling api at the end of the chain (restify -> mongodb -> external api) forces everything to be stored in memory.

Update 1 . I replaced Mongoose with the standard MongoDB driver. Memory usage seems to be growing faster, but the leak persists.

---

I have been working on this leak for a couple of days.

I use the APIs using Restify and Mongoose , and for every API call I do at least one MongoDB lookup. I have about 1-2 thousand users who get to the API several times a day.

What i tried

• I highlighted my code simply using Restify and used ApacheBench to run a huge number of requests (100k +). During the test, the memory usage is about 60 MB.
• I highlighted my code simply using Restify and Mongoose and tested it in the same way as described above. Memory usage is around 80 MB.
• I tested the full production code locally using ApacheBench. Memory usage is around 80 MB.
• I automatically dumped the heap at intervals. The biggest heap heap I had was 400 MB. All that I see is that there are many lines and arrays, but I cannot clearly see the pattern in it.

So what could be wrong?

I performed the above tests using only one API user. This means that Mongoose only captures the same document again and again. The difference with production is that many different users get into the API, which means that the mongoose receives a lot of different documents.

When I start the nodejs server, the memory grows rapidly to 100 MB-200 MB. It eventually stabilizes around 500 MB. Could this mean that it has a memory leak for each user? As soon as each user visits the API, does it stabilize?

I have included my code below, which describes the general structure of my API. I would really like to know if there is a critical error in my code or any other approach to figuring out what causes high memory.

the code

app.js

var restify = require('restify'); var MongoClient = require('mongodb').MongoClient; // ... setup restify server and mongodb require('./api/message')(server, db); 

api / message.js

 module.exports = function(server, db) { // Controllers used for retrieving accounts via MongoDB and communication with an external api var accountController = require('../controllers/accounts')(db); var messageController = require('../controllers/messages')(); // Restify bind to put server.put('/api/message', function(req, res, next) { // Token from body var token = req.body.token; // Get account by token accountController.getAccount(token, function(error, account) { // Send a message using external API messageController.sendMessage(token, account.email, function() { res.send(201, {}); return next(); }); }); }); }; 

Controllers / accounts.js

 module.exports = function(db) { // Gets account by a token function getAccount(token, callback) { var ObjectID = require('mongodb').ObjectID; var collection = db.collection('accounts'); collection.findOne({ token: token }, function(error, account) { if (error) { return callback(error); } if (account) { return callback('', account); } return callback('Account not found'); }); } }; 

Controllers / messages.js

 module.exports = function() { function sendMessage(token, email, callback) { // Get a token used for external API getAccessToken(function() {} // ... Setup client // Do POST client.post('/external_api', values, function(err, req, res, obj) { return callback(); }); }); } return { sendMessage: sendMessage }; }; 

Snapshot heap of alleged leak

Answer 1

There may be an error in getters, I got it when using virtual or getters for the mongoose scheme https://github.com/LearnBoost/mongoose/issues/1565

Answer 2

In fact, it’s normal to see only strings and arrays, since most programs are mainly based on them. Therefore, the profiler, which allows you to sort by the total number of objects, is not very useful, because they many times give the same results for many different programs.

The best way to use chrome memory profiling is to take one snapshot, for example, after one user calls the API, and then a second heap snapshot after the second user, called the API.

The profiler gives you the opportunity to compare two snapshots and see what the difference is between them (see this tutorial ), this will help to understand why the memory grew unexpectedly.

Objects are stored in memory because there is still a reference to them that prevents garbage collection.

So, another way to try to use the profiler to search for memory leaks is to search for an object that, in your opinion, should not be there and see what it means to save the path and see if there are any unexpected paths.

Answer 3

Not sure if this helps, but is it possible to try to delete unnecessary data?

api / message.js

  // Send a message using external API messageController.sendMessage(token, account.email, function() { res.send(201, {}); next(); // remove 'return' }); 

Controllers / accounts.js

 module.exports = function(db) { // Gets account by a token function getAccount(token, callback) { var ObjectID = require('mongodb').ObjectID; var collection = db.collection('accounts'); collection.findOne({ token: token }, function(error, account) { if (error) { callback(error); // remove 'return' } else if (account) { callback('', account); // remove 'return' } else { callback('Account not found'); // remove 'return' } }); } return { // I guess you missed to copy this onto the question. getAccount: getAccount }; }; 

Controllers / messages.js

  // Do POST client.post('/external_api', values, function(err, req, res, obj) { callback(); // remove 'return' }); 

Answer 4

Your problem is getAccount mixed with how the GC works.

When you call many functions, GC only clears one at a time, and the older one in memory, the less likely it is to collect it, so on your account you need at least that I can consider 6 calls global. gc () or auto is executed before it can be built by this time, GC accepts what it is probably not going to, so it does not check it anyway.

 collection{ findOne{ function(error, account){ callback('', account) sendMessage(...) getAccessToken(){ Post } } } } } } 

as suggested by Gene, remove this chain.

PS: This is just an idea of ​​how the GC works and is implementation dependent, but you get the point.