How should the node.js stack be installed for a high demand application?

Question

I am currently working on a Node.js application used by more than 25,000 people, we primarily use the Sails.js environment, and we got MongoDB. The application runs on an EC2 instance with 30 GB of RAM, the database runs on a Mongolab AWS cluster in that same zone as EC2. We even have an instance of Elastic Cache Redis with a memory capacity of 1.5 GB.

Thus, the main and enormous problem we are facing is LATENCY . When we reach the peak of concurrent users requesting the application, we get several applications with timeouts and sails reaching more than 7.5 GB of RAM, HTTP requests to the API take more than 15 seconds (which is unacceptable), and when they even receive 502 and 504 replies sent by Nginx.

I can notice that Mongo operations are recorded as the main delay problem, however even GET requests take a long time when there is a peak of demand. I can’t access the working servers, I only have the pm2 key metrics monitoring tool (which is really great) and the New Relic alerts.

So, I would like to know a roadmap to deal with these problems, maybe you will be offered more detailed information, so far I can say that the application seems stable when few users are present.

What are the main factors and settings to consider?

I still know what I should do, but I'm not sure about the details or how.

IMHO:

• Cache as much as possible.
• Delay write operations MongoDB.
• Separate Mongo databases with higher write requirements.
• virtualization?
• Configure node settings.

While optimizing the code, I posted another question using stackoverflow with one example code template that I am following .

What are your recommendations and opinions for production applications?

Answer 1

Basically, most of the highlights are already present in the answers. I just let them down.

There are several basic things you can do to optimize your application.

• Try moving the node.js form to io.js , it is still slightly improved and the latest updated version is updated. (But read carefully about the experimental features). Or at least from node.js v10 to v12 . There have been many performance optimizations.

• Avoid using synchronous functions that use I / O or work with large amounts of data.

• Switch from one node process to clustering .

• Check application for memory leak. I use memwatch-next for node.js v12 and memwatch for node.js v10

• Trying to avoid saving data in global variables

• Use caching. For data that should be available all over the world, you can use Redis or Memcached also a large store.

• Avoid using async with Promises . Both libraries do the same. Therefore, there is no need to use both of them. (I saw this in your code example).

• Combine async.waterfall with async.waterfall methods where you can do this. For example, if you need to get some data from mongo that is only related to the user, you can select the user, and then select all the other data that you need in parallel.

• If you are using sails.js , make sure it is in production mode. (Suppose you have already done this)

• Disable all interceptors that you do not need. In most cases, the grunt hook is useless. And if you do not need Socket.io in your application, disable it using the .sailsrc file. Something like:

  {"generators": {"modules": {}}, "hooks": {"grunt": false, "sockets": false}}

Other hooks that can be disabled are as follows: i18n , csrf , cors . BUT only if you do not use them in your system.

11. Disable useless globalization. In config/globals.js . I assume that _ , async , services may be disabled by default. Just because sails.js uses the old version of the lodash and async libraries, and newer versions have much better performance.

12. Manually install lodash and async in the sails.js project and use the new versions. (see paragraph 11)

13. Some write to mongo operations may be performed after the result is returned to the user. For example: you can call the res.view() method, which will send a response to the user before Model.save() , but the code will continue to work with all variables, so you can save the data in mongo DB. Thus, the user does not see a delay during recording.

14. You can use queues such as RabbitMQ to perform operations that require a lot of resources. For example: if you need to store a large collection of data, you can send it to RabbitMQ and return a response to the user. Then process this message in the data warehouse of the process "background". It will also help you with the scaling of your application.

Answer 2

First, make sure you are not using synchronous I / O. If you can run io.js , there is a flag --trace-sync-io ( iojs --trace-sync-io server.js ) that will warn you if you use synchronous code with the following console warning: WARNING: Detected use of sync API .

Secondly, find out why the use of your RAM is so great. If this is due to the large amount of data loaded into memory (XML parsing, large amount of data returned from MongoDB, etc.), you should consider using streams . Garbage collection V8 (the Google JavaScript virtual machine used in Node.js / io.js ) can slow down if your memory usage is very high. Learn more here: Node.js Performance Tip of the Week: Managing Garbage Collection and Node.js Performance Tip of the Week: Heap Profiling

Third, experiment with Node.js clustering and MongoDB sharding .

Lastly, check if you are using or can switch to MongoDB 3.x. We only see significant performance gains by upgrading from 2.x to 3.x.

Answer 3

For Mongodb, you can use mongtop to find out which databases are being challenged, 2.2+ uses for each database lock if large workloads are recorded in the database to be affected, since mongodb uses the writer greedy locks

And for node.js, you can check if there are any event loop delays that might explain API request delays

 (function getEventLoopDelay() { var startTime = Date.now(); setTimeout(function() { console.log(Math.max(Date.now() - startTime - 1000, 0)); getEventLoopDelay(); }, 100); })();