Handling lengthy processes in NodeJS?

Question

I saw some old posts touching on this topic, but I wanted to know what a modern, modern approach is.

Use case: (1) suppose you want to perform a long task in a video file, say 60 seconds , say jspm install , which can take up to 60 seconds. (2) you can NOT subdivide the task.

Other requirements include:

• need to know when the task ends
• nice to stop the task
• stability: if one task dies, it does not crash the server
• should be able to handle 100 simultaneous requests

I saw the solutions mentioned:

• nodejs child process
• webworkers
• - not used for tasks related to the CPU
• generators - not used for tasks related to the CPU
• https://adambom.imtqy.com/parallel.js/
• https://github.com/xk/node-threads-a-gogo
• any others?

What is a modern, standards-based approach? Also, if nodejs is not suitable for this type of task, then this is also a valid answer.

Answer 1

Short answer: Depends

If you mean the nodejs server then the answer is not suitable for this use case. A single-threaded Nodejs event cannot handle CPU-related tasks, so it makes sense to transfer work to another process or thread. However, for this use case, when the task associated with the processor has been running for a long time, it makes sense to find some way of queuing tasks ... i.e. It makes sense to use a work queue.

However, for this particular case of using JS code ( jspm API ) it makes sense to use a work queue that uses nodejs. Therefore, the solution is this: (1) use a nodejs server that does nothing but queue tasks in the work queue. (2) use the nodejs work queue (e.g. kue ) to do the actual work. Use cluster to spread work across different CPUs. The result is a simple, single server that can handle hundreds of requests (without strangulation). (Well, almost, see the note below ...)

Note:

• The solution above uses processes. I did not investigate streaming solutions because it seems that they were not pleasant for node.
• work queue + cluster gives you the equivalent of a thread pool.
• yes, in the worst case, the 100th parallel request will take 25 minutes to complete work on a 4-core computer. The solution is to deploy another worker queue server (if I'm not mistaken, with a db-enabled kue such as kue , this is trivial), just indicate that each point server points to the same db).

Answer 2

You mention a CPU-related task and a long-term one, which is definitely not a thing node.js. You also mention hundreds of simultaneous tasks.

You can take a look at something like the Gearman job server for such things - this is a special solution.

Alternatively, you can still control node.js requests, just don’t do the actual job.

If it is relatively acceptable to have lower than optimal performance, and you want to save your code in JavaScript, you can still do it, but you should have some kind of job queue - something like Redis or RabbitMQ comes to mind.

I think that the job queue will be a must for long, hundreds / sec tasks, regardless of execution time. Unless you can create this task on other servers / services / machines - then you do not care, your node.js API is just the front and the level of managing the cluster of tasks, then node.js works fine for work, and you need focus on the cluster of tasks, and then you could ask a more correct question.

Now node.js can be useful for you here, it can help manage and keep these hundreds of tasks, depending on where they come from (i.e. you can only allow requests to your task server for specific users, or limit functionality pause others, etc.