Low priority job in node.js

Question

Based on the fact that node uses one thread to control everything, I am curious how I should handle a low priority job

This job is constantly running and analyzing data (let's say it's setTimeout ), but it takes a lot of time, and I want it to have a very low priority in the scheduler.

I don’t think I can run it in a separate process, because I need to change its working instructions very often (it works using the variables of my main process, so it should be able to access them constantly).

You can imagine this as a large stack of tasks, which it will constantly work, but the working set for these tasks is controlled by the main process.

• A separate process will need constant requests between them to exchange data, so this seems like a bad idea.
• A single thread, like a web worker, could be faster? I don't know if node supports, but
• Best of all, it would be a simple event function running on a timer that respects the principles of node, but I need to somehow tell the scheduler to spend less time on this function than on the rest of the program. Is there any way to do this?

Answer 1

First of all, let me say:

This is a big problem.

In general, running background tasks in a single streaming environment is problematic for obvious reasons.

Let me start from the end, your third approach.

Using the function with a timer:

Running the second function with a timer sounds good, but how do you know if the time is up?

You can do something like:

 function myFunc(){ var date = Date.now(); while(date - Date.now() < 5 || nothingElseToDo()) { // 5 miliseconds // do something } setImmediate(myFunc); // continue processing after event loop did a cycle } 

However, it is not very efficient or accurate, and 5 milliseconds is a lot.

How about your second approach, flow?

Alternatively, you can use threads with threads_a_gogo, which seems to suit your use very well for creating background threads.

NOTE. This is not a viable example, threads_a_gogo does not work in node 0.10.x

They have a detailed example (I copy here) on GitHub showing how to use streams with an event emitter:

quickIntro_evented_childThreadCode.js

// This is the code that .load () ed into the child / background stream:

 function fibo (n) { return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1; } thread.on('giveMeTheFibo', function onGiveMeTheFibo (data) { this.emit('theFiboIs', fibo(+data)); //Emits 'theFiboIs' in the parent/main thread. }); 

main file:

 var thread= require('threads_a_gogo').create(); thread.load(__dirname + '/quickIntro_evented_childThreadCode.js'); //Emit 'giveMeTheFibo' in the child/background thread. thread.emit('giveMeTheFibo', 35); //Listener for the 'theFiboIs' events emitted by the child/background thread. thread.on('theFiboIs', function cb (data) { process.stdout.write(data); this.emit('giveMeTheFibo', 35); }); (function spinForever () { process.stdout.write("."); process.nextTick(spinForever); })(); 

This type of event emitter, while the main application captures events in a stream suitable for your use case.

What will I do:

I also won’t write all the data that I need to crunch in the background in a parallel queue in some kind of data store (possibly redis) and perform asynchronous recording from NodeJS. Then I would read them with my "task", the code (process, yes) to complete these tasks.

Thus, it is clear that this task is not part of the normal server flow.