What is the best way to iterate or recurs through a huge number of huge functions without exceeding the stack limit?

Question

I have an application that I am writing in Node.js that has to handle a lot of configuration requests and databases to process user data. The problem I am facing is that after 11,800+ function calls, Node will throw an error and exit the process.

The error says: RangeError: maximum call stack size exceeded

I am curious if anyone else has such a situation, and know how they handle this. I have already started breaking my code into several additional working files, but every time I process Node data, it should touch 2 databases (no more than 25 calls to update various tables) and perform a number of sanitary checks.

I fully agree to admit that I might be doing something suboptimal if that is the case, but it would be helpful if there would be a better guide.

Here is an example of the code that I run for the data:

app.post('/initspeaker', function(req, res) { // if the Admin ID is not present ignore if(req.body.xyzid!=config.adminid) { res.send( {} ); return; } var gcnt = 0, dbsize = 0, goutput = [], goutputdata = [], xyzuserdataCallers = []; xyz.loadbatchfile( xyz.getbatchurl("speakers", "csv"), function(data) { var parsed = csv.parse(data); console.log("lexicon", parsed[0]); for(var i=1;i<parsed.length;i++) { if(typeof parsed[i][0] != 'undefined' && parsed[i][0]!='name') { var xyzevent = require('./lib/model/xyz_speaker').create(parsed[i], parsed[0]); xyzevent.isPresenter = true; goutput.push(xyzevent); } } dbsize = goutput.length; xyzuserdataCallers = [new xyzuserdata(), new xyzuserdata(), new xyzuserdata(), new xyzuserdata(), new xyzuserdata(), new xyzuserdata(), new xyzuserdata(), new xyzuserdata() ]; // insert all Scheduled Items into the DB xyzuserdataCallers[0].sendSpeakerData(goutput[0]); for(var i=1;i<xyzuserdataCallers;i++) { xyzuserdataCallers[i].sendSpeakerData(8008); } //sendSpeakerData(goutput[0]); }); var callback = function(data, func) { //console.log(data); if(data && data!=8008) { if(gcnt>=dbsize) { res.send("done"); } else { gcnt++; func.sendSpeakerData(goutput[gcnt]); } } else { gcnt++; func.sendSpeakerData(goutput[gcnt]); } }; // callback loop for fetching registrants for events from SMW var xyzuserdata = function() {}; xyzuserdata.prototype.sendSpeakerData = function(data) { var thisfunc = this; if(data && data!=8008) { //console.log('creating user from data', gcnt, dbsize); var userdata = require('./lib/model/user').create(data.toObject()); var speakerdata = userdata.toObject(); speakerdata.uid = uuid.v1(); speakerdata.isPresenter = true; couchdb.insert(speakerdata, config.couch.db.user, function($data) { if($data==false) { // if this fails it is probably due to a UID colliding console.log("*** trying user data again ***"); speakerdata.uid = uuid.v1(); arguments.callee( speakerdata ); } else { callback($data, thisfunc); } }); } else { gcnt++; arguments.callee(goutput[gcnt]); } }; }); 

A couple of classes and elements that need to be introduced are described here:

• I am using Express.js + hosted by CouchDB and this responds to a POST request.
• There is a CSV parser class that loads a list of events that drive speaker data.
• Each event can contain n users (currently about 8K users for all events)
• I use a template that loads all data / users before trying to parse them.
• Each loaded by the user (external data source) is converted into an object that I can use, as well as disinfect (slashes, etc.)
• Each user is then inserted into CouchDB

This code works in the application, but after a while I get an error that more than 11 800 + calls were made, and the application breaks. This is not an error that contains a stack trace, as if it were a code error, it exits due to the number of calls made.

Again, any help / comment / direction would be appreciated.

Answer 1

It seems that xyzuserdata.sendSpeakerData and the callback are used recursively to support consecutive database calls. At some point, your call stack ends ...

There are several modules that facilitate serial execution, such as Step or Flow-JS .

Flow-JS even has a convenience function for sequentially applying a function to array elements:

 flow.serialForEach(goutput, xyzuserdata.sendSpeakerData, ...) 

I wrote a small test program using flow.serialForEach, but unfortunately I was able to get the Maximum call stack size exceeded error. Flow-JS seems to use the call stack in the same way as keeping synchronization.

Another approach that does not create a call stack is to avoid recursion and use setTimeout with a timeout of 0 to schedule a callback. See http://metaduck.com/post/2675027550/asynchronous-iteration-patterns-in-node-js

You can try replacing the callback call with

 setTimeout(callback, 0, [$data, thisfunc]) 

Answer 2

Recursion is very useful for synchronizing asynchronous operations - why is it used in flow.js, etc.

However, if you want to process an unlimited number of elements in an array or a buffered stream, you will need to use node.js event emitter.

in pseudo code :

  ee = eventemitter arr = A_very_long_array_to_process callback = callback_to_call_once_either_with_an_error_or_when_done // the worker function does everything processOne() { var next = arr. shift(); if( !arr ) ee.emit ( 'finished' ) return process( function( err, response) { if( err ) callback( err, response ) else ee.emit( 'done-one' ) } ); } // here we process the final event that the worker will throw when done ee.on( 'finished', function() { callback( null, 'we processed the entire array!'); } ); // here we say what to do after one thing has been processed ee.on( 'done-one', function() { processOne(); } ); // here we get the ball rolling processOne();