Throttle Requests at Node.js

Question

I have an array. I can iterate over it using the foreach method.

data.forEach(function (result, i) { url = data[i].url; request(url); }); 

The request function makes an http request to the specified URL. However, the simultaneous execution of all these queries leads to different problems.

So, I thought that I should slow down the process by introducing some kind of timer.

But I have no idea how you can combine the forach loop with setTimeOut / setInterval

Please note that I am doing this on the server (nodejs) and not in the browser.

Thanks for the help.

Answer 1

Instead of setTimeout you can run them sequentially. I assume there is a callback the request() parameter for your request() function.

 function makeRequest(arr, i) { if (i < arr.length) { request(arr[i].url, function() { i++; makeRequest(arr, i); }); } } makeRequest(data, 0); 

---

If you need a bit more time between requests, add setTimeout to the callback.

 function makeRequest(arr, i) { if (i < arr.length) { request(arr[i].url, function() { i++; setTimeout(makeRequest, 1000, arr, i); }); } } makeRequest(data, 0); 

Answer 2

Since your problem is global, you must configure the request function to execute all 5 requests at once - using the global static counter. If your request was before something like

 function request(url, callback) { ajax(url, callback); } 

now use something like

 var count = 0; var waiting = []; function request(url, callback) { if (count < 5) { count++; ajax(url, function() { count--; if (waiting.length) request.apply(null, waiting.shift()); callback.apply(this, arguments); }); } else waiting.push(arguments); } 

Answer 3

 data.forEach(function (result, i) { url = data[i].url; setTimeout( function () { request(url); }, 1000 * (i + 1) // where they will each progressively wait 1 sec more each ); }); 

Answer 4

you can delay the call with setTimeout. The following code ensures that every request will be called after timerMultiPlier milliseconds from the previous request.

 var timerMultiPlier = 1000; data.forEach(function (result, i) { setTimeout(function(){ url = data[i].url; request(url); }, timerMultiPlier*i ); }); 

Answer 5

You can compensate for the delay in execution of each element by index, for example:

 data.forEach(function (result, i) { setTimeout(function() { url = data[i].url; request(url); }, i * 100); }); 

This will cause each iteration to take about 100 milliseconds after the previous one. You can change 100 to any number you want to change the delay.