Node.JS + mongo: .find (). each () stops after the first batch

Question

It puzzled me.

I have a standalone (on the command line) node script, the purpose of which is to iterate through all the documents in a large collection (several hundred thousand of them) and perform several calculations for each document, run a little extra JS code, and then refresh the document with some new values.

In the documentation for cursor.each() , as soon as I have my cursor from collection.find() , the .each(cb) method should execute cb(item) for each item in the entire collection.

Code example:

 myDb.collection('bigcollection').find().each(function(err, doc) { if (err) { console.log("Error: " + err); } else { if (doc != null) { process.stdout.write("."); } else { process.stdout.write("X"); } } }); 

What I expect is to print a few hundred thousand . and then print X at the end, because cursor.each() should "Iterate all documents for this cursor" and in the code example: "If the element is zero, then the cursor is exhausted / empty and closed."

But what he actually does is print exactly 101 . , without X at the end.

If I adjust the batch size ( .find().batchSize(10).each(... ), it goes through this number of documents before it crashes.

So why is it just processing the first batch? Am I somehow reading the documentation for .each () incorrectly? Does this have to do with the fact that this is a command line script, and somehow the entire script exits before the second batch of results is returned or something else? If so, how can I make sure that it actually processes all the results?

As a node side, I tried using .stream () and .forEach (), and in both of these cases it overloads after the first batch.

UPDATE: Well, that is interesting. Just tried to connect to my production server instead of my mongo instance on localhost and voila, it scans the entire collection as it should. The server is running mongodb 3.0.6, my local instance is 3.2.3. My version of the node mongodb driver is 2.0.43.

Answer 1

I have 200 documents in my collection and the following code is going well. In other words, the problem could not be reproduced. As you can see, I reduced the batch size to 10.

 var url = 'mongodb://localhost:27017/test'; MongoClient.connect(url, function(err, db) { if (err) { console.log(err); } else { var counter = 0; db.collection('collection').find({}).batchSize(10).each(function(e, r){ if(err){ console.log("E: " + err); db.close(); } else{ if(r == null){ db.close(); } else{ counter += 1; console.log("X: " + counter); } } }); } }); 

If you are still facing the same problem, I would suggest updating the MongoDB driver to the latest version. Since drivers are being actively developed, sometimes errors get into the released version, causing strange behavior.