MongoDB - paging

Question

When using MongoDB, are there any special templates for creating, for example. view watched? say, on a blog that lists the last 10 posts where you can navigate back to older posts.

Or allow it using an index e.g. blogpost.publishdate and just skip and limit the result?

Answer 1

Using the skip + limit is not a good way to swap when performance is a problem or with large collections; as the page number increases, it will be slower and slower. Using skip requires the server to go through all documents (or index values) from 0 to the offset value (skip).

It is much better to use a range query (+ limit) in which you pass the range value of the last page. For example, if you sort by “publishdate”, you simply pass the last value “publishdate” as query criteria to get the next page of data.

Answer 2

• A range-based subcategory is difficult to implement if you need to sort items in different ways.
• Remember, if the value of the sort parameter field is not unique, then swapping based on the range will become unrealistic.

Possible solution: try to simplify desgin, thinking about whether we can sort only by id or some unique value?

And if we can, then we can use range-based mapping.

A common way is to use sort (), skip () and limit () to implement swap, as described above.

Answer 3

This is the solution I used when my collection became too big to return in one request. It uses the built-in _id field _id and allows you to scroll through the collection using the specified batch size.

Here it is the npm module, mongoose-paging , the full code is below:

 function promiseWhile(condition, action) { return new Promise(function(resolve, reject) { process.nextTick(function loop() { if(!condition()) { resolve(); } else { action().then(loop).catch(reject); } }); }); } function findPaged(query, fields, options, iterator, cb) { var Model = this, step = options.step, cursor = null, length = null; promiseWhile(function() { return ( length===null || length > 0 ); }, function() { return new Promise(function(resolve, reject) { if(cursor) query['_id'] = { $gt: cursor }; Model.find(query, fields, options).sort({_id: 1}).limit(step).exec(function(err, items) { if(err) { reject(err); } else { length = items.length; if(length > 0) { cursor = items[length - 1]._id; iterator(items, function(err) { if(err) { reject(err); } else { resolve(); } }); } else { resolve(); } } }); }); }).then(cb).catch(cb); } module.exports = function(schema) { schema.statics.findPaged = findPaged; }; 

Attach it to your model as follows:

 MySchema.plugin(findPaged); 

Then run the following queries:

 MyModel.findPaged( // mongoose query object, leave blank for all {source: 'email'}, // fields to return, leave blank for all ['subject', 'message'], // number of results per page {step: 100}, // iterator to call on each set of results function(results, cb) { console.log(results); // this is called repeatedly while until there are no more results. // results is an array of maximum length 100 containing the // results of your query // if all goes well cb(); // if your async stuff has an error cb(err); }, // function to call when finished looping function(err) { throw err; // this is called once there are no more results (err is null), // or if there is an error (then err is set) } ); 

Answer 4

Range-based swapping is doable, but you need to be smart about how you perform the minimum / maximum request.

If you can afford it, try caching the query results in a temporary file or collection. Thanks to the TTL collections in MongoDB, you can paste your results into two collections.

• Search + User + query parameters (TTL independently)
• Query results (TTL independently + cleanup interval + 1)

Using both confirmations, you will not get partial results when the TTL is close to the current time. You can use a simple counter when you save the results to query a VERY simple range at this point.

Answer 5

The following is an example of getting a list of CreatedDate orders for CreatedDate documents (where pageIndex zero-based) using the official C # driver.

 public void List<User> GetUsers() { var connectionString = "<a connection string>"; var client = new MongoClient(connectionString); var server = client.GetServer(); var database = server.GetDatabase("<a database name>"); var sortBy = SortBy<User>.Descending(u => u.CreatedDate); var collection = database.GetCollection<User>("Users"); var cursor = collection.FindAll(); cursor.SetSortOrder(sortBy); cursor.Skip = pageIndex * pageSize; cursor.Limit = pageSize; return cursor.ToList(); } 

All sorting and swapping operations are performed on the server side. Although this is an example in C #, I think the same can be applied to other language ports.

See http://docs.mongodb.org/ecosystem/tutorial/use-csharp-driver/#modifying-a-cursor-before-enumerating-it .

Answer 6

  // file:ad-hoc.js // an example of using the less binary as pager in the bash shell // // call on the shell by: // mongo localhost:27017/mydb ad-hoc.js | less // // note ad-hoc.js must be in your current directory // replace the 27017 wit the port of your mongodb instance // replace the mydb with the name of the db you want to query // // create the connection obj conn = new Mongo(); // set the db of the connection // replace the mydb with the name of the db you want to query db = conn.getDB("mydb"); // replace the products with the name of the collection // populate my the products collection // this is just for demo purposes - you will probably have your data already for (var i=0;i<1000;i++ ) { db.products.insert( [ { _id: i, item: "lamp", qty: 50, type: "desk" }, ], { ordered: true } ) } // replace the products with the name of the collection cursor = db.products.find(); // print the collection contents while ( cursor.hasNext() ) { printjson( cursor.next() ); } // eof file: ad-hoc.js