Geek Answers Handbook

Is it possible to avoid $ unwind / aggregation on a large array with $ elemMatch and a regular request?

I have a set of documents (call it "logs") that looks something like this:

{ "_id" : ObjectId("52f523892491e4d58e85d70a"), "ds_id" : "534d35d72491de267ca08e96", "eT" : NumberLong(1391784000), "vars" : [{ "n" : "ActPow", "val" : 73.4186401367188, "u" : "kWh", "dt" : "REAL", "cM" : "AVE", "Q" : 99 }, { "n" : "WinSpe", "val" : 3.06327962875366, "u" : "m/s", "dt" : "REAL", "cM" : "AVE", "Q" : 99 }] }

The vars array contains about 150 subdocuments, not just the two shown above. Now I would like to execute a query that extracts val from two vars in the vars array, which I showed above.

Using the aggregation structure, I was able to come up with the following:

 db.logs.aggregate( [ { $match : { ds_id: "534d35d72491de267ca08e96", eT: { $lt : 1391784000 }, vars: { $elemMatch: { n: "PowCrvVld", val: 3 }} } }, { $unwind : "$vars" }, { $match : { "vars.n" : { $in : ["WinSpe", "ActPow"] }}, { $project : { "vars.n" : 1, N : 1} } ]);

While this works, I am running against the limit of 16 MB when running large queries. If I have about 150 subdocuments in a vars array, I would also like to avoid $unwind , if possible.

Using a regular request and using $elemMatch , I was able to get ONE of the values:

 db.logs.TenMinLog.find({ ds_id : "534d35d72491de267ca08e96", eT : { $lt : 1391784000 }, vars : { $elemMatch : { n : "PowCrvVld", val : 3 } } }, { ds_id : 1, vars : { $elemMatch : { n : "ActPow", cM : "AVE" } });

As for my question, there is a way to use $ elemMatch in an array several times in the <projection> part of the search. If not, is there another way to easily get these two nested documents without using $ unwind? I am also open to other proposals that will be more effective, which I may not suspect. Thanks!

0
source share
3 answers

If you are using MongoDB 2.6, you can use $ redact to trim elements from the vars array.

In MongoDB 2.6, you can also return results as a cursor to avoid the 16 MB limit. From the docs:

In MongoDB 2.6, an aggregate command can return results as a cursor or store results in a collection that do not fall within the size limit. Db.collection.aggregate () returns a cursor and result sets of any size can be returned.

+1
source

I would think a lot about upgrading to MongoDB 2.6. Aggregation has been improved to return a cursor that removes the 16 MB limit of the document:

Changed in version 2.6:

The db.collection.aggregate () method returns a cursor and result sets of any size can be returned. Previous versions returned all the results to a single document, and a size limit of 16 megabytes was set in the result set.

http://docs.mongodb.org/manual/core/aggregation-pipeline/

There are also a number of improvements that may be useful for more complex aggregation requests:

Aggregation improvements

The aggregation pipeline adds the ability to return result sets of any size, either by returning a cursor or writing output to a collection. In addition, the aggregation pipeline supports variables and adds new operations for processing sets and editing data.

Now db.collection.aggregate () returns a cursor that allows the aggregation pipeline to return result sets of any size. aggregation pipelines now support an explanation operation to help analyze aggregation. Aggregation can now use a more efficient sorting process based on an external drive.

New pipeline stages:

  • $ out to output to the collection.
  • $ redact stage to provide additional control over data access.

New or changed operators:

  • set expression statements.
  • $ let and $ map allow you to use variables.
  • $ literal and the $ size operator.
  • the $ cond expression now accepts either an object or an array.

http://docs.mongodb.org/manual/release-notes/2.6/

+1
source

Maybe it works.

 db.logs.TenMinLog.find({ ds_id : "534d35d72491de267ca08e96", eT : { $lt : 1391784000 }, vars : { $or: [{ $elemMatch : { n : "PowCrvVld", val : 3 }, { $elemMatch : { n : <whatever>, val : <whatever> }] } } }, { ds_id : 1, vars : { $elemMatch : { n : "ActPow", cM : "AVE" } });

Hope it works the way you want.

0
source

More articles:


All Articles