MongoDB 2.4.8 limited set and tail cursor consuming all memory

Question

We are currently studying Capped Collections and Tailable Cursors in MongoDB to create a notification queue system. However, after creating the simple LinqPad test (code below) that we noticed at startup, Mongo constantly allocates memory until there are resources available, even if we do not insert any records. This allocation continues until all system RAM is used up, after which Mongo simply stops responding.

Since we're new to Capped Collections and Tailable Cursors, I wanted us not to miss something obvious before posting the error.

Note. We tried to use the logging code below with the same results.

• Platform : Windows Server 2012 64bit
• MongoDB : version 2.4.8 64bit
• Driver : official C # 10gen v1.8.3.9

Linqpad script

var conn = new MongoClient("mongodb://the.server.url").GetServer().GetDatabase("TestDB"); if(!conn.CollectionExists("Queue")) { conn.CreateCollection("Queue", CollectionOptions .SetCapped(true) .SetMaxSize(100000) .SetMaxDocuments(100) ); //Insert an empty document as without this 'cursor.IsDead' is always true var coll = conn.GetCollection("Queue"); coll.Insert( new BsonDocument(new Dictionary<string, object> { { "PROCESSED", true }, }), WriteConcern.Unacknowledged ); } var coll = conn.GetCollection("Queue"); var query = coll.Find(Query.EQ("PROCESSED", false)) .SetFlags(QueryFlags.AwaitData | QueryFlags.NoCursorTimeout | QueryFlags.TailableCursor); var cursor = new MongoCursorEnumerator<BsonDocument>(query); while(true) { if(cursor.MoveNext()) { string.Format( "{0:yyyy-MM-dd HH:mm:ss} - {1}", cursor.Current["Date"].ToUniversalTime(), cursor.Current["X"].AsString ).Dump(); coll.Update( Query.EQ("_id", cursor.Current["_id"]), Update.Set("PROCESSED", true), WriteConcern.Unacknowledged ); } else if(cursor.IsDead) { "DONE".Dump(); break; } } 

Answer 1

I seem to have found a solution to the problem !!

The problem in the above code revolves around the request:

 Query.EQ("PROCESSED", false) 

When I removed this and replaced it with a query based on the document id, the memory consumption issue disappeared. Further thinking, this “PROCESSED” property is really not required in the query as cursor.MoveNext () will always return the next new document (if any). Heres a reorganized LinqPad script based on the above code ....

 var conn = new MongoClient("mongodb://the.server.url").GetServer().GetDatabase("TestDB"); if(conn.CollectionExists("Queue")) { conn.DropCollection("Queue"); } conn.CreateCollection("Queue", CollectionOptions .SetCapped(true) .SetMaxSize(100000) .SetMaxDocuments(100) .SetAutoIndexId(true) ); //Insert an empty document as without this 'cursor.IsDead' is always true var coll = conn.GetCollection("Queue"); coll.Insert( new BsonDocument(new Dictionary<string, object> { { "PROCESSED", true }, { "Date", DateTime.UtcNow }, { "X", "test" } }), WriteConcern.Unacknowledged ); //Create query based on latest document id BsonValue lastId = BsonMinKey.Value; var query = coll.Find(Query.GT("_id", lastId)) .SetFlags(QueryFlags.AwaitData | QueryFlags.NoCursorTimeout | QueryFlags.TailableCursor); var cursor = new MongoCursorEnumerator<BsonDocument>(query); while(true) { if(cursor.MoveNext()) { string.Format( "{0:yyyy-MM-dd HH:mm:ss} - {1}", cursor.Current["Date"].ToUniversalTime(), cursor.Current["X"].AsString ).Dump(); } else if(cursor.IsDead) { "DONE".Dump(); break; } } 

Answer 2

The same thing here - without this extra request.

After another investigation (actually VERY MUCH MORE) i the problem found is as follows:

If the first MoveNext does not return a record, a problem exists. It doesn't matter which request. It doesn't matter how many entries are in the collection.

If you change the query returning the last record, as the first result everything works fine. You can refuse it, because you already know it ...

The top example ends because you first retrieve ALL the records in the collection.