How to provide exclusive reads in MongoDb findAndModify?

Question

I have a MongoDB collection (used as a job queue) from which several processes read records using findAndModify . findAndModify searches for records in which the active field is false, setting it to true so that other processes do not read the same record.

The problem is that looking at the logs, I see that different processes are still reading the same notes. This occurs when two processes are read from the queue at the same time. Is there a way to make sure that only one process is read from a collection at a time?

I am using Mongo 2.2.3 and pymongo 2.2.

Thank you very much!

EDIT: The magazines in question are:

worker.3 2013-03-18 23: 57: 45,434 default-worker-3
project_name INFO Queue task: job ID: 5147a90f68e8fe0097002bdf

worker.3 2013-03-18 23: 57: 47 608 default-worker-3
project_name INFO Login: 14497 documents

worker.2 2013-03-18 23: 57: 45 440 default-worker-2
project_name INFO Queue task: job ID: 5147a90f68e8fe0097002bdf

worker.2 2013-03-18 23: 57: 47,658 default-worker-2
project_name INFO Login: 14497 documents

As you can see, worker.3 and worker.2 are reading the same job from the queue (it has the same mongodb id for both workers).

Find_and_modify command:

 query = {"active": False} try: return self.collection.find_and_modify( query=query, update={"$set": {"active": True}}, upsert=False, sort={"added_on": 1}, limit=1 ) except Exception, exc: LOGGER.exception(exc) 

Answer 1

First make a simple request to mongodb to get one job entry,

 job = db.coll.find({query}).limit(1) 

Then update the entry with the job id and where active=false

 update_response = db.coll.update( {_id:job.id, active=false}, {$set:{active:true}}, false, false ) 

If the task has already been updated by another process, the update will not be completed due to the request limit active=false . Go to update_response that the record has been updated:

 if update_response['n'] > 0 and update_response['updatedExisting']==true: return job 

If your update failed, select another task and try again.

Answer 2

Let me make something very clear - two different findAndModify commands cannot return the same document in your script.

Impossible. Here are the first few lines of the method that does the work:

  Lock::DBWrite lk( ns ); Client::Context cx( ns ); BSONObj doc; bool found = Helpers::findOne( ns.c_str() , queryOriginal , doc ); 

Mark line 122 where the WRITE lock will be obtained before the search.

https://github.com/mongodb/mongo/blob/master/src/mongo/db/commands/find_and_modify.cpp#L122

It is not possible for two processes to maintain a write lock at the same time. It seems more likely that something else happens (several documents having the same id value, the same document returned by the function that calls find_and_modify, returning it in two threads, something else we don’t know enough to reflect on).

FindAndModify is an atomic command that contains an exclusive write lock for the duration of its execution. My suggestion was to figure out what the logs really show you, and not make changes to your code based on incorrect / unfounded assumptions about what should happen.

Answer 3

Divide your “lock” in two steps. The first update record that requests the first object that does not have a lock time stamp or has an expired time stamp and sets a new lock. Then just find the same object using the lock data you just set.