PyMongo raises [errno 49] cannot assign the requested address after a large number of requests

Question

I have a collection of MongoDB s> 1,000,000 documents. I execute the initial .find({ my_query }) to return a subset of these documents (~ 25,000 documents), which I then put into the list object.

Then I iterate over each of the objects, analyzing some values ​​from the returned document in the list and performing an additional query, using these analyzed values ​​through the code:

 def _perform_queries(query): conn = pymongo.MongoClient('mongodb://localhost:27017') try: coll = conn.databases['race_results'] races = coll.find(query).sort("date", -1) except BaseException, err: print('An error occured in runner query: %s\n' % err) finally: conn.close() return races 

In this case, my dictionary query :

 {"$and": [{"opponents": {"$elemMatch": {"$and": [ {"runner.name": name}, {"runner.jockey": jockey} ]}}}, {"summary.dist": "1"} ]} 

Here is my problem. I created an index on opponents.runner.name and opponents.runner.jockey . This makes the queries really very fast. However, after approximately 10,000 requests in a row, pymongo throws an exception:

 pymongo.errors.AutoReconnect: [Errno 49] Can't assign requested address 

When I delete the index, I do not see this error. But for each request it takes about 0.5 seconds , which is unsuitable for my case.

Does anyone know why [Errno 49] can't assign requested address can occur? I saw several other SO questions related to can't assign requested address but not related to pymongo , and the answers there do not lead anywhere.

UPDATE:

Following Serge's advice below, here is the result of ulimit -a :

 core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 2560 pipe size (512 bytes, -p) 1 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 709 virtual memory (kbytes, -v) unlimited 

My MongoDB runs on OS X Yosemite.

Answer 1

This is due to misuse of PyMongo. You create a new MongoClient for each request, which requires you to open a new socket for each new request. This wins the PyMongo connection pool, and in addition, it is very slow, it also means that you open and close sockets faster than your TCP stack can support: you leave too many sockets in TIME_WAIT state to end up leaving ports .

Fortunately, the fix is ​​simple. Create one MongoClient and use it in everything:

 conn = pymongo.MongoClient('mongodb://localhost:27017') coll = conn.databases['race_results'] def _perform_queries(query): return coll.find(query).sort("date", -1)