Python: Unable to start a new thread. <100 active threads

Question

I get the following error:

 ----- Match 93028: ------ Patch 5.11 ------78 Threads Active ----- Match 93029: ------ Patch 5.11 ------77 Threads Active ----- Match 93030: ------ Patch 5.11 ------76 Threads Active ----- Match 93031: ------ Patch 5.11 ------71 Threads Active ----- Match 93032: ------ Patch 5.11 ------55 Threads Active ----- Match 93033: ------ Patch 5.11 ------56 Threads Active ----- Match 93034: ------ Patch 5.11 ------57 Threads Active ----- Match 93035: ------ Patch 5.11 ------58 Threads Active ----- Match 93036: ------ Patch 5.11 ------59 Threads Active Traceback (most recent call last): File "pulldata.py", line 91, in <module> getPatchData('5.11', '511') File "pulldata.py", line 64, in getPatchData matchThread.start() File "/usr/lib/python3.4/threading.py", line 850, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't start new thread 

This is usually caused by having too many threads open, but as you can see, I also print the number of active threads. There are <100 active threads, so I'm not sure what the problem is. Here is the relevant code:

 slot = threading.BoundedSemaphore(value=1000) def getMatchData(index,match,patch): global requestsSent global logfile print("----- Match {0}: ------ Patch {1} ------{2} Threads Active".format(index,patch,threading.active_count())) logfile.write("Parsing Match {0} for patch {1}:\n".format(index,patch)) #match is a class. get is a function that sends a request to the server and returns a request object from where I get the json response. data = match.get().json() #processdata slot.release() def getPatchData(patch, name): global logfile threads = [] matches = getAllMatches(patch) for index, match in enumerate(matches): slot.acquire() matchThread = threading.Thread(target=getMatchData, args=(index,match,patch)) threads.append(matchThread) matchThread.start() for t in threads: if not t.isAlive(): threads.remove(t) for t in threads: t.join() 

The semaphore of slots should limit the number of active threads, but I don’t think I have ever reached 1000 threads. Before I assumed that this error was caused by the threads pointed to by my array of threads, so I added code to remove them from the array when they are no longer active.

I can’t understand why I get it, I can’t start a new thread when there are only 59 active threads.

Also, is there a better way to achieve what I'm trying to do? Each thread sends an API request. I tried to do this without concurrency, but I did not even come close to my limit.

Answer 1

I ran into a similar problem, and this is how I solved it.

I don’t know which OS the OP uses, but Linux usually has a limit on the number of processes per user . You can see it with ulimit -u (or also ulimit -a ). The definition is a little wrong, since the limit is actually in the number of OS threads (or LWP). (see accepted answer: https://superuser.com/questions/376532/does-gnu-linux-counts-processes-and-threads-together-when-i-limit-their-number )

On my system, the limit is set to 400 (but it can be changed by the administrator).

You can view a list of all your threads with the command:

ps -fLu <your_username>

In my case, my python application would throw the same exception that the OP is reporting, but threading.active_count () will return 7.

It turned out that I had many remaining processes from previous sessions (I was too keen on nohup ...), with several threads each hanging in the system. Removing from them got rid of the error creating the stream.