Jython: subprocess.Popen runs out of file descriptors

Question

I am using Jython 2.51 to implement Python to write a script that re-calls another process via subprocess.Popen and uses PIPE to connect stdout and stderr to the parent process and stdin to the child process. After several hundred iteration cycles, I seem to have run out of file descriptors.

The Python subprocess documentation has very little mention of releasing file descriptors, except for the close_fds option, which is not described very clearly (Why should there be any file descriptors other than 0, 1, and 2 in the first place?). I assume that in CPython, reference counting takes care of the problem of freeing resources. What is the correct way to make sure all descriptors are freed when everything is done with a Popen object in Jython?

Edit: just in case, it matters, it's a multi-threaded program, so there are several Popen processes running at the same time.

Answer 1

This is only the answer to part of your question, but I understand that when you create a new process, it usually inherits all the descriptors of the parent process. This includes such open files and sockets that you are listening to.

On UNIX, this is a side effect of using "fork", which duplicates the current process and all its descriptors before loading a new executable. On Windows, this is more explicit, but Python does it anyway to match behavior across platforms as much as possible.

The close_fds parameter, when True, closes all these inherited descriptors after spawning of the subprocess, so the new executable starts from scratch. But if your subprocesses are executed one at a time and end when they are finished, this should not be a problem.