Geek Answers Handbook

Python subprocess exhausts file descriptors

I have a long python project that uses a subprocess module to run various other programs. It waits for the completion of each program, then terminates the shell and returns to the wait loop.

In the end, this leads to the fact that the computer on which it is running stops, with the error that there are no more file descriptors.

I cannot find anywhere in the docs subprocess what happens with file descriptors when the child process closes. At first, I thought that they would close automatically, since the subprocess.call () command waits until the child completes.

But if that were the case, I would not have a problem. I also thought that if something remains, python will garbage collect it when the function ends and the file descriptors go out of scope. But this does not seem to be the case.

How do I access these file descriptors? the subprocess.call () function returns the exit code, and does not open the file descriptors. Is there something else I'm missing here?

This project acts as a glue between various corporate applications. These applications cannot be pipelined, and they are gui systems. So, the only thing I can do is run them using the built-in macros. These macros output text files that I use for the next program in the channel.

Yes, it is as bad as it seems. Fortunately, all files have fairly unique names. So, here, in the next few days, I will use the built-in sys tool suggested below to try and find the file. I will let you know how it turns out.

Most of the files that I don’t open, I just move them with the win32file.CopyFile () function.

+5
source share
4 answers

I had the same problem.

subprocess.Popen() Windows. - , . , subprocess.Popen - Windows, Linux.

Popen (, - , , ), , , Windows, Linux Popen.communicate(). , "" Python IOError: [Errno 24] Too many open files.

Python

, Python script. , script. , 0 100, .

fd_table_status.py:

import os
import stat

_fd_types = (
    ('REG', stat.S_ISREG),
    ('FIFO', stat.S_ISFIFO),
    ('DIR', stat.S_ISDIR),
    ('CHR', stat.S_ISCHR),
    ('BLK', stat.S_ISBLK),
    ('LNK', stat.S_ISLNK),
    ('SOCK', stat.S_ISSOCK)
)

def fd_table_status():
    result = []
    for fd in range(100):
        try:
            s = os.fstat(fd)
        except:
            continue
        for fd_type, func in _fd_types:
            if func(s.st_mode):
                break
        else:
            fd_type = str(s.st_mode)
        result.append((fd, fd_type))
    return result

def fd_table_status_logify(fd_table_result):
    return ('Open file handles: ' +
            ', '.join(['{0}: {1}'.format(*i) for i in fd_table_result]))

def fd_table_status_str():
    return fd_table_status_logify(fd_table_status())

if __name__=='__main__':
    print fd_table_status_str()

:

$> python fd_table_status.py
Open file handles: 0: CHR, 1: CHR, 2: CHR
$>

, fd_table_status_str() Python. "CHR" " " . Python stat.

script Linux Windows:

test_fd_handling.py:

import fd_table_status
import subprocess
import platform

fds = fd_table_status.fd_table_status_str

if platform.system()=='Windows':
    python_exe = r'C:\Python27\python.exe'
else:
    python_exe = 'python'

print '1) Initial file descriptors:\n' + fds()
f = open('fd_table_status.py', 'r')
print '2) After file open, before Popen:\n' + fds()
p = subprocess.Popen(['python', 'fd_table_status.py'],
                     stdin=subprocess.PIPE,
                     stdout=subprocess.PIPE,
                     stderr=subprocess.PIPE)
print '3) After Popen, before reading piped output:\n' + fds()
result = p.communicate()
print '4) After Popen.communicate():\n' + fds()
del p
print '5) After deleting reference to Popen instance:\n' + fds()
del f
print '6) After deleting reference to file instance:\n' + fds()
print '7) child process had the following file descriptors:'
print result[0][:-1]

Linux

1) Initial file descriptors:
Open file handles: 0: CHR, 1: CHR, 2: CHR
2) After file open, before Popen:
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG
3) After Popen, before reading piped output:
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG, 5: FIFO, 6: FIFO, 8: FIFO
4) After Popen.communicate():
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG
5) After deleting reference to Popen instance:
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG
6) After deleting reference to file instance:
Open file handles: 0: CHR, 1: CHR, 2: CHR
7) child process had the following file descriptors:
Open file handles: 0: FIFO, 1: FIFO, 2: FIFO, 3: REG

Windows

1) Initial file descriptors:
Open file handles: 0: CHR, 1: CHR, 2: CHR
2) After file open, before Popen:
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG
3) After Popen, before reading piped output:
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG, 4: FIFO, 5: FIFO, 6: FIFO
4) After Popen.communicate():
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG, 5: FIFO, 6: FIFO
5) After deleting reference to Popen instance:
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG
6) After deleting reference to file instance:
Open file handles: 0: CHR, 1: CHR, 2: CHR
7) child process had the following file descriptors:
Open file handles: 0: FIFO, 1: FIFO, 2: FIFO

4, Windows , Linux. Popen .

Btw, 7 , Python Windows, .

+3

python ? .Popen(), subprocess.call()

http://bugs.python.org/issue6274

, python-2.6

+2

, , , , python.

heapy.

+1

File descriptors disappear when the process is running, so it must be the parent holding the file descriptors (this can be checked with lsof). What does the code do in the parent?

0
source

More articles:


All Articles