How can I invoke a command using Python by capturing stderr and stdout without waiting for stderr / stdout to close?

Question

Now I have a code that uses Popen.communicate()from a subprocess (setting stdin=PIPEand stderr=PIPE) to run a command and capture both stderr and stdout.

The problem is that it communicate()not only waits for the command to exit, but also expects stdout and stderr to be closed. The command I run spawns a child process that keeps stderr open, so although the command is complete (and appears as "nonexistent" in ps) communicate()it still hangs.

I just want to wait for the command to complete without waiting for stderr / stdout. But I still want to capture any stderr / stdout output specified during the execution of the command. The documentation for iswait() followed by a red box with a refusal:

This will be inhibited when using stdout = PIPE and / or stderr = PIPE, and the child process generates sufficient output to the channel so that it blocks waiting for the OS buffer to receive more data. Use the communication method () to avoid this.

Obviously, I also want to avoid deadlocks.

What is the correct way to accomplish this task?

Answer 1

", " " ps , , select fcntl, .. stdout/stderr .

: A B (cmd, ), B C (), A , B EOF:

#!/usr/bin/env python
import os
from select import select
from subprocess import Popen, PIPE

p = Popen(cmd, stdout=PIPE, stderr=PIPE, bufsize=0)
read_set = [p.stdout, p.stderr]
pipename = {p.stdout: "stdout", p.stderr: "stderr"}
timeout = 0.5 # ugly but it works
while read_set and p.poll() is None: # while subprocess is running or until EOF
    for pipe in select(read_set, [], [], timeout)[0]:
        data = os.read(pipe.fileno(), 1<<30)
        if data:
            print("got from %s: %r" % (pipename[pipe], data))
        else: # EOF
            pipe.close()
            read_set.remove(pipe)
print("exit code %s" % (p.wait(),))

# child exited, wait for grandchild to print
for pipe in read_set:
    print("read the rest of %s: %r" % (pipename[pipe], pipe.read()))
    pipe.close()

cmd:

import sys
from textwrap import dedent

cmd = [sys.executable, '-u', '-c', dedent("""
    # inception style
    import os
    import sys
    from subprocess import Popen
    from textwrap import dedent

    Popen([sys.executable, '-u', '-c', dedent('''
        import os
        import sys
        import time

        time.sleep(60)
        print("grandchild %d done" % os.getpid())
        sys.stderr.write("grandchild stderr")
        sys.exit(20)
    ''')]) # stdout/stderr are not redirected

    print('child %d done' % os.getpid())
    sys.stderr.write('child stderr')
    sys.exit(19)
""")]

Answer 2

. , , - std.err. . (1.py):

import time
import sys

for i in xrange(1000):
    print >>sys.stderr, "hello", i
    time.sleep(1)

(example2.py):

import subprocess

cmd = "python example1.py"

print subprocess.check_output(cmd, shell=True)

example2.py script:

python example2.py ( , :)):

hello 0
hello 1
hello 2
hello 3
hello 4
hello 5
hello 6

, , , ,