Clean up Python Popen

Question

I wanted to use the python equivalent to connect some shell commands in perl. Something like an open version of python (PIPE, "command |").

I go to the subprocess module and try:

p = subprocess.Popen("zgrep thingiwant largefile", shell=True, stdout=subprocess.PIPE) 

This works to read the output in the same way as in perl, but it does not clear. When I exit the interpreter, I get

 grep: writing output: Broken pipe 

spewed all over stderr several million times. Probably, I naively hoped that all this would take care of me, but this is not so. Calling a termination or killing on p does not seem to help. Look at the process table, I see that it kills the / bin / sh process, but leaves the child gzip in place to complain about the broken pipe.

What is the right way to do this?

Answer 1

The problem is that the pipe full. The subprocess stops, waiting for the pipe to fail, but then your process (the Python interpreter) terminates by breaking its end (hence the error message).

p.wait() will not help:

Warning This will be inhibited if the child process generates sufficient output to the stdout or stderr channel, so that it blocks waiting for the OS buffer to receive more data. Use communicate() to avoid this.

http://docs.python.org/library/subprocess.html#subprocess.Popen.wait

p.communicate() will not help:

Note Reading data is buffered in memory, so do not use this method if the data size is large or unlimited.

http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate

p.stdout.read(num_bytes) will not help:

Warning Use communicate() rather than .stdin.write , .stdout.read or .stderr.read to avoid deadlocks because any of the other OS buffer buffers fills and blocks the child process.

http://docs.python.org/library/subprocess.html#subprocess.Popen.stdout

The moral of this story is that for great performance, subprocess.PIPE doom you to a certain failure if your program tries to read data (it seems to me that you should put p.stdout.read(bytes) in while p.returncode is None: , but the warning above indicates that this could lead to blocking).

The documents suggest replacing the shell with the following:

 p1 = Popen(["zgrep", "thingiwant", "largefile"], stdout=PIPE) p2 = Popen(["processreceivingdata"], stdin=p1.stdout, stdout=PIPE) output = p2.communicate()[0] 

Note that p2 takes its standard input directly from p1 . This should avoid deadlocks, but given conflicting warnings above, who knows.

In any case, if this last part does not work for you (it should, however), you can try to create a temporary file, write all the data from the first call to it, and then use the temporary file as an input to the next process.

Answer 2

After opening the channel, you can work with the output of the command: p.stdout :

 for line in p.stdout: # do stuff p.stdout.close() 

Answer 3

How did you complete this process?

The correct way is to use

 p.communicate() 

See the docs for more details.

Answer 4

You need to wait to complete the process:

 import subprocess p = subprocess.Popen("cat /mach_kernel", shell=True) p.wait() 

Alternatively, you can write the standard output of the program (like yours) and, possibly, its standard error, and then call communicate :

 import subprocess p = subprocess.Popen("cat /mach_kernel", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) stdout, stderr = p.communicate()