Polling method subprocess.popen return None for a long process

Question

I execute the curl command through a subprocess. This curl command starts video processing on another server and waits for a response. Once the process is complete, the remote server will return a json object. I check the status of the subprocess using the poll () value, which is None - the process is not completed, 0 - the process completed successfully and 1- for an error.

I get the correct answer if the processing takes about 30 minutes / or less on the remote server, but if the processing takes longer, I get only "No", although I see that the remote server is complete and already returned the json object.

Can someone tell me what is the possible reason for the polling () returning only None after a certain time. Thank you in advance.

My Popen Object:

object = subprocess.Popen(str(curlCmd), shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) 

and I call object.poll() every 2 seconds to check if the process completed successfully.

Answer 1

It seems like a known issue in Popen.poll you can try to use the solutions described in this link. http://www.gossamer-threads.com/lists/python/bugs/633489

Answer 2

.poll() is None means the child is still working.

The curl process may freeze as soon as it fills the stdout or stderr OS buffer line (about 64K in my Linux box) if you are not reading from your end of the pipe.

That is, while you are waiting for the completion of the child, the child is waiting for you to wipe the buffer buffer - a dead end . From subprocess docs :

This [ .wait() ] will be a dead end when using stdout=PIPE or stderr=PIPE , and the child process generates sufficient output to the pipe so that it blocks waiting for the operating system buffer to receive more data. Use Popen.communicate() when using pipes to avoid this.

You can use async.io streams to use pipes at the same time.

Answer 3

Finally, I ended up using PyCurl instead of creating a subprocess and invoking the curl command through it. It seems like this is a subprocess with a high level of errors, where the .poll method does not return any after a certain time, the reason is still unclear. I would like to inform people who use the subprocess-poll method (without waiting / communication) to know about this if you use a lengthy process. Thanks, JF Sebastian and Pranav for your guidance.

Answer 4

In python 2.7 and possibly 3.x you can use:

 object._internal_poll(_deadstate=127) 

instead of the usual Popen.poll() as a workaround. This will return 127 instead of None if the process is complete.

Of course, this is an internal module method, and there is no guarantee that it will work after updating the Python library.

Answer 5

In addition to the hegemon, this code works on 2.7:

 process = subprocess.Popen(cmd, cwd=current_dir, shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE ) process._internal_poll(_deadstate='dead') while timeout > 0: if process.poll() is not 'dead':