Spawning and waiting for child processes in Python

Question

The relevant part of the code is as follows:

pids = [] for size in SIZES: pids.append(os.spawnv(os.P_NOWAIT, RESIZECMD, [RESIZECMD, lotsOfOptions])) # Wait for all spawned imagemagick processes to finish while pids: (pid, status) = os.waitpid(0, 0) if pid: pids.remove(pid) 

What this should do is discard all processes, and then wait for each process to complete before continuing. What he does is work for the most part, but sometimes crashes in the next section (when he expects all of these processes to be completed).

Is there something wrong with this? Is there a better way to do this?

The environment it should work with is CentOS with Python 2.4, but I am testing Cygwin with Python 2.5, so it may happen that it does not work on my machine, but will work on Linux (the Linux machine is very slow, and this error is rare, so I could not get it).

Answer 1

The recommended way to start a subprocess is to use the subprocess module.

 pipe = Popen(["program", "arg1", "arg2"]) pipe.wait() 

Answer 2

I would recommend you install python-subprocess32 - a reliable backport of the Python 3 version of the standard subprocess library module, suitable for Python 2.4 to 2.7 and by far the best way to start subprocesses in Python 2. Then in a loop you will do

 pids.append(subprocess.Popen([RESIZECMD, lot, of, options]) 

and the next cycle will be simple

 for pid in pids: pid.wait()