Python multiprocessing process.

Question

I am new to python, and in particular to the multiprocessing module. However, I managed to write a very simple script to run several processes (say 100) at 24 cpus each. However, I noticed that the process is not executed sequentially, but randomly. Is there a way to execute processes sequentially. Here is my code:

#!/usr/bin/env python import multiprocessing import subprocess def prcss(cmd): sbc = subprocess.call com = sbc(cmd, shell='True') return (com) if __name__=='__main__': cmd = [] for j in range(1,11): for i in range(10): sis = '~/codes-paul/sisyphus/sisyphus '+str(j)+'/sisyphus.setup > '+str(j)+'/out'+str(i)+'.dat' cmd.append(sis) pool=multiprocessing.Pool(processes=24) pool.map(prcss,cmd) 

After I run the python code, I do "ps -ef | grep myname". Instead of getting:

 '/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out0.dat.dat' '/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out1.dat.dat '/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out2.dat.dat '/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out3.dat.dat . . . . .I am getting: '/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out0.dat.dat' '/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out3.dat.dat '/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out6.dat.dat '/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out9.dat.dat . . . . 

Any idea why the commands are not running sequentially?

Answer 1

Since you are creating a process pool, the commands are actually run sequentially, but you have no guarantee that the process will be completed first. You will notice that the order will change almost every time you run your code.