Run multiple python programs at the same time

Question

I have a python script run.py :

def do(i): # doing something with i, that takes time start_i = sys.argv[1] end_i = sys.argv[2] for i in range(start_i, end_i): do(i) 

Then I run this script:

 python run.py 0 1000000 

After 30 minutes, the script ends. But it is too long for me.

So, I create a bash script run.sh :

 python run.py 0 200000 & python run.py 200000 400000 & python run.py 400000 600000 & python run.py 600000 800000 & python run.py 800000 1000000 

Then I run this script:

 bash run.sh 

After 6 minutes, the script ends. Pretty good. I'm happy.

But I think there is another way to solve the problem (without creating a bash script), right?

Answer 1

You are looking for a multiprocessing package, and especially the Pool class:

 from multiprocessing import Pool p = Pool(5) # like in your example, running five separate processes p.map(do, range(start_i, end_i)) 

Besides combining this into a single command, this has other advantages compared to your approach to calling python run.py 0 200000 & etc. If some processes take longer than others (and therefore python run.py 0 200000 can finish before the rest), this will guarantee that all 5 threads continue to work until they are all executed.

Please note that depending on the architecture of your computer, starting too many processes at the same time may slow them down (firstly, it depends on the number of cores of your processor, as well as what else you use at the same time).

Answer 2

Your python program may create independent processes instead of bash, but that is not much different. What do you think of your decision that you are missing?