Subprocess execution order in a multiprocessor module

Question

To start a multiprocessor with a for loop.

import os from multiprocessing import Process def run_proc(name): print('child process %s (%s) running ...' %(name,os.getpid())) if __name__ == '__main__': print('parent process %s.' %os.getppid()) for i in range(5): p = Process(target=run_proc,args=(str(i),)) print('process will start'+str(i)) p.start() p.join() print('process is end') 

I got the result.

 parent process 6497. process will start process will start child process 0 (6984) running ... process will start process will start process will start child process 2 (6986) running ... child process 1 (6985) running ... child process 3 (6987) running ... child process 4 (6988) running ... process is end 

Why is the subprocess created early on later?
Why can't I get the following result?

 parent process 6497. process will start process will start child process 0 (6984) running ... process will start process will start process will start child process 1 (6986) running ... child process 2 (6985) running ... child process 3 (6987) running ... child process 4 (6988) running ... process is end 

What Jean-Francois Fabre says about how to create the following result:

 parent process 6497. process will start child process 0 (9639) running ... process will start child process 1 (9640) running ... process will start child process 2 (9641) running ... process will start child process 3 (9643) running ... process will start child process 4 (9644) running ... process is end 

I can make it just change p.join in for loop, like below:

 import os from multiprocessing import Process def run_proc(name): print('child process %s (%s) running ...' %(name,os.getpid())) if __name__ == '__main__': print('parent process %s.' %os.getppid()) for i in range(5): p = Process(target=run_proc,args=(str(i),)) print('process will start'+str(i)) p.start() p.join() print('process is end') 

I want to know why my code leads to the following output

 parent process 6497. process will start process will start child process 0 (6984) running ... process will start process will start process will start child process 2 (6986) running ... child process 1 (6985) running ... child process 3 (6987) running ... child process 4 (6988) running ... process is end 

instead:

 parent process 6497. process will start process will start child process 0 (6984) running ... process will start process will start process will start child process 1 (6986) running ... child process 2 (6985) running ... child process 3 (6987) running ... child process 4 (6988) running ... process is end 

This is another problem.

Answer 1

Here you have a race condition between the main process that creates the processes and the processes that they are trying to start (and the subprocesses themselves).

Once you issue the p.start() command from the main process, the child process can start (and print). But the main process is also working on creating the next child process. Who will print the next line first? hard to understand. If the main process succeeds in creating the following child process, it is now a race condition between both children: what are you experiencing.

Processes can run in parallel, but they still have synchronization points when invoking the operating system. Anyone who reaches the OS first is first served (for example: print to the console).

Of course, if order is restored in the p.join() loop, it also cancels the multiprocessing effect, since the main process waits for the child process to complete before another one is created.

This usually doesn’t matter since you are doing parallel tasks.

First, I created processes in the understanding of the list, then a cycle to start them, with a slight delay, to make sure that the process is started and printed before the next process is created.

 process_list = [Process(target=run_proc,args=(str(i),)) for i in range(5)] for i,p in enumerate(process_list): print('process {} will start'.format(i)) p.start() time.sleep(0.1) 

When the main process waits for a while, it provides breathing space for the child process to start and print.

Also note that your last p.join() only joins the last process of the loop, it should be (now using our new process_list ):

 for p in process_list: p.join() 

Starting / ending in order is not very important in most cases. You can pre-calculate all the order information in the main process (for example, you did by assigning process numbers to increasing numbers).

Please note that the classic problem may not be ensuring that the processes start sequentially, but the result that they give can be compared with the input you provided (pass the list of inputs to the processes and give the list of outputs, so at the end you know which input provided, what is the conclusion).

In this case, find multiprocessing.pool and the map function ( sequentially launching Python processes multiprocessing.pool )