Creating child processes inside a child process with multiprocessing Python

Question

I observed this behavior when trying to create nested child processes in Python. Here is the parent program parent_process.py :

 import multiprocessing import child_process pool = multiprocessing.Pool(processes=4) for i in range(4): pool.apply_async(child_process.run, ()) pool.close() pool.join() 

The parent program calls the run function in the following child_process.py child program:

 import multiprocessing def run(): pool = multiprocessing.Pool(processes=4) print 'TEST!' pool.close() pool.join() 

When I run the parent program, nothing was printed, and the program quickly exited. However, if print 'TEST!' moves one line up (before creating nested child processes), 'TEST!' printed 4 times.

Since errors in the child process will not be displayed, this seems to indicate that the program crashes when the child process creates its own nested child processes.

Can someone explain what is going on behind the scenes? Thanks!

Answer 1

According to multiprocessing documentation, daemon processes cannot spawn child processes.

multiprocessing.Pool uses daemon processes to ensure that they will not flow when your program is inaccessible.

Answer 2

As noxdafox said, multiprocessing.Pool uses daemon processes. I found a simple workaround that uses multiprocess.Process instead of multiprocess.Process :

Parent program:

 import multiprocessing import child_process processes = [None] * 4 for i in range(4): processes[i] = multiprocessing.Process(target=child_process.run, args=(i,)) processes[i].start() for i in range(4): processes[i].join() 

Children's program (named child_process.py ):

 import multiprocessing def test(info): print 'TEST', info[0], info[1] def run(proc_id): pool = multiprocessing.Pool(processes=4) pool.map(test, [(proc_id, i) for i in range(4)]) pool.close() pool.join() 

The output is 16 lines of TEST :

 TEST 0 0 TEST 0 1 TEST 0 3 TEST 0 2 TEST 2 0 TEST 2 1 TEST 2 2 TEST 2 3 TEST 3 0 TEST 3 1 TEST 3 3 TEST 3 2 TEST 1 0 TEST 1 1 TEST 1 2 TEST 1 3