Import modules that use MultiProcessing Python

Question

I want to use the multiprocessing module to speed up the execution time of some transport planning models. I optimized as much as I could using the "normal" methods, but this is based on an absurdly parallel problem. For example, perform the same set of matrix operations with four 4 different sets of inputs, all independent information.

Pseudocode:

for mat1,mat2,mat3,mat4 in zip([a1,a2,a3,a4],[b1,b2,b3,b4],[c1,c2,c3,c4],[d1,d2,d3,d4]): result1 = mat1*mat2^mat3 result2 = mat1/mat4 result3 = mat3.T*mat2.T+mat4 

So, all I really want to do is handle iterations of this loop in parallel on a quad-core computer. I have read here other places in the multiprocessor module, and it seems to be perfect for counting, except as required:

  if __name__ == '__main__' 

From what I understand, does this mean that you can only execute multiprocessor code with a script? those. if I do something like:

  import multiprocessing from numpy.random import randn a = randn(100,100) b = randn(100,100) c = randn(100,100) d = randn(100,100) def process_matrix(mat): return mat^2 if __name__=='__main__': print "Multiprocessing" jobs=[] for input_matrix in [a,b,c,d]: p = multiprocessing.Process(target=process_matrix,args=(input_matrix,)) jobs.append(p) p.start() 

It works fine, however, believing that I saved the above as "matrix_multiproc.py" and defined a new file, "importing_test.py", which simply indicates:

  import matrix_multiproc 

Multiprocessing does not occur, because now the name is "matrix_multiproc" and not " main "

Does this mean that I can never use parallel processing of an imported module? All I'm trying to do is run my model like:

  def Model_Run(): import Part1, Part2, Part3, matrix_multiproc, Part4 Part1.Run() Part2.Run() Part3.Run() matrix_multiproc.Run() Part4.Run() 

Sorry for the very long question, which is probably the simple answer, thanks!

Answer 1

Does this mean that I can never use parallel processing of an imported module?

No, it is not. You can use multiprocessing anywhere in your code, provided that the protection if __name__ == '__main__' used in the main module of the program.

On Unix systems, you don’t even need this protector, since it has a fork() system call to create child processes from the main python process.

On Windows, on the other hand, fork() emulates multiprocessing , creating a new process that starts the main module again using another __name__ . Without protection, here the main application will try to create new processes again, which will lead to an endless cycle and very quickly eats up the entire computer memory.