Are shared read-only data copied to different processes for multiprocessing?

Question

The piece of code that I have looks something like this:

glbl_array = # a 3 Gb array def my_func( args, def_param = glbl_array): #do stuff on args and def_param if __name__ == '__main__': pool = Pool(processes=4) pool.map(my_func, range(1000)) 

Is there a way to make sure (or encourage) that different processes do not receive a copy of glbl_array, but share it. If there is no way to stop the copy, I will go with the memmapped array, but my access patterns are not very regular, so I expect memmapped arrays to be slower. Above was like the first thing to try. This is on Linux. I just wanted advice from Stackoverflow and I don't want to annoy the system administrator. Do you think this will help if the second parameter is a genuine immutable object, for example glbl_array.tostring() .

Answer 1

You can use shared memory material from multiprocessing with Numpy quite easily:

 import multiprocessing import ctypes import numpy as np shared_array_base = multiprocessing.Array(ctypes.c_double, 10*10) shared_array = np.ctypeslib.as_array(shared_array_base.get_obj()) shared_array = shared_array.reshape(10, 10) #-- edited 2015-05-01: the assert check below checks the wrong thing # with recent versions of Numpy/multiprocessing. That no copy is made # is indicated by the fact that the program prints the output shown below. ## No copy was made ##assert shared_array.base.base is shared_array_base.get_obj() # Parallel processing def my_func(i, def_param=shared_array): shared_array[i,:] = i if __name__ == '__main__': pool = multiprocessing.Pool(processes=4) pool.map(my_func, range(10)) print shared_array 

which prints

 [[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] [ 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.] [ 2. 2. 2. 2. 2. 2. 2. 2. 2. 2.] [ 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.] [ 4. 4. 4. 4. 4. 4. 4. 4. 4. 4.] [ 5. 5. 5. 5. 5. 5. 5. 5. 5. 5.] [ 6. 6. 6. 6. 6. 6. 6. 6. 6. 6.] [ 7. 7. 7. 7. 7. 7. 7. 7. 7. 7.] [ 8. 8. 8. 8. 8. 8. 8. 8. 8. 8.] [ 9. 9. 9. 9. 9. 9. 9. 9. 9. 9.]] 

However, Linux has copy-on-write semantics on fork() , so even without using multiprocessing.Array data will not be copied if it is not written.

Answer 2

The following code works on Win7 and Mac (possibly on Linux, but not tested).

 import multiprocessing import ctypes import numpy as np #-- edited 2015-05-01: the assert check below checks the wrong thing # with recent versions of Numpy/multiprocessing. That no copy is made # is indicated by the fact that the program prints the output shown below. ## No copy was made ##assert shared_array.base.base is shared_array_base.get_obj() shared_array = None def init(shared_array_base): global shared_array shared_array = np.ctypeslib.as_array(shared_array_base.get_obj()) shared_array = shared_array.reshape(10, 10) # Parallel processing def my_func(i): shared_array[i, :] = i if __name__ == '__main__': shared_array_base = multiprocessing.Array(ctypes.c_double, 10*10) pool = multiprocessing.Pool(processes=4, initializer=init, initargs=(shared_array_base,)) pool.map(my_func, range(10)) shared_array = np.ctypeslib.as_array(shared_array_base.get_obj()) shared_array = shared_array.reshape(10, 10) print shared_array 

Answer 3

For those stuck on Windows that does not support fork() (unless CygWin is used), the pv answer does not work. Globals are not available for child processes.

Instead, you should pass shared memory during the Pool / Process initializer as such:

 #! /usr/bin/python import time from multiprocessing import Process, Queue, Array def f(q,a): m = q.get() print m print a[0], a[1], a[2] m = q.get() print m print a[0], a[1], a[2] if __name__ == '__main__': a = Array('B', (1, 2, 3), lock=False) q = Queue() p = Process(target=f, args=(q,a)) p.start() q.put([1, 2, 3]) time.sleep(1) a[0:3] = (4, 5, 6) q.put([4, 5, 6]) p.join() 

(this is not numpy and this is not good code, but it illustrates the point ;-)

Answer 4

If you are looking for an option that works efficiently on Windows and works well for irregular access patterns, branching, and other scenarios, where you may need to analyze different matrices based on a combination of a matrix with shared memory and a process, local data, the mathDict toolkit in the ParallelRegression package was Designed to handle this exact situation.