Multiprocessing.

So, I have this program, which basically sums up arrays. I used multiprocessing because the program needs to parse 700,000 lines. Below I made a small script that illustrates the problem I am facing

import multiprocessing as mp import numpy as np import ctypes t = range(100000) list_1 = np.zeros((10)) for i in t: list_1 += np.array([1]*10) list_t = mp.Array(ctypes.c_double, 10) list_2 = np.ctypeslib.as_array(list_t.get_obj()) list_2 = list_2.reshape(10) # No copy was made assert list_2.base.base is list_t.get_obj() def proc(x): global list_2 list_2 += np.array([1]*10) if __name__ == '__main__': pool = mp.Pool(processes = 6) pool.map(proc,t) 

Exit from which:

 In [2]: list_1 Out[2]: array([ 100000., 100000., 100000., 100000., 100000., 100000., 100000., 100000., 100000., 100000.]) In [3]: list_2 Out[3]: array([ 55828., 65441., 99300., 68068., 78774., 78514., 82393., 83446., 82854., 86987.]) 

Obviously, I wanted both arrays to be similar to list_1. What's going on here?

+2
source share

All Articles