Python multiprocessing for parallel processes

Question

I'm sorry if this is too easy for some people, but I still don't get the python multiprocessing trick. I read
http://docs.python.org/dev/library/multiprocessing
http://pymotw.com/2/multiprocessing/basics.html and many other lessons and examples that google gives me ... many of them are also here.

Well, my situation is that I have to compute a lot of numpy matrices, and I need to store them in a single numpy matrix after that. Let's say I want to use 20 cores (or I can use 20 cores), but I was not able to successfully use the pool resource, because it supports processes until the pool "dies." So I thought about doing something like this:

from multiprocessing import Process, Queue import numpy as np def f(q,i): q.put( np.zeros( (4,4) ) ) if __name__ == '__main__': q = Queue() for i in range(30): p = Process(target=f, args=(q,)) p.start() p.join() result = q.get() while q.empty() == False: result += q.get() print result 

but then it seems that the processes do not start in parallel, but they start sequentially (please correct me if I am wrong), and I do not know if they die after they perform their calculations (so that more than 20 processes those have done their part, leaving the core free for another process). In addition, for a very large number (say, 100,000), storing all of these matrices (which can be very large) in the queue will use a lot of memory, which makes the code useless, since the idea is that each result at each iteration in the end result, for example, using the lock () methods and the receive () and release ()) methods, but if this code is not intended for parallel processing, locking is also useless ...

Hope someone can help me.

Thanks in advance!

Answer 1

You are right, they are executed sequentially in your example.

p.join() causes the current thread to block until it completes execution. You either want to join your processes separately outside the for loop (for example, by storing them in a list and then iterating over it) or use something like numpy.Pool and apply_async with a callback. It will also allow you to add it directly to your results, rather than supporting the objects around.

For example:

 def f(i): return i*np.identity(4) if __name__ == '__main__': p=Pool(5) result = np.zeros((4,4)) def adder(value): global result result += value for i in range(30): p.apply_async(f, args=(i,), callback=adder) p.close() p.join() print result 

Closing and then pooling at the end ensures the completion of the pool processes and the completion of the result object. You can also explore using Pool.imap as a solution to your problem. This particular solution would look something like this:

 if __name__ == '__main__': p=Pool(5) result = np.zeros((4,4)) im = p.imap_unordered(f, range(30), chunksize=5) for x in im: result += x print result 

This is cleaner for your specific situation, but it may not be what you end up trying to do.

Regarding saving all your various results, if I understand your question, you can simply add it to the result in the callback method (as mentioned above) or in order of time using imap / imap_unordered (which still saves the results, but you will clear him as he builds). Then it does not need to be stored longer than required to add the result.