Python answer with multiprocessing.pools.map_async not working?

Question

I cannot get my callbacks to work when using map_async (). It works when I use slightly modified code to iterate over my array, adding tasks via apply_async (). From the documentation, it seems that I should use callbacks with map_async (), but maybe this is kind of a newb error ...

from multiprocessing import Pool,TimeoutError from time import sleep servers=["s1","s2","s3","s4","s5","s6"] def f(x): print("start f(" + x + ")") sleep(5) print("end f(" + x + ")") return "did " + x def mycallback(x): print("My callback " + str(x)) def myerrorcallback(r): print("My errorcallback " + str(r)) if __name__ == '__main__': pool = Pool(processes=4) results = pool.map_async(f, servers, chunksize=1, callback=mycallback, error_callback=myerrorcallback) print(results.get(timeout=11)) 

On startup, I get:

 D:\python> f.py start f(s1) start f(s2) start f(s3) start f(s4) end f(s1) start f(s5) end f(s2) start f(s6) end f(s4) end f(s3) end f(s5) end f(s6) ['did s1', 'did s2', 'did s3', 'did s4', 'did s5', 'did s6'] 

When I use the modified code using apply_async (), I get output from callbacks instead. Changed code - just change the last part to:

 if __name__ == '__main__': pool = Pool(processes=4) for server in servers: pool.apply_async(f, (server,), callback=mycallback, error_callback=myerrorcallback) pool.close() pool.join() 

Results in:

 D:\python\>fb.py start f(s1) start f(s2) start f(s3) start f(s4) end f(s1) start f(s5) My callback did s1 end f(s2) My callback did s2 start f(s6) end f(s3) My callback did s3 end f(s4) My callback did s4 end f(s5) My callback did s5 end f(s6) My callback did s6 

Answer 1

Well, I took the opportunity and filed a bug. It turns out that this is actually a bug in version 3.3 and the patch is on.

http://bugs.python.org/issue16307

Answer 2

In python 3.4.3 callback is called with cumulative results, but the return value of the callback is ignored. What is the point?

 from multiprocessing import Pool data = [1, 2, 3] def calc(x): return 2*x def increment(results): print('callback called') return [x+100 for x in results] with Pool(None) as pool: #None => calls cpu.count() results = pool.map_async(calc, data, callback=increment) print(results.get()) --output:-- callback called [2, 4, 6]