Multiprocessing python apply_async never returns result in Windows 7

Question

I am trying to execute a very simple example of multiprocessing:

import multiprocessing as mp def cube(x): return x**3 pool = mp.Pool(processes=2) results = [pool.apply_async(cube, args=x) for x in range(1,7)] 

However, on my Windows machine, I cannot get the result (on ubuntu 12.04LTS it works fine).

If I check the results , I see the following:

 [<multiprocessing.pool.ApplyResult object at 0x01FF0910>, <multiprocessing.pool.ApplyResult object at 0x01FF0950>, <multiprocessing.pool.ApplyResult object at 0x01FF0990>, <multiprocessing.pool.ApplyResult object at 0x01FF09D0>, <multiprocessing.pool.ApplyResult object at 0x01FF0A10>, <multiprocessing.pool.ApplyResult object at 0x01FF0A50>] 

If I run results[0].ready() , I always get False .

If I run results[0].get() , the python interpreter freezes, waiting for a result that never appears.

The example is as simple as it is, so I think it is a low-level error related to the OS (I am on Windows 7). But maybe someone has a better idea?

Answer 1

There are a few errors here. First, you must declare Pool inside if __name__ == "__main__": guard when running on Windows . Secondly, you must pass the args keyword argument to the sequence, even if you pass only one argument. So, put this together:

 import multiprocessing as mp def cube(x): return x**3 if __name__ == "__main__": pool = mp.Pool(processes=2) results = [pool.apply_async(cube, args=(x,)) for x in range(1,7)] print([result.get() for result in results]) 

Output:

 [1, 8, 27, 64, 125, 216] 

Edit:

Oh, as moarningsun mentions, multiprocessing does not work well in the interactive interpreter:

Note

The functionality in this package requires the __main__ module to be imported by children. This is described in the Programming Guide, however it is worth noting here. This means that some examples, such as multiprocessing.Pool examples, will not work in the interactive interpreter.

So, you really need to execute the code as a script in order to check it correctly.

Answer 2

I ran python 3 and the IDE was spyder in anaconda (windows) and so this trick does not work for me. I tried a lot, but it didn't matter. I got the cause of my problem and the same as given in the note. But after a long day of searching, I had some kind of solution, and this helped me to run the same code on my Windows computer. This website helped me find a solution:

http://python.6.x6.nabble.com/Multiprocessing-Pool-woes-td5047050.html

Since I used python 3, I changed the program a bit:

 from types import FunctionType import marshal def _applicable(*args, **kwargs): name = kwargs['__pw_name'] code = marshal.loads(kwargs['__pw_code']) gbls = globals() #gbls = marshal.loads(kwargs['__pw_gbls']) defs = marshal.loads(kwargs['__pw_defs']) clsr = marshal.loads(kwargs['__pw_clsr']) fdct = marshal.loads(kwargs['__pw_fdct']) func = FunctionType(code, gbls, name, defs, clsr) func.fdct = fdct del kwargs['__pw_name'] del kwargs['__pw_code'] del kwargs['__pw_defs'] del kwargs['__pw_clsr'] del kwargs['__pw_fdct'] return func(*args, **kwargs) def make_applicable(f, *args, **kwargs): if not isinstance(f, FunctionType): raise ValueError('argument must be a function') kwargs['__pw_name'] = f.__name__ # edited kwargs['__pw_code'] = marshal.dumps(f.__code__) # edited kwargs['__pw_defs'] = marshal.dumps(f.__defaults__) # edited kwargs['__pw_clsr'] = marshal.dumps(f.__closure__) # edited kwargs['__pw_fdct'] = marshal.dumps(f.__dict__) # edited return _applicable, args, kwargs def _mappable(x): x,name,code,defs,clsr,fdct = x code = marshal.loads(code) gbls = globals() #gbls = marshal.loads(gbls) defs = marshal.loads(defs) clsr = marshal.loads(clsr) fdct = marshal.loads(fdct) func = FunctionType(code, gbls, name, defs, clsr) func.fdct = fdct return func(x) def make_mappable(f, iterable): if not isinstance(f, FunctionType): raise ValueError('argument must be a function') name = f.__name__ # edited code = marshal.dumps(f.__code__) # edited defs = marshal.dumps(f.__defaults__) # edited clsr = marshal.dumps(f.__closure__) # edited fdct = marshal.dumps(f.__dict__) # edited return _mappable, ((i,name,code,defs,clsr,fdct) for i in iterable) 

After this function, the above problem code also changed a bit:

 from multiprocessing import Pool from poolable import make_applicable, make_mappable def cube(x): return x**3 if __name__ == "__main__": pool = Pool(processes=2) results = [pool.apply_async(*make_applicable(cube,x)) for x in range(1,7)] print([result.get(timeout=10) for result in results]) 

And I got the output as:

 [1, 8, 27, 64, 125, 216] 

I think this post may be useful for some Windows users.