I am trying to complete a job using the ipyparallel system with lru_cache and lru_cache confusing the problems.
From terminal:
ipcluster start -n 2
In ipython laptop:
from ipyparallel import Client clients = Client() def world(): return hello() + " World" def hello(): return "Hello" world() 'Hello World'
Running using ipyparallel requires this:
clients[:].push(dict(hello=hello))
Without the previous line, a failure occurs that is not unexpected, but works fine if it is running:
clients[:].apply_sync(world) ['Hello World', 'Hello World']
Everything works as expected, however with lru_cache parallel step generates errors
from ipyparallel import Client from functools import lru_cache clients = Client() def world(): return hello() + " World" @lru_cache(maxsize=2048) def hello(): return "Hello" clients[:].push(dict(hello=hello)) clients[:].apply_sync(world)
This fails with the following error:
[0:apply]: ---------------------------------------------------------------------------NameError Traceback (most recent call last)<string> in <module>() <ipython-input-17-9ac5ef032485> in world() NameError: name 'hello' is not defined [1:apply]: ---------------------------------------------------------------------------NameError Traceback (most recent call last)<string> in <module>() <ipython-input-17-9ac5ef032485> in world() NameError: name 'hello' is not defined
I understand that this could be a namespace problem where lru_cache uses closure, however I hope there is a workaround so that I can use it. Or, if someone tells me that this is simply not possible, that would also be useful, thanks.
I use:
ipykernel (4.1.1)
ipyparallel (4.1.0)
ipython (4.0.1)
Python (3.5.1)