Import custom modules into IPython.parallel using sync_imports ()

Question

I played with IPython.parallel and I wanted to use some of my own native modules, but could not do this, as explained on the cookbook , using dview.sync_imports() . The only thing that worked for me was something like

 def my_parallel_func(args): import sys sys.path.append('/path/to/my/module') import my_module #and all the rest 

and then basically just

 if __name__=='__main__': #set up dview... dview.map( my_parallel_func, my_args ) 

The right way to do this, in my opinion, would be something like

  with dview.sync_imports(): import sys sys.path.append('/path/to/my/module') import my_module 

but this causes an error saying that there is no module named my_module .

So what is the correct way to do this using dview.sync_imports() ??

Answer 1

The problem is that you change PYTHONPATH only in the local process running the Client, and not in the remote processes running in ipcluster .

This behavior can be observed if you execute the following code:

 from IPython.parallel import Client rc = Client() dview = rc[:] with dview.sync_imports(): import sys sys.path[:] = ['something'] def parallel(x): import sys return sys.path print 'Local: ', sys.path print 'Remote: ', dview.map_sync(parallel, range(1)) 

Basically, all the modules you want to use with sync_imports should already be in PYTHONPATH .

If it is not in PYTHONPATH , you must add it to the path in the function that you are executing remotely, and then import the module into the function.