Does Python support multiprocessing.Pool remote subprocesses?

Question

I recently discovered how awesome it is Pool.map()for simple multiprocessing tasks, but my local machine has only two processors, and I was hoping to take advantage of some remote computing resources.

Is it possible to create a processing pool for use on remote machines as well as on local machines in a transparent way?

What python code will I need to run on remote computers? How to configure a pool on a local (main) machine?

On the documentation page for multiprocessing , It says that managers should be used for shared objects, but I don’t understand how they should be used for a remote pool. Nothing other than Managers seems to refer to ports and addresses, so I assume that all remote actions occur through this class.

Answer 1

multiprocessing.Pooldoesn't seem to be intended for distributed processing - if this is what you want to do, I suggest you check the execnetpackage instead. This works on SSH channels and sends code to a remote Python interpreter if necessary.