Python multiprocessing with pathos

Question

I am trying to use the pathos of Python to assign computations to separate processes in order to speed it up using a multi-core processor. My code is organized as:

class: def foo(self,name): ... setattr(self,name,something) ... def boo(self): for name in list: self.foo(name) 

Since I had problems with trace with multiprocessing. Pool, I decided to try pathos. I tried, as suggested in previous topics:

 import pathos.multiprocessing 

but this led to an error: there is no modular multiprocessing that I cannot find in the latest version of pathos.

Then I tried modifying the boo method:

 def boo(self): import pathos pathos.pp_map.pp_map(self.foo,list) 

Now there is no error, but foo does not work - an instance of my class has no new attributes. Please help me, because I have no idea where to go next, after a day spent on it.

Answer 1

I am the author of pathos . I'm not sure what you want to do from your code above. However, I may perhaps shed some light. Here is the code:

 >>> from pathos.multiprocessing import ProcessingPool >>> class Bar: ... def foo(self, name): ... return len(str(name)) ... def boo(self, things): ... for thing in things: ... self.sum += self.foo(thing) ... return self.sum ... sum = 0 ... >>> b = Bar() >>> results = ProcessingPool().map(b.boo, [[12,3,456],[8,9,10],['a','b','cde']]) >>> results [6, 4, 5] >>> b.sum 0 

So what happens above is that the boo method of the Bar b instance is called where b.boo is passed to the new python process and then evaluated for each of the nested lists. You can see that the results are correct ... len ("12") + len ("3") + len ("456") is 6, etc.

However, you can also see that when browsing b.sum it is mysteriously still 0 . Why is b.sum still zero? Well, what does multiprocessing (and therefore pathos.multiprocessing ) do, COPY what you go through the map into another python process ... and then copy the instance (in parallel) and return any results called by the called method . Note that you must perform RETURN results, or print them, or register them, or send them to a file, or otherwise. They cannot return to the original instance, as you might expect, because this is not the original instance passed to other processors. Copies of the instance are created and then deleted - each of them has the sum attribute, but the original `b.sum 'is not affected.

However, pathos has plans to do something like the above work, as you might expect - where the original IS object is updated, but so far it does not work.

EDIT: If you install using pip , note that the latest released version of pathos is several years old and may not install correctly or cannot install all submodules. A new release of pathos is expected, but until then it is better to get the latest version of code from github and install from there. The trunk is mostly stable in development. I think your problem may be that not all packages were installed due to the "new" pip - the "old" pathos incompatibility in the installation. If pathos.multiprocessing missing, this is the most likely culprit.

Get pathos from github here: https://github.com/uqfoundation/pathos