I would like to use multiprocessing in Python to handle the high computational cost function ktimes and collect the returned results in a list. Let me show my function for example
def _heavy_func(value):
a, b = 0, 1
for item in range(value):
a, b = b, a + b
import time
time.sleep(1.3)
return a
Then I call _heavy_functo show the non-parallel way
In [1]: print [ _square_and_offset(i) for i in range(12)]
His performance
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
IPython CPU timings (estimated):
User : 13.01 s.
System : 0.00 s.
Wall time: 13.01 s.
Now I have placed my function oriented towards the parallelism decorator, for example
from functools import wraps
from multiprocessing import Pool
def parallel_decor(n_procs=None):
def _parallel_decor(function):
@wraps(function)
def __parallel_decor(iterable_values, *args, **kwargs):
args = list(args)
p = Pool(n_procs)
result = [p.apply_async(function, args=[value]+args, kwds=kwargs) for value in iterable_values]
p.close()
try:
return [r.get() for r in result]
except KeyError:
return []
return __parallel_decor
return _parallel_decor
@parallel_decor(5) # 5 processes
def _heavy_func(value):
a, b = 0, 1
for item in range(value):
a, b = b, a + b
import time
time.sleep(1.3)
return a
def main():
_heavy_func(range(12))
if __name__ == "__main__":
main()
Sorry, I have an error
PicklingError Traceback (most recent call last)
C:\Users\Laptop\Desktop\sp_parallel.py in <module>()
30
31 if __name__ == "__main__":
C:\Users\Laptop\Desktop\sp_parallel.py in main()
27
28 def main():
30
31 if __name__ == "__main__":
C:\Users\Laptop\Desktop\sp_parallel.py in __parallel_decor(iterable_values, *args, **kwargs)
11 p.close()
12 try:
14 except KeyError:
15 return []
C:\Anaconda\lib\multiprocessing\pool.pyc in get(self, timeout)
565 return self._value
566 else:
568
569 def _set(self, i, obj):
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
According to PicklingError in multiprocessing , the following types can be pickled - these are functions defined at the top level of the module. My question is, is there any chance of changing my approach to have a pickled c function in the decorator?
Thanks in advance