Parallel decorator with Apply_async ()

I would like to use multiprocessing in Python to handle the high computational cost function ktimes and collect the returned results in a list. Let me show my function for example

def _heavy_func(value):
        a, b = 0, 1
        for item in range(value):
                a, b = b, a + b
        import time
        time.sleep(1.3)
        return a

Then I call _heavy_functo show the non-parallel way

In [1]: print [ _square_and_offset(i) for i in range(12)]

His performance

[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
IPython CPU timings (estimated):
  User   :      13.01 s.
  System :       0.00 s.
Wall time:      13.01 s.

Now I have placed my function oriented towards the parallelism decorator, for example

from functools import wraps
from multiprocessing import Pool

def parallel_decor(n_procs=None):
    def _parallel_decor(function):
        @wraps(function)
        def __parallel_decor(iterable_values, *args, **kwargs):
            args = list(args)
            p = Pool(n_procs)
            result = [p.apply_async(function, args=[value]+args, kwds=kwargs) for value in iterable_values]
            p.close()
            try:
                return [r.get() for r in result]
            except KeyError:
                return []
        return __parallel_decor
    return _parallel_decor

@parallel_decor(5) # 5 processes
def _heavy_func(value):
        a, b = 0, 1
        for item in range(value):
                a, b = b, a + b
        import time
        time.sleep(1.3)
        return a

def main():
    _heavy_func(range(12))

if __name__ == "__main__":
    main()

Sorry, I have an error

---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
C:\Users\Laptop\Desktop\sp_parallel.py in <module>()
     30 
     31 if __name__ == "__main__":
---> 32         main()

C:\Users\Laptop\Desktop\sp_parallel.py in main()
     27 
     28 def main():
---> 29         _heavy_func(range(12))
     30 
     31 if __name__ == "__main__":

C:\Users\Laptop\Desktop\sp_parallel.py in __parallel_decor(iterable_values, *args, **kwargs)
     11                         p.close()
     12                         try:
---> 13                                 return [r.get() for r in result]
     14                         except KeyError:
     15                                 return []

C:\Anaconda\lib\multiprocessing\pool.pyc in get(self, timeout)
    565             return self._value
    566         else:
--> 567             raise self._value
    568 
    569     def _set(self, i, obj):

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

According to PicklingError in multiprocessing , the following types can be pickled - these are functions defined at the top level of the module. My question is, is there any chance of changing my approach to have a pickled c function in the decorator?

Thanks in advance

+4

All Articles