Geek Answers Handbook

How to get around python multiprocessing etching error without being at the top level?

I investigated this question several times, but did not find a workaround that either works in my case or the one that I understand, so please bear with me.

In principle, I have a hierarchical organization of functions, and this prevents me from multiprocessing at the top level. Unfortunately, I do not believe that I can change the layout of the program, because I need all the variables that I create after the initial inputs.

For example, let's say I have this:

import multiprocessing

  def calculate(x):
    # here is where I would take this input x (and maybe a couple more inputs)
    # and build a larger library of variables that I use further down the line

    def domath(y):
      return x * y

    pool = multiprocessing.Pool(3)
    final= pool.map(domath, range(3))

calculate(2)

This results in the following error:

Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

I thought about globals, but I'm afraid that I will have to define too much, and this may slow down my program. Is there a workaround without rebuilding the entire program?

+4
3

pathos.multiprocessing, fork multiprocessing, dill pickle. dill python. .

>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> 
>>> def calculate(x):
...   def domath(y):
...     return x*y
...   return Pool().map(domath, range(3))
... 
>>> calculate(2)
[0, 2, 4]

... . , multiprocessing.

>>> class Foo(object):
...   def __init__(self, x):
...     self.x = x
...   def doit(self, y):
...     return ProcessingPool().map(self.squared, calculate(y+self.x))
...   def squared(self, z):
...     return z*z
... 
>>> def thing(obj, y):
...   return getattr(obj, 'doit')(y)
... 
>>> ProcessingPool().map(thing, ProcessingPool().map(Foo, range(3)), range(3))
[[0, 0, 0], [0, 4, 16], [0, 16, 64]]

pathos : https://github.com/uqfoundation

+2

, , . , . , , .

.

import marshal
import pickle
import types

, , , , :

def pack(fn):
    code = marshal.dumps(fn.__code__)
    name = pickle.dumps(fn.__name__)
    defs = pickle.dumps(fn.__defaults__)
    clos = pickle.dumps(fn.__closure__)
    return (code, name, defs, clos)

, . . , , :

def unpack(code, name, defs, clos):
    code = marshal.loads(code)
    glob = globals()
    name = pickle.loads(name)
    defs = pickle.loads(defs)
    clos = pickle.loads(clos)
    return types.FunctionType(code, glob, name, defs, clos)

. , . :

def test_function(a, b):
    from random import randint
    return randint(a, b)

, , , :

packed = pack(test_function)
print((packed))

, , , :

unpacked = unpack(*packed)
print((unpacked(2, 20)))

, .

+2

?

( , ):

$ cat /tmp/tmp.py
import multiprocessing

def calculate(x):
    # here is where I would take this input x (and maybe a couple more inputs)
    # and build a larger library of variables that I use further down the line

    pool = multiprocessing.Pool(3)
    _lst = [(x, y) for x in (x,) for y in range(3)]
    final= pool.map(domath, _lst)
    print(final)

def domath(l):
    return l[0] * l[1]

calculate(2)

$ python /tmp/tmp.py
[0, 2, 4]

$
0

More articles:


All Articles