Why can't I use operator.itemgetter in multiprocessing.Pool?

Question

The following program:

import multiprocessing,operator f = operator.itemgetter(0) # def f(*a): return operator.itemgetter(0)(*a) if __name__ == '__main__': multiprocessing.Pool(1).map(f, ["ab"]) 

The following error failed:

 Process PoolWorker-1: Traceback (most recent call last): File "/usr/lib/python3.2/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/usr/lib/python3.2/multiprocessing/process.py", line 116, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.2/multiprocessing/pool.py", line 102, in worker task = get() File "/usr/lib/python3.2/multiprocessing/queues.py", line 382, in get return recv() TypeError: itemgetter expected 1 arguments, got 0 

Why am I getting an error (on cPython 2.7 and 3.2 on Linux x64) and why does it disappear if I uncomment the third line?

Answer 1

The problem is that the multiprocessing module transfers objects by copying to other processes (obviously), and itemgetter objects cannot be copied using any of the obvious methods:

 In [10]: a = operator.itemgetter(0) Out[10]: copy.copy(a) TypeError: itemgetter expected 1 arguments, got 0 In [10]: a = operator.itemgetter(0) Out[10]: copy.deepcopy(a) TypeError: itemgetter expected 1 arguments, got 0 In [10]: a = operator.itemgetter(0) Out[10]: pickle.dumps(a) TypeError: can't pickle itemgetter objects # etc. 

The problem is not even to call f inside other processes; he is trying to copy it first. (If you look at the stack traces I skipped above, you will see a lot more information on why this fails.)

Of course, this usually doesn't matter, because it's almost as easy and simple to build a new control on the fly to copy it. And this is what your alternate f function does. (Copying a function that creates the control on the fly does not, of course, require copying an instance of the element.)

You can turn the "f" into a lambda. Or write a trivial function (named or lambda) that does the same without using itemgetter. Or write a replacement entry to be copied (which, obviously, would not be so difficult). But you cannot directly use itemgetter objects as is the way you want.