Exchange data using dispatcher () in python multiprocessing module

Question

I tried to exchange data using the multiprocessing module (python 2.7, Linux), I got different results when using slightly different code:

 import os import time from multiprocessing import Process, Manager def editDict(d): d[1] = 10 d[2] = 20 d[3] = 30 pnum = 3 m = Manager() 

1st version:

 mlist = m.list() for i in xrange(pnum): mdict = m.dict() mlist.append(mdict) p = Process(target=editDict,args=(mdict,)) p.start() time.sleep(2) print 'after process finished', mlist 

This generates:

after the completion of the process [{1:10, 2: 20, 3: 30}, {1:10, 2: 20, 3: 30}, {1:10, 2: 20, 3: 30}]

Second version:

 mlist = m.list([m.dict() for i in xrange(pnum)]) # main difference to 1st version for i in xrange(pnum): p = Process(target=editDict,args=(mlist[i],)) p.start() time.sleep(2) print 'after process finished', mlist 

This generates:

upon completion of the process [{}, {}, {}]

I don’t understand why the result is so different.

Answer 1

This is because you are accessing the variable at the index of the list a second time, while the first time you are passing a valid variable. As stated in multiprocessor docs :

Modifications of the changed values ​​or elements in the dict and list proxy files will not be transmitted through the dispatcher, since the proxy server does not know when its values ​​or elements will be changed.

This means that to track items that have been changed in the container (dictionary or list), you must reassign them after each edit. Consider the following change (for explanatory purposes, I do not claim that this is pure code):

 def editDict(d, l, i): d[1] = 10 d[2] = 20 d[3] = 30 l[i] = d mlist = m.list([m.dict() for i in xrange(pnum)]) for i in xrange(pnum): p = Process(target=editDict,args=(mlist[i], mlist, i,)) p.start() 

If you type mlist now, you will see that it has the same result as your first attempt. Reassignment will allow the container proxy to track the updated item again.

Your main problem in this case is that you have a dict (proxy) inside the list proxy: updates to the contained container will not be noticed by the manager and, therefore, will not have the expected changes. Please note that the dictionary itself will be updated in the second example, but you just do not see it, because the manager did not synchronize.