Geek Answers Handbook

Python Multiprocessor Locks

This multiprocessor code works as expected. It creates 4 Python processes and uses them to print numbers from 0 to 39, with a delay after each print. A.

import multiprocessing import time def job(num): print num time.sleep(1) pool = multiprocessing.Pool(4) lst = range(40) for i in lst: pool.apply_async(job, [i]) pool.close() pool.join()

However, when I try to use a multiprocessor package. Lock to prevent multiple processes from printing to standard, the program immediately exits without output.

 import multiprocessing import time def job(lock, num): lock.acquire() print num lock.release() time.sleep(1) pool = multiprocessing.Pool(4) l = multiprocessing.Lock() lst = range(40) for i in lst: pool.apply_async(job, [l, i]) pool.close() pool.join()

Why the introduction of a multiprocessor system. Does Lock make this code inoperative?

Update: it works when the lock is declared globally (where I did some fuzzy tests to check if the lock works), unlike the code above, which passes the lock as an argument (in the documentation on Python multiprocessing, the locks are passed as argument). The code below has a ban declared globally, as opposed to passing as an argument in the code above.

 import multiprocessing import time l = multiprocessing.Lock() def job(num): l.acquire() print num l.release() time.sleep(1) pool = multiprocessing.Pool(4) lst = range(40) for i in lst: pool.apply_async(job, [i]) pool.close() pool.join()
+7
python multiprocessing
source share
2 answers

If you change pool.apply_async to pool.apply , you will get this exception:

 Traceback (most recent call last): File "p.py", line 15, in <module> pool.apply(job, [l, i]) File "/usr/lib/python2.7/multiprocessing/pool.py", line 244, in apply return self.apply_async(func, args, kwds).get() File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get raise self._value RuntimeError: Lock objects should only be shared between processes through inheritance

pool.apply_async just hides it. I don't like to say this, but using a global variable is probably the easiest way for your example. Hope Velociraptors Don't Get You.

+6
source share

I think the reason is that the multiprocessor pool uses pickle to transfer objects between processes. However, a Lock cannot be pickled:

 >>> import multiprocessing >>> import pickle >>> lock = multiprocessing.Lock() >>> lp = pickle.dumps(lock) Traceback (most recent call last): File "<pyshell#3>", line 1, in <module> lp = pickle.dumps(lock) ... RuntimeError: Lock objects should only be shared between processes through inheritance >>>

See the sections "Picklability" and "Better to inherit than pickle / unpickle" https://docs.python.org/2/library/multiprocessing.html#all-platforms

+3
source share

More articles:


All Articles