Geek Answers Handbook

Variable exchange between processes

I have a bootloader function that downloads multiple files in parallel. I use multiprocessing.Pool.map_asyncto load different pieces of the same file. I would like to show the boot status bar. To do this, I need to know the already loaded bytes ( total_bytes_dl).

    pool = multiprocessing.Pool(processes)
    mapObj = pool.map_async(f, args)

    while not mapObj.ready():
        status = r"%.2f MB / %.2f MB" % (total_bytes_dl / 1024.0 / 1024.0, filesize / 1024.0 / 1024.0,)
        status = status + chr(8)*(len(status)+1)
        print status,
        time.sleep(0.5)

Is there a way to set a variable that will be used for all these processes and the main process, so each process can add the number of bytes loaded?

+6
source share
4 answers

The solution was to implement a new process and pass the overall ctypes value:

from ctypes import c_int
import dummy

shared_bytes_var = multiprocessing.Value(c_int)

def Func(...):
    ....
    pool = multiprocessing.Pool(initializer=_initProcess,initargs=(shared_bytes_var,))
    ....

def _initProcess(x):
  dummy.shared_bytes_var = x
+3
source

Use a Queue object selected as follows:

que = multiprocessing.Manager().Queue()

, que.put(bytes) , . :

downloaded = 0
while not mapObj.ready():
    for _ in range(q.qsize()):
        downloaded += q.get()
    print downloaded, r"bytes downloaded\r",
    time.sleep(0.5)

.. multiprocessing.Queue(), multiprocessing.Manager().Queue(). . .

+2

You can use a multiprocess queue object that employees could use to send status data. Your main process will need to read the status records from the queue and update the status accordingly.

0
source

More articles:


All Articles