Geek Answers Handbook

Multiprocessing with a pool in python: in approximately multiple instances with the same name at the same time

I have a lot of new things for multiprocessing. However, suppose we have a program as shown below. The program is working fine. Now to the question. In my opinion, we will have 4 copies SomeKindOfClasswith the same name ( a) at the same time. How is this possible? Moreover, is there a potential risk of this kind of programming?

from multiprocessing.dummy import Pool
import numpy
from theFile import someKindOfClass

n = 8 
allOutputs = numpy.zeros(n)

def work(index):   
    a = SomeKindOfClass()
    a.theSlowFunction()
    allOutputs[index] = a.output

pool = Pool(processes=4) 
pool.map(work,range(0,n))
+4
source share
4 answers

In fact, you will have 8instances SomeKindOfClass(one for each worker), but only 4 will be active at a time.

multiprocessing vs multiprocessing.dummy

, multiprocessing.dummy, threading. " python" ( ). " Python" ; "" . Python GIL, - python , , python.

from multiprocessing import Pool, , allOutputs , ( , , , , , , main()). , multiprocessing . allOutputs, . , , .

, - , .

, , , - Manager

import multiprocessing

def worker(args):
    index, array = args
    a = SomeKindOfClass()
    a.some_expensive_function()
    array[index] = a.output


def main():
    n = 8
    manager = multiprocessing.Manager()
    array = manager.list([0] * n)
    pool = multiprocessing.Pool(4)
    pool.map(worker, [(i, array) for i in range(n)])
    print array
+2

a work, . python . , id:

print(id(a))

.

+3

, , . , , . , ( , ).

, . - , (, ), .

, , (, -, -), , .

+1

, - ( ). , SomeKindOfClass - . ? Python , 4 . if __init__ == '__main__' pool.map(work,range(0,n)). .

Problems can be if it SomeKindOfClassstores the state on disk - for example, writes something to a file or reads it.

+1
source

More articles:


All Articles