Geek Answers Handbook

Python What is the difference between a workflow pool and just running multiple processes?

I am not sure when to use the worker pool against multiple processes.

processes = [] for m in range(1,5): p = Process(target=some_function) p.start() processes.append(p) for p in processes: p.join()

against

 if __name__ == '__main__': # start 4 worker processes with Pool(processes=4) as pool: pool_outputs = pool.map(another_function, inputs)
+7
python process multiprocessing pool worker
source share
1 answer

As stated on PYMOTW :

The pool class can be used to manage a fixed number of employees for simple cases when the work performed can be divided and distributed between employees independently.

Return values ​​from jobs are collected and returned as a list.

The pool arguments include the number of processes and the function is launched when the task process starts (called once for each child).

Please see the examples here to better understand its application, functionality, and parameters.

In principle, a pool is an assistant that facilitates the management of processes (workers) in cases where all they need to do is use common input data, process it in parallel and create a joint output.

There are a lot of things in the pool that you must otherwise code yourself (not too difficult, but still convenient to find a pre-prepared solution)

i.e.

  • input splitting
  • The function of the target process is simplified: it can be designed to expect only one input element. The pool will call this, providing each item from the subset allocated to this worker.
  • the expectation that employees will finish their work (i.e., join the processes)
  • ...
  • combining the result of each worker to create the final result
+4
source share

More articles:


All Articles