Python multiprocessing pool, waiting for processes and restarting user processes

Question

I used a python multiprocessor and am waiting for all processes using this code:

... results = [] for i in range(num_extract): url = queue.get(timeout=5) try: print "START PROCESS!" result = pool.apply_async(process, [host,url],callback=callback) results.append(result) except Exception,e: continue for r in results: r.get(timeout=7) ... 

I try to use pool.join but I get an error:

 Traceback (most recent call last): File "C:\workspace\sdl\lxchg\walker4.py", line 163, in <module> pool.join() File "C:\Python25\Lib\site-packages\multiprocessing\pool.py", line 338, in joi n assert self._state in (CLOSE, TERMINATE) AssertionError 

Why join does not work? And what a good way to wait for all processes.

My second question is: how can I restart a specific process in a pool? I need this because of a memory leak. Now, in fact, I am rebuilding the entire pool after all the processes have completed their tasks (create a new pool of objects to restart the process).

What I need: for example, I have 4 processes in the pool. Then the process will receive its task, after completing the task I need to kill the process and start a new one (to update the memory leak).

Answer 1

You get an error because you need to call pool.close() before calling pool.join()

I don’t know how to close the process started with apply_async , but see if closing the pool correctly will not lead to a memory leak.

The reason I think the Pool class has a bunch of attributes, which are threads running in daemon mode. All these threads are cleared by the join method. The code that you have now will not clear them, so if you create a new Pool , you will still have all the threads that will be executed from the last.