How to catch exceptions from workers in multiprocessing

Question

I work with the multiprocessing module in Python (2.7.3) and want to debug some things that happen to my workers. However, it seems like I cannot catch any exceptions in worker threads.

Minimal example:

import multiprocessing as mp a=[1] def worker(): print a[2] def pool(): pool = mp.Pool(processes=1) pool.apply_async(worker, args = ()) pool.close() pool.join() print "Multiprocessing done!" if __name__ == '__main__': pool() 

An increase in IndexError is expected, but only my conclusion

  Multiprocessing done! 

Is there a way to show me all the exceptions that occur in worker threads without manually raising mine?

Answer 1

The error does not occur if you do not call the get method AsyncResult (return value of apply_async ):

According to AsyncResult.get documentation :

Return the result when it appears. If the timeout is not None and the result does not arrive within timeout seconds, then multiprocessing.TimeoutError. If the remote call raised an exception, then this exception will be re-thrown using get () .

 def pool(): pool = mp.Pool(processes=1) result = pool.apply_async(worker, args=()) result.get() # <------------ pool.close() pool.join() print "Multiprocessing done!" 

Answer 2

I think falsetra gave you what you need. I just wanted to expand a little more.

If it is important for you to get not only the error, but also the original context (i.e., to know that the exception occurred on line 1 of the worker ()), you can check out this good post from Ned Batchelder , which explains how to re-throw exceptions with their original context.

This does not work for mp.Pool, so just in case you need something else. This SO question covers your question using more explicit multiprocessing methods instead of mp.Pool.