Multiprocessor pool freezes when there is an exception in any thread

Question

I am new to Python and trying to use the multiprocessing.pool program to process files, it works fine until there are no exceptions. If any thread / process gets an exception, the whole program waits for the thread

code snippet:

cp = ConfigParser.ConfigParser() cp.read(gdbini) for table in cp.sections(): jobs.append(table) #print jobs poolreturn = pool.map(worker, jobs) pool.close() pool.join() 

Error message:

---

 Traceback (most recent call last): File "/opt/cnet-python/default-2.6/lib/python2.6/threading.py", line 525, in __bootstrap_inner self.run() File "/opt/cnet-python/default-2.6/lib/python2.6/threading.py", line 477, in run self.__target(*self.__args, **self.__kwargs) File "/opt/cnet-python/default-2.6/lib/python2.6/multiprocessing/pool.py", line 259, in _handle_results task = get() TypeError: ('__init__() takes exactly 3 arguments (2 given)', <class 'ConfigParser.NoOptionError'>, ("No option 'inputfilename' in section: 'section-1'",)) 

---

I went further, added an exception handler to complete the process

 try: ifile=cp.get(table,'inputfilename') except ConfigParser.NoSectionError,ConfigParser.NoOptionError: usage("One of Parameter not found for"+ table) terminate() 

but still he is waiting, not sure what is missing.

Answer 1

In Python 3.2+, this works as expected. For Python 2, this bug was fixed in r74545 and will be available in Python 2.7.3. At the same time, you can use the configparser library, which is the backport of the configparser file from 3.2+. Check this.

Answer 2

I had the same problem. This happens when a workflow throws a user exception that has its own constructor. Make sure that your exception (ConfigParser.NoOptionError in this case) throws a basic exception with exactly two arguments:

 class NoOptionError(ValueError): def __init__(self, message, *args): super(NoOptionError, self).__init__(message, args)