Python Multiprocessing> = list 125 never ends

Question

I am trying to implement this multiprocessor tutorial for my own purposes. At first I thought it did not scale well, but when I made a reproducible example, I found that if the list of elements exceeds 124, it seems to never return an answer. At x = 124 it lasts .4 seconds, but when I set it to x = 125 , it never ends. I am running Python 2.7 on Windows 7.

 from multiprocessing import Lock, Process, Queue, current_process import time class Testclass(object): def __init__(self, x): self.x = x def toyfunction(testclass): testclass.product = testclass.x * testclass.x return testclass def worker(work_queue, done_queue): try: for testclass in iter(work_queue.get, 'STOP'): print(testclass.counter) newtestclass = toyfunction(testclass) done_queue.put(newtestclass) except: print('error') return True def main(x): counter = 1 database = [] while counter <= x: database.append(Testclass(10)) counter += 1 print(counter) workers = 8 work_queue = Queue() done_queue = Queue() processes = [] start = time.clock() counter = 1 for testclass in database: testclass.counter = counter work_queue.put(testclass) counter += 1 print(counter) print('items loaded') for w in range(workers): p = Process(target=worker, args=(work_queue, done_queue)) p.start() processes.append(p) work_queue.put('STOP') for p in processes: p.join() done_queue.put('STOP') newdatabase = [] for testclass in iter(done_queue.get, 'STOP'): newdatabase.append(testclass) print(time.clock()-start) print("Done") return(newdatabase) if __name__ == '__main__': database = main(124) database2 = main(125) 

Answer 1

OK! From the docs :

A warning. As mentioned above, if a child process has queued items (and it is not used by JoinableQueue.cancel_join_thread), then this process will not end until all buffered items have been dropped into the pipe. This means that if you try to join this process, you can get stuck if you are not sure that all the items that were placed in the queue were destroyed. Similarly, if the child process is not demonic, then the parent process may freeze on exit when it tries to join all of its non-demonic children. Note that a queue created using the dispatcher does not have this problem. See the programming guide.

As I noted in a comment earlier, the code tries .join() process the queue done_queue Queue - and that after changing the code in a funky way, to be sure done_queue was merged to .join() 'ing, the code worked fine for a million elements .

So, this is a case of pilot error, although rather obscure. As for why the behavior depends on the number passed to main(x) , it is unpredictable: it depends on how the buffering inside is done. So much fun; -)