Multiprocessing.Queue and Queue.Queue are different?

Question

If I use Queue.Queue , then my read() function does not work, why? But if I use multiprocessing.Queue , it works well:

 from multiprocessing import Pool, Process, Queue import os, time # from Queue import Queue def write(q): for v in ['A', 'B', 'C']: print 'Put %s to queue ' % v q.put_nowait(v) time.sleep(0.2) def read(q): while 1: if not q.empty(): v = q.get(True) print "Get %s from queue" % v time.sleep(0.2) else: break if __name__ == '__main__': q = Queue() pw = Process(target=write, args=(q, )) pr = Process(target=read, args=(q, )) pw.start() pw.join() pr.start() pr.join() print "all done..." 

Answer 1

Queue.Queue is just a queue in memory that knows how to handle multiple threads using it at the same time. It only works if both the manufacturer and the consumer are in the same process.

After you use them in separate system processes, what is the multiprocessing library, everything is a little more complicated, because the processes no longer use the same memory. You need some kind of interprocess communication method that allows the two processes to communicate with each other. It could be shared memory, a pipe or socket, or perhaps something else. This is what multiprocessing.Queue does. It uses channels to allow two processes to interact. It just implements the same API as Queue.Queue , because most Python programmers are already familiar with it.

Also note that as you use the queue, you have a race condition in your program. Think about what happens if the write process writes to the queue right after q.empty() called in the read process. Usually you add some special element to the queue (for example, None ), which means that the consumer can stop.