How to multiplex multiple Python blocking generators into one?

Question

Consider the following pseudo code:

def g_user(): while True: yield read_user_input() def g_socket(): while True: yield read_socket_input() def g_combined(gu, gs): # should read user or socket input, whichever is available while True: sel = select(gu, gs) if sel.contains(gu): yield gu.next() if sel.contains(gs): yield gs.next() gc = g_combined ( g_user(), g_socket() ) 

How to implement this in the easiest way?

Answer 1

See how someone has already implemented this: http://www.dabeaz.com/generators/genmulti.py

The mirror is here:

 import Queue, threading def gen_multiplex(genlist): item_q = Queue.Queue() def run_one(source): for item in source: item_q.put(item) def run_all(): thrlist = [] for source in genlist: t = threading.Thread(target=run_one,args=(source,)) t.start() thrlist.append(t) for t in thrlist: t.join() item_q.put(StopIteration) threading.Thread(target=run_all).start() while True: item = item_q.get() if item is StopIteration: return yield item 

Answer 2

To make select.select() workable, you will need to set the fileno() method on your generators, which will return the base file descriptor.

Answer 3

The itertools module from the standard library contains nice features. In this particular case, the loop function is good.

help loop says:

loop (iterable) → loop object

Returns items from an iterable until it is exhausted. then repeat the sequence indefinitely.

Using a loop in your solution:

 import itertools def g_user(): while True: yield read_user_input() def g_socket(): while True: yield read_socket_input() def g_combine(*generators): for generator in itertools.cycle(generators): yield generator.next() g_combined = g_combine(g_user(), g_socket()) 

Considerations for this code:

• g_combine ends with the first generator that calls StopIteration

• g_combine accepts N generators

• How to correct exceptions? Think about it.