Simultaneous use of two iterators

Question

Suppose I have two iterators and I want to compute

fancyoperation1(iter1), fancyoperation2(iter2)

Normally I would just use fancyoperation1(iter1), fancyoperation2(iter2). However, if these iterators are associated with the same source, possibly teed from the same iterator, I cannot do this without storing a lot of temporary data in memory. In this case, I know several options:

• I could also rewrite fancyoperation1it fancyoperation2into one function that performs both at the same time, but it can be a lot of code duplication, and I cannot understand or have the source code for any function. In addition, this will need to be done anew for each pair of operations.
• I could use streams. Perhaps synchronization can be recorded once in a helper function, and the overhead probably won't be too bad unless I need to switch flows too often.
• I could store a lot of temporary data in memory.

I do not like the disadvantages of these options. Is there a way to do what I want in a single thread without overwriting things or using large amounts of memory? I tried to do this with coroutines, but Python yieldseems not powerful enough.

(I don’t have this problem right now, but I’m wondering what to do if it ever appears.)

Answer 1

, ( ). yield ( ) . , :

def fancyoperation1(it):
    for x in it:
        ...
    cleanup()

# into something like this

def fancyoperation1():
    while True:
        try:
            x = yield
        except GeneratorExit:
            break
        ...
    cleanup()

, , - , . ( iter1, iter2 = tee(underlying_iter)):

f1, f2 = fancyoperation1(), fancyoperation2()
f1.send(None) # start coroutines
f2.send(None)

for x in underlying_iterator:
    f1.send(x)
    f2.send(x)
f1.close()
f2.close()