Python Nested Generators

Question

I tried to implement the inverse function itertools.izip on Python 2.7.1. The fact is that I find a problem, and I have no explanation. Solution 1, iunzip_v1 works fine. But solution 2. iunzip_v2 does not work as expected. So far I have not found any important information about this problem and have read PEP about generators, it sounds like this should work, but it is not.

import itertools from operator import itemgetter def iunzip_v1(iterable): _tmp, iterable = itertools.tee(iterable, 2) iters = itertools.tee(iterable, len(_tmp.next())) return tuple(itertools.imap(itemgetter(i), it) for i, it in enumerate(iters)) def iunzip_v2(iterable): _tmp, iterable = itertools.tee(iterable, 2) iters = itertools.tee(iterable, len(_tmp.next())) return tuple((elem[i] for elem in it) for i, it in enumerate(iters)) 

result:

 In [17]: l Out[17]: [(0, 0, 0), (1, 2, 3), (2, 4, 6), (3, 6, 9), (4, 8, 12)] In [18]: map(list, iunzip.iunzip_v1(l)) Out[18]: [[0, 1, 2, 3, 4], [0, 2, 4, 6, 8], [0, 3, 6, 9, 12]] In [19]: map(list, iunzip.iunzip_v2(l)) Out[19]: [[0, 3, 6, 9, 12], [0, 3, 6, 9, 12], [0, 3, 6, 9, 12]] 

It seems that iunzip_v2 uses the last value, so generators do not store the value while they are created inside the first generator. Something is missing for me, and I do not know what it is.

Thanks in advance if something can clarify this situation for me.

UPDATE: I found an explanation here of PEP-289 , my first reading was on PEP-255. The solution I'm trying to implement is lazy, therefore:

  zip(*iter) or izip(*...) 

doesn't work for me because * arg extends the argument list.

Answer 1

You invent the wheel in a crazy way. izip is its own inverse:

 >>> list(izip(*izip(range(10), range(10)))) [(0, 1, 2, 3, 4, 5, 6, 7, 8, 9), (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)] 

But that will not quite answer your question, will it?

The problem with your nested generators is the problem of defining the area, which arises due to the fact that the most internal generators are not used until the most external generator is started:

 def iunzip_v2(iterable): _tmp, iterable = itertools.tee(iterable, 2) iters = itertools.tee(iterable, len(_tmp.next())) return tuple((elem[i] for elem in it) for i, it in enumerate(iters)) 

Here you create three generators, each of which uses the same variable, i . Copies of this variable are not executed. Then tuple exhausts the most external generator, creating a tuple of generators:

 >>> iunzip_v2((range(3), range(3))) (<generator object <genexpr> at 0x1004d4a50>, <generator object <genexpr> at 0x1004d4aa0>, <generator object <genexpr> at 0x1004d4af0>) 

At this point, each of these generators will execute elem[i] for each element of it . And since i now 3 for all three generators, you get the last element every time.

The reason the first version works is because itemgetter(i) is a closure with its own scope - so every time it returns a function, it generates a new scope in which the value of i does not change.

Answer 2

Okay, this is a little complicated. When you use a name of type i , the value it stands for is only viewed at runtime. In this code:

 return tuple((elem[i] for elem in it) for i, it in enumerate(iters)) 

you return the number of generators (elem[i] for elem in it) , and each of them uses the same name i . When the function returns, the loop in tuple( .. for i in .. ) completed, and i been set to this final value ( 3 in your example). After you evaluate these generators in lists, they all create the same values ​​because they use the same i .

Btw:

 unzip = lambda zipped: zip(*zipped)