Why understand a list in a loop variable, but generators do not?

Question

If I do something with a list, it writes a local variable:

i = 0 test = any([i == 2 for i in xrange(10)]) print i 

Prints "9". However, if I use a generator, it does not write a local variable:

 i = 0 test = any(i == 2 for i in xrange(10)) print i 

Prints "0".

Is there a good reason for this difference? Is this a constructive solution or just a random by-product of how generators and lists are implemented? Personally, it would have seemed better to me if local variables were not written in the contexts of the lists.

Answer 1

The creator of Pythons, Guido van Rossum, mentions this when he wrote about which were evenly built into Python 3: (my attention)

We also made another change in Python 3 to improve the equivalence between list views and generator expressions. In Python 2, understanding the list of "leaks" of the loop control variable to the surrounding area:

 x = 'before' a = [x for x in 1, 2, 3] print x # this prints '3', not 'before' 

It was an artifact of the initial implementation of lists; it has been one of Python's “dirty little secrets” for many years. It began as a deliberate compromise to make blind perception of lists quick, and although this was not a usual trap for beginners, it definitely stung people occasionally. For generator expressions, we could not do this. Generator expressions are implemented using generators, the execution of which requires a separate execution frame. Thus, generator expressions (especially if they are repeated in a short sequence) were less effective than list comprehensions.

However, in Python 3, we decided to fix the "dirty little secret" from the concepts of a list using the same implementation strategy as for generator expressions. Thus, in Python 3, the above example (after modifying the use of print (x) :-) will print "before", proving that "x" in the list comprehension temporarily obscures, but does not override "x" in the surrounding sphere.

So in Python 3 you will no longer see this.

Interestingly, errors in Python 2 also do not; this is mainly because the dict errors were passed from Python 3 and as such already had this fix.

There are other questions that also cover this topic, but I'm sure you already saw them when you searched for this topic, right?;)

• Renaming lists in Python even after understanding. It is right?
• Why is the list comprehension variable available after the operation is completed?

Answer 2

As PEP 289 (Generator Expressions) explains:

A loop variable (if it is a simple variable or a tuple of simple variables) is not exposed to the surrounding function. This makes implementation easier and makes typical use cases more reliable.

It seems that this was done for implementation reasons.

Personally, it would have seemed better to me if local variables were not written in the contexts of the lists.

PEP 289 also clarifies this:

Matching lists also "leaks" their variable loop into the surrounding area. This will also change in Python 3.0, so that the semantic definition of list comprehension in Python 3.0 will be equivalent to list ().

In other words, the behavior you describe is really different in Python 2, but it is fixed in Python 3.

Answer 3

Personally, it would have seemed better to me if local variables were not written in the contexts of the lists.

You're right. This is fixed in Python 3.x. The behavior does not change in 2.x, so that it does not affect the existing code that (ab) uses this hole.

Answer 4

Because because.

No, really, that. Quirk implementation. And maybe a bug, as it is fixed in Python 3.

Answer 5

One of the subtle consequences of the dirty secret described above is that list(...) and [...] do not have the same side effects in Python 2:

 In [1]: a = 'Before' In [2]: list(a for a in range(5)) In [3]: a Out[3]: 'Before' 

So there is no side effect for expressing the generator inside the list-constructor, but the side effect is in the direct sense of the list:

 In [4]: [a for a in range(5)] In [5]: a Out[5]: 4 

Answer 6

As a byproduct of wandering about how list views are implemented, I found a good answer to your question.

In Python 2, take a look at the bytecode generated for a simple list comprehension:

 >>> s = compile('[i for i in [1, 2, 3]]', '', 'exec') >>> dis(s) 1 0 BUILD_LIST 0 3 LOAD_CONST 0 (1) 6 LOAD_CONST 1 (2) 9 LOAD_CONST 2 (3) 12 BUILD_LIST 3 15 GET_ITER >> 16 FOR_ITER 12 (to 31) 19 STORE_NAME 0 (i) 22 LOAD_NAME 0 (i) 25 LIST_APPEND 2 28 JUMP_ABSOLUTE 16 >> 31 POP_TOP 32 LOAD_CONST 3 (None) 35 RETURN_VALUE 

it essentially translates into a simple for-loop that uses syntactic sugar for it. As a result, the same semantics are used as for for-loops :

 a = [] for i in [1, 2, 3] a.append(i) print(i) # 3 leaky 

In the case of list comprehension (C), Python uses the "hidden list name" and the special LIST_APPEND command to handle the creation, but actually does nothing.

So, your question should generalize to why Python writes a for loop variable to for-loop s; it's nice to reply to a blog post from Eli Benderski .

Python 3, as already mentioned, and others, has changed the semantics of list comprehension to better fit the structure of generators (by creating a separate code object for understanding) and is essentially syntactic sugar for the following:

 a = [i for i in [1, 2, 3]] # equivalent to def __f(it): _ = [] for i in it _.append(i) return _ a = __f([1, 2, 3]) 

this will not leak because it does not start in the very top area, as the Python 2 equivalent does. i only filters in __f and then is destroyed as a local variable for this function.

If you want, look at the bytecode generated for Python 3, at dis('a = [i for i in [1, 2, 3]]') . You will see how the "hidden" code object is loaded, and then the function call ends.