I found myself waving a list hammer

Question

... and each for loop looked like a list comprehension.

Instead:

for stuff in all_stuff: do(stuff) 

I did (without assigning a list to anything):

 [ do(stuff) for stuff in all_stuff ] 

This is a generic pattern found in the list-comp statement . one) OK, so right? Wrong. 2) Couldn't that be a code style? Super wrong.

1) Yes, it was wrong. As NiklasB points out, HowTos is creating a new list.

2) It is possible, but it is not obvious and explicit, therefore it is better not to use it.

I did not mean that these instructions were mainly based on the command line. After my team screamed at me, I wonder why the hell I collected massive lists and then released them, it occurred to me that I could introduce a serious memory error.

So here is my question / s. If I did this in a very long process, when a lot of data was consumed, would this “list” continue to use my memory until it disappears? When will the garbage collector get the memory back? After the volume that is built in this list is lost?

I think yes, he will continue to consume my memory. I don't know how the python garbage collector works, but I would venture to say that this list will exist until the last next is called on all_stuff .

EDIT.

The essence of my question is solved much cleaner in this matter (thanks for the Niklas link)

Answer 1

If I were to do this in a very long process, when a lot of data was consumed, would this “list” continue to use memory until the holidays?

That's right.

When will the garbage collector require the return of memory? After the volume that is built in this list is lost?

CPython uses reference counting, so this is the most likely case. Other implementations work differently, so don't count on it.

Thanks to Karl, pointing out that thanks to the complex memory management mechanisms used by CPython, this does not mean that memory will immediately return to the OS after that.

I don't know how the python garbage collector works, but I would venture to say that this list will exist until the last one is called on all_stuff.

I don't think such a garbage collector works like this. They are usually tag-and-sweep, so it may take quite some time before the list is garbage collected.

This is a generic pattern found in the list-comp statement.

Absolutely not. The fact is that you iterate over the list in order to do something with each element ( do side effects are caused for it). In all List-comp HOWTO examples, the list is repeated to create a new list based on the elements of the old. Consider an example:

 # list comp, creates the list [0,1,2,3,4,5,6,7,8,9] [i for i in range(10)] # loop, does nothing for i in range(10): i # meh, just an expression which doesn't have an effect 

Perhaps you will agree that this cycle is completely meaningless, since it does nothing, contrary to the understanding that builds the list. In your example, this is the opposite: Understanding is completely pointless because you don't need a list! More information on this issue can be found in the related question.

By the way, if you really want to write this loop in one line, use a consumer generator, for example deque.extend . This will be slightly slower than the original for loop in this simple example:

 >>> from collections import deque >>> consume = deque(maxlen=0).extend >>> consume(do(stuff) for stuff in all_stuff) 

Answer 2

Try to manually execute the GC and reset the statistics.

gc.DEBUG_STATS

Print statistics at the time of collection. This information may be useful in setting the frequency of collection.

FROM

http://docs.python.org/library/gc.html

Answer 3

CPython GC will reap it if there is no reference to it outside the loop. Jython and IronPython follow the rules of the underlying GC.

Answer 4

If you like this idiom, do returns something that always evaluates to True or False, and would consider a similar alternative without ugly side effects, you can use the generator expression in conjunction with any or all .

For functions that return False (or do not return):

 any(do(stuff) for stuff in all_stuff) 

For functions returning True values:

 all(do(stuff) for stuff in all_stuff) 

Answer 5

I don't know how the python garbage collector works, but I would venture to say that this list will exist until the last one is called on all_stuff.

Well, of course, this will happen as you create a list that will have the same number of all_stuff elements. An interpreter cannot drop a list before it is complete, right? You can invoke gc.collect between one of these loops and the other, but each one will be completely constructed before it can be fixed.

In some cases, you can use a generator expression instead of understanding the list, so it does not need to create a list with all your values:

 (do_something(i) for i in xrange(1000)) 

However, you still have to “convince” this generator in some way ...