How to avoid unintentional capture of a local area in functional literals?

Question

I will ask about this with an example of Scala, but it is possible that this affects other languages ​​that allow hybrid imperative and functional styles.

Here is a short example ( UPDATED , see below):

def method: Iterator[Int] { // construct some large intermediate value val huge = (1 to 1000000).toList val small = List.fill(5)(scala.util.Random.nextInt) // accidentally use huge in a literal small.iterator filterNot ( huge contains _ ) } 

Now iterator.filterNot works lazily and it's great! As a result, we expect the returned iterator to not consume much memory (indeed, O (1)). Unfortunately, we made a terrible mistake: since filterNot lazy, it refers to the literal huge contains _ function.

Thus, although we thought that this method would require a large amount of memory during its operation and that this memory could be freed immediately after the method was completed, in fact the memory was stuck until we forget the returned Iterator .

(I just made a mistake that took a long time to track down! You can catch such things by looking at the heaps of the dump ...)

What are the best methods to prevent this problem?

It seems that the only solution is to thoroughly test functional literals that go beyond the scope and that capture intermediate variables. This is a little inconvenient if you create a lax fee and plan to return it. Can anyone think of some good tricks, Scala-specific or others, that avoid this problem and allow me to write good code?

UPDATE: the example I gave earlier was dumb, as the huynhjl answer below shows. It was:

 def method: Iterator[Int] { val huge = (1 to 1000000).toList // construct some large intermediate value val n = huge.last // do some calculation based on it (1 to n).iterator map (_ + 1) // return some small value } 

In fact, now that I understand a little better how it works, I'm not so worried!

Answer 1

Are you sure you are not simplifying the test? Here is what I run:

 object Clos { def method: Iterator[Int] = { val huge = (1 to 2000000).toList val n = huge.last (1 to n).iterator map (_ + 1) } def gc() { println("GC!!"); Runtime.getRuntime.gc } def main(args:Array[String]) { val list = List(method, method, method) list.foreach(m => println(m.next)) gc() list.foreach(m => println(m.next)) list.foreach(m => println(m.next)) } } 

If you understood correctly, because main uses iterators even after calling gc() , the JVM will hold huge objects.

This is how I run it:

 JAVA_OPTS="-verbose:gc" scala -cp classes Clos 

This is what it prints towards the end:

 [Full GC 57077K->57077K(60916K), 0.3340941 secs] [Full GC 60852K->60851K(65088K), 0.3653304 secs] 2 2 2 GC!! [Full GC 62959K->247K(65088K), 0.0610994 secs] 3 3 3 4 4 4 

So, it seems to me that the huge objects have been fixed ...