What will be the behavior of the line-seq?

Question

I would like to understand the behavior of a lazy sequence if I go with doseq but keep a part of the first element.

  (with-open [log-file-reader (clojure.java.io/reader (clojure.java.io/file input-file-path))] ; Parse line parse-line returns some kind of representation of the line. (let [parsed-lines (map parse-line (line-seq log-file-reader)) first-item (first parsed-lines)] ; Iterate over the parsed lines (doseq [line parsed-lines] ; Do something with a side-effect ))) 

I don’t want to keep any list, I just want to perform a side effect with each item. I believe that there were no problems without first-item .

I have memory problems in my program, and I think that maybe keeping a link to something at the beginning of the parsed-line sequence means that the whole sequence is saved.

What is defined here? If the sequence is saved, is there a general way to take a copy of the object and include the implemented part of the sequence for garbage collection?

Answer 1

Sequence retention occurs here

 ... (let [parsed-lines (map parse-line (line-seq log-file-reader)) ... 

The sequence of lines in the file is lazily generated and parsed, but the entire sequence is kept within the let . This sequence is implemented in doseq , but doseq not a problem; it does not support the sequence.

 ... (doseq [line parsed-lines] ; Do something ... 

You don’t necessarily care about saving the sequence in let , because the let area is limited, but here your file is supposedly large and / or you stay within the dynamic let for or maybe return a closure containing it in the “do something” section .

Note that holding onto any given element of a sequence, including the first, does not contain a sequence. The term head-holding is a little incorrect if you think the chapter is the first element, as in the "chapter of the list" in Prolog. The problem is the link to the sequence.

Answer 2

The JVM will never return memory to the OS when it becomes part of the java heap, and unless you configure it differently, the default maximum heap size is pretty large (usually 1/4 of free memory). Therefore, if you only experience vague issues such as "Gosh, it takes up a lot of memory" and not "Well, the JVM threw OutOfMemoryError", you probably just didn't configure the JVM the way you would like an act. partition-by little impatient as it contains one or two partitions in memory at once, but if your partitions are not huge, you should not get out of the heap with this code. Try installing -Xmx100m or whatever you think is a reasonable heap size for your program and see if you have any problems.