How can I avoid heap memory shortages while processing huge sequences in Clojure?

Question

I am trying to create a sequence that matches the broad search for a very wide, deep tree ... and I run into memory problems when I go too far along the sequence. Having asked on the IRC channel and looked here, reason No. 1 in such problems is inadvertently held on to the head; but I don’t see where I do it.

The code is pretty simple; here is the version that displays the problem:

(def atoms '(abc)) (defn get-ch [n] (map #(str n %) atoms)) (defn add-ch ([] (apply concat (iterate add-ch atoms))) ([n] (mapcat get-ch n))) (dorun (take 20000000 (add-ch))) 

And here is another version (which I started working with before getting help from # clojure) that displays the same problem:

 (def atoms '(abc)) (defn get-children [n] (map #(str n %) atoms)) (defn add-layer ([] (add-layer atoms)) ([n] (let [child-nodes (mapcat get-children n) ] (lazy-seq (concat n (add-layer child-nodes)))))) (dorun (take 20000000 (add-layer))) 

Both give me "OutOfMemoryError Java heap space". I run them from REPL in Eclipse / CounterClockwise on a Macbook Air.

I am new to Clojure, therefore, after I fought off this for a day, I hope this is something trivial that I don’t notice about. I understand that I can increase the heap size to make the problem less likely, but the sequences that I ultimately want to process are so extensive that I don't think it will help me.

I tried replacing "take" (in the examples above) with "drop" to avoid holding my head - it does not matter.

Answer 1

I missed the dorun. The problem seems to be related to StringBuilder string.

This works if I replace get-children as shown below:

  (defn get-children [n] (map #(if (seq? n) (conj n %) (conj (list n) %)) atoms))