Lenic concatenation of sequence in Clojure

Question

Here's a beginner's question: is there a way in Clojure to lazily concatenate an arbitrary number of sequences? I know the lazy-cat macro, but I cannot come up with its correct application for an arbitrary number of sequences.

My use case is lazy loading data from an API through programmed (biased / restricted) requests. Each query executed using request-fn below will retrieve 100 results:

 (map request-fn (iterate (partial + 100) 0)) 

If there are no more results, request-fn returns an empty sequence. This is when I stop the iteration:

 (take-while seq (map request-fn (iterate (partial + 100) 0))) 

For example, an API can return up to 500 results and can be ridiculed as:

 (defn request-fn [offset] (when (< offset 500) (list offset))) 

If I want to concatenate the results, I can use (apply concat results) , but it readily evaluates the sequence of results:

 (apply concat (take-while seq (map request-fn (iterate (partial + 100) 0)))) 

Is there a way to lazily concatenate a sequence of results using either lazy-cat or something else?

Answer 1

For the record, apply will only consume a sufficient sequence of arguments, since it must determine which of the arity needs to be called for the provided function. Since the maximum arity of concat is 3, apply will display no more than 3 elements from the base sequence.

If these API calls are expensive, and you really can't afford to do unnecessary ones, then you'll need a function that accepts seq-of-seqs and lazily combines them one at a time. I don’t think there is something built-in, but it’s quite simple to write your own:

 (defn lazy-cat' [colls] (lazy-seq (if (seq colls) (concat (first colls) (lazy-cat' (next colls))))))