Iterator blocks in Clojure?

Question

I use clojure.contrib.sql to retrieve some records from a SQLite database.

 (defn read-all-foo [] (with-connection *db* (with-query-results res ["select * from foo"] (into [] res)))) 

Now I really don’t want to implement the whole sequence before returning from the function (i.e. I want it to be lazy), but if I return res directly or wrap it with some kind of lazy shell (for example, I want to make a specific map transformation in the sequence of results ), the SQL bindings will reset, and the connection will be closed after I return, so the implementation of the sequence will throw an exception.

How can I wrap an entire function in a closure and return some kind of iterator block (e.g. yield in C # or Python)?

Or is there another way to return a lazy sequence from this function?

Answer 1

resultset-seq returned with-query-results is probably already lazy, how are you going to get it. As you said, the tape only works as long as the handle is open. There is no way around this. You cannot read from the database if the database descriptor is closed.

If you need to perform I / O and save data after closing the handle, open the pen, quickly disassemble it (defeat laziness), close the pen and continue working with the results. If you want to doseq over some data without storing it all in memory at once, open the descriptor, get the lazy seq on the data, doseq above it, then close the descriptor.

So, if you want to do something with each row (for side effects) and discard the results without eating the whole set of results in memory, you can do this:

 (defn do-something-with-all-foo [f] (let [sql "select * from foo"] (with-connection *db* (with-query-results res [sql] (doseq [row res] (f row)))))) user> (do-something-with-all-foo println) {:id 1} {:id 2} {:id 3} nil ;; transforming the data as you go user> (do-something-with-all-foo #(println (assoc % :bar :baz))) {:id 1, :bar :baz} {:id 2, :bar :baz} {:id 3, :bar :baz} 

If you want your data to hang in the long run, you can also impose all of this on the use of the read-all-foo function above (thus defeating laziness). If you want to convert the data, then map according to the results after you have extracted everything. Your data will be all in memory at that moment, but the map call itself and your data transformations after printing will be lazy.

Answer 2

In fact, you can add a “final side effect” to the lazy sequence that will run once when the entire sequence is consumed for the first time:

 (def s (lazy-cat (range 10) (do (println :foo) nil))) (first s) ; => returns 0, prints out nothing (doall (take 10 s)) ; => returns (0 1 2 3 4 5 6 7 8 9), prints nothing (last s) ; => returns 9, prints :foo (doall s) ; => returns (0 1 2 3 4 5 6 7 8 9), prints :foo ; or rather, prints :foo if it it the first time s has been ; consumed in full; you'll have to redefine it if you called ; (last s) earlier 

I'm not sure that I will use this to close the connection to the database, although, in my opinion, it is considered best practice not to keep in touch with the database indefinitely and putting your call to close the connection at the end of your lazy sequence of results will not only maintain the connection longer than necessary, but also to open the possibility that your program will not work for an independent reason, without closing the connection. So for this scenario, I would usually just rip all the data. As Brian says, you can store anything you want, raw, than do any conversion lazily, so you should be fine until you try to pull a really huge data set into one piece.

But then I do not know your exact circumstances, therefore, if it makes sense from your point of view, you can definitely call the function to close the connection at the end of its sequence of results. As Michel Borkent points out, you cannot use with-connection if you want to do this.

Answer 3

I had never used SQLite with Clojure before, but I assume that when connecting closes the connection when its body has been evaluated. Therefore, you need to manage the connection yourself if you want it to be open, and close it when you finish reading the elements you are interested in.

Answer 4

It is not possible to create a function or macro "on top" with-connection and with-query-results to add laziness. Both close their Connection and ResultSet, respectively, when the control flow leaves the lexical domain.

As Michal said, it would be nice to create a lazy seq by closing it with ResultSet and Connection lazily. As he also said, it would not be a good idea if you cannot guarantee that the sequences will eventually be completed.

A possible solution could be:

 (def *deferred-resultsets*) (defmacro with-deferred-close [&body] (binding [*deferred-resultsets* (atom #{})] (let [ret# (do ~@body )] ;;; close resultsets ret# )) (defmacro with-deferred-results [bind-form sql & body] (let [resultset# (execute-query ...)] (swap! *deferred-resultsets* conj resultset# ) ;;; execute body, similar to with-query-results ;;; but leave resultset open )) 

This would allow, for example, storing the results until the current query is completed.