Dynamically create high-performance features in clojure

Question

I am trying to use Clojure to dynamically create functions that can be applied to large amounts of data, i.e. the requirement is that functions are compiled into bytecode for fast execution, but their specification is unknown until the execution time.

eg. Suppose I define functions with a simple DSL, for example:

(def my-spec [:add [:multiply 2 :param0] 3]) 

I would like to create a compile-spec function to:

 (compile-spec my-spec) 

Would return a compiled function of one parameter x, which returns 2x + 3.

What is the best way to do this in Clojure?

Answer 1

Hamza Erkikaya has already made the most important point, which is that Clojure code is always compiled. I just add illustrations and some information about some hanging fruit for your optimization efforts.

Firstly, the above point with Clojure code, which always compiles, includes closures returned by higher-order functions and functions created by calling forms eval on fn / fn* and, indeed, everything else that can act as a Clojure function . So you don't need a separate DSL to describe functions, just use higher order functions (and possibly macros):

 (defn make-affine-function [ab] (fn [x] (+ (* ax) b))) ((make-affine-function 31 47) 5) ; => 202 

It would be more interesting if your specifications included information on parameter types, since then you might be interested in writing a macro to generate code using these types of hints. The simplest example I can imagine is the option above:

 (defmacro make-primitive-affine-function [tab] (let [cast #(list (symbol (name t)) %) x (gensym "x")] `(fn [~x] (+ (* ~(cast a) ~(cast x)) ~(cast b))))) ((make-primitive-affine-function :int 31 47) 5) ; => 202 

Use :int :long :float or :double (or characters that do not match the namespace names of the corresponding names) as the first argument to take advantage of unrelated primitive arithmetic suitable for your argument types. Depending on what your function does, this can give you a very significant increase in performance.

Other types of hints are usually syntax #^Foo bar ( ^Foo bar does the same in 1.2); if you want to add them to macro-generated code, examine the with-meta function (you need to combine '{:tag Foo} into the metadata of the characters representing the formal arguments, to your functions, or let are the entered locales that you want to place type hints).

---

Oh, and in case you still want to know how to realize your original idea ...

You can always build a Clojure expression to define your function - (list 'fn ['x] (a-magic-function-to-generate-some-code some-args ...)) and call eval as a result. This will allow you to do something like the following (it would be easier to require the specification to include a list of parameters, but here the arguments assuming that the arguments should be deduced from the specification are all called paramFOO and must be sorted lexicographically):

 (require '[clojure.walk :as walk]) (defn compile-spec [spec] (let [params (atom #{})] (walk/prewalk (fn [item] (if (and (symbol? item) (.startsWith (name item) "param")) (do (swap! params conj item) item) item)) spec) (eval `(fn [~@(sort @params)] ~@spec)))) (def my-spec '[(+ (* 31 param0) 47)]) ((compile-spec my-spec) 5) ; => 202 

In the vast majority of cases, there is no good reason for this, and it should be avoided; use higher order functions and macros instead. However, if you do something like, say, evolutionary programming, then it is there, providing maximum flexibility - and the result is still a compiled function.

Answer 2

Even if you don't compile your AOT code as soon as you define the function, it compiles to bytecode on the fly.