Geek Answers Handbook

Effective binary serialization for Clojure / Java

I am looking for a way to efficiently serialize Clojure objects in binary format, i.e. Not just doing classic serialization of print and read.

i.e. I want to do something like:

(def orig-data {:name "Data Object" 
                :data (get-big-java-array) 
                :other (get-clojure-data-stuff)})

(def binary (serialize orig-data))

;; here "binary" is a raw binary form, e.g. a Java byte array
;; so it can be persisted in key/value store or sent over network etc.

;; now check it works!

(def new-data (deserialize binary))

(= new-data orig-data)
=> true

The motivation is that I have several large data structures that contain a significant amount of binary data (in Java arrays), and I want to avoid the overhead of converting this data to text and vice versa. In addition, I am trying to keep the format compact in order to minimize the use of network bandwidth.

Features I would like to have:

  • Lightweight, clean Java implementation
  • Support for all standard Clojure data structures, as well as all Java primitives, arrays, etc.
  • / - , " "

/ Clojure?

+5
4

, - , Java? , , - ?

A Clojure Java :

(defn serializable? [v]
  (instance? java.io.Serializable v))

(defn serialize 
  "Serializes value, returns a byte array"
  [v]
  (let [buff (java.io.ByteArrayOutputStream. 1024)]
    (with-open [dos (java.io.ObjectOutputStream. buff)]
      (.writeObject dos v))
    (.toByteArray buff)))

(defn deserialize 
  "Accepts a byte array, returns deserialized value"
  [bytes]
  (with-open [dis (java.io.ObjectInputStream.
                   (java.io.ByteArrayInputStream. bytes))]
    (.readObject dis)))

 user> (= (range 10) (deserialize (serialize (range 10))))
 true

, , . Java Clojure atom/agent/future, , Java , Clojure , .

- -. , , .

, Java-, Java . ( , 100k : serialize 0.9 ms, 100kB, 490 ms, 700kB.)

, (= new-data orig-data) ( Java equals, , ), / .

user> (def a (range 10))
user> (= a (range 10))
true
user> (= (into-array a) (into-array a))
false
user> (.equals (into-array a) (into-array a))
false
user> (java.util.Arrays/equals (into-array a) (into-array a))
true
+8

Google protobuf? , GitHub Clojure.

+4

Nippy - imho: https://github.com/ptaoussanis/nippy

+4

, , , . , , , ... Clojure , print-method .

, , . java, protobuf clojure: , , .

+3

More articles:


All Articles