Spark Kryo: register your own serializer

Question

I have a class that implements the Kryo native serializer, implementing the read() and write() com.esotericsoftware.kryo.Serializer from com.esotericsoftware.kryo.Serializer (see example below). How can I register this custom serializer with Spark?

Here is a sudo-code example of what I have:

 class A() CustomASerializer extends com.esotericsoftware.kryo.Serializer[A]{ override def write(kryo: Kryo, output: Output, a: A): Unit = ??? override def read(kryo: Kryo, input: Input, t: Class[A]): A = ??? } val kryo: Kryo = ... kryo.register(classOf[A], new CustomASerializer()); // I can register my serializer 

Now in Spark:

 val sparkConf = new SparkConf() sparkConf.registerKryoClasses(Array(classOf[A])) 

Unfortunately, Spark does not allow me to register my own serializer. Any idea if there is a way to do this?

Answer 1

Create your own KryoRegistrator using a registered user registrar:

 package com.acme class MyRegistrator extends KryoRegistrator { override def registerClasses(kryo: Kryo) { kryo.register(classOf[A], new CustomASerializer()) } } 

Then set spark.kryo.registrator to your full registrar name, for example. com.acme.MyRegistrator :

 val conf = new SparkConf() conf.set("spark.kryo.registrator", "com.acme.KryoRegistrator")