I am trying to run Spark 1.4.1 on a CDH5.4 cluster through yarn. Serialization is for use by Kryo.
I have a large object that I send to performers as a broadcast. The object seems to serialize just fine. However, when he tries to deserialize, Cryo throws a ClassNotFoundException for the class, which in the fat jar I spark-submit .
What could be causing this classpath problem with Kryo for artists? Where should I start looking for a diagnosis of the problem? I appreciate any help you can provide.
Refresh . For some reason, the jar file that I send using spark-submit is not added to the classpath. I checked the node manager directory and it loads - it is renamed __app__.jar and placed in the working directories of the container via a symbolic link. The __spark__.jar file containing the spark assembly is in the same directory, and IS is included in the classpath. A folder containing it is also included.
When I run in local mode, the presented human jar is included as a user-added class path element.
What could be the reason for this?
source share