SaveToCassandra with spark-cassandra throws connector java.lang.ClassCastException

Question

When I try to save data in Cassandra (in Scala), I get the following exception:

java.lang.ClassCastException: com.datastax.driver.core.DefaultResultSetFuture cannot be discarded by com.google.common.util.concurrent.ListenableFuture

Please note that I do not get this error every time, but from time to time it happens randomly, which makes it more dangerous in production.

I am using YARN and I have shaded com.google. ** to avoid collision of the Guava character.

Here's the code snippet:

rdd.saveToCassandra(keyspace,"movie_attributes", SomeColumns("movie_id","movie_title","genre")) 

Any help would be greatly appreciated.

UPDATE Adding information from the pom file as requested:

 <dependency> <groupId>com.datastax.spark</groupId> <artifactId>spark-cassandra-connector_2.10</artifactId> <version>1.5.0</version> </dependency> <dependency> <groupId>com.datastax.spark</groupId> <artifactId>spark-cassandra-connector-java_2.10</artifactId> <version>1.5.0</version> </dependency> **Shading guava** <relocation> <!-- Conflicts between Cassandra Java driver and YARN --> <pattern>com.google</pattern> <shadedPattern>oryx.com.google</shadedPattern> <includes> <include>com.google.common.**</include> </includes> </relocation> 

Spark: 1.5.2 Cassandra Version: 2.2.3

Answer 1

Almost everyone who works on C * and Spark has seen such errors. The main reason is explained here .

The C * driver depends on the relatively new version of guava, while Spark depends on the older guava. To solve this problem before connector 1.6.2, you need to explicitly insert the C * driver and guava into your application.

Since 1.6.2 and 2.0.0-M3, by default the connector comes with the correct C * driver and shaded guava. So you should be fine with the connector artifact included in your project.

Things get complicated if your Spark application uses other libraries depending on the C * driver. Then you have to manually enable the non-shaded version of the connector, fix the C * driver and the shaded guava, and deploy the live jar. You essentially make your own connector package. In this case, you can no longer use --package to start the Spark cluster.

TL; dg

use connector 1.6.2 / 2.0.0-M3 or higher. 99% you should be fine.