Unable to open native spark connection

Question

I am launching Spark with Spark 1.4 and Cassandra 2.18. I telnet from the master and runs on a cassandra machine. Sometimes work is done fine, and sometimes I get the following exception. Why does this happen only occasionally?

"Exception in the thread" main "org.apache.spark.SparkException: The operation is interrupted due to the failure of the stage: Task 0 at stage 0.0 failed 4 times, last failure: Lost task 0.3 at stage 0.0 (TID 7, 172.28. 0.162) : java.io.IOException: Failed to open native connection to Cassandra at: {172.28.0.164}: 9042 at com.datastax.spark.connector.cql.CassandraConnector $ .com $ datastax $ spark $ connector $ cql $ CassandraConnector $ $ createSession (CassandraConnector.scala: 155) "

Sometimes this also gives me this exception along with the top one:

Called: com.datastax.driver.core.exceptions.NoHostAvailableException: all the hosts (s) tried for the request did not work (tried: /172.28.0.164: 9042 (com.datastax.driver.core.TransportException: [/172.28. 0.164: 9042] Connection closed))

Answer 1

I had a second “NoHostAvailableException” error that occurred to me several times this week when I was porting a Python spark to Java Spark.

I had problems with the fact that the driver stream was almost out of memory, and the GC occupied all my cores (98% of all 8 cores), constantly stopping the JVM.

In python, when this happens, it is much more obvious (for me), so it took me a little time to figure out what was happening, so I got this error several times.

I had two theories for the root cause, but the solution was not that the GC was going crazy.

• The first theory was that since it stopped so often, I simply could not connect to Cassandra.
• The second theory: Kassandra was running on the same machine as Spark, and the JVM received 100% of the entire processor, so Cassandra just couldn't answer on time, and she looked at the driver as if there was no Cassandra host.

Hope this helps!