Many errors ERROR ErrorMonitor: AssociationError when starting the ignition

Question

I use a spark on a meso (with kafka and cassandra). When I start, I have a lot of errors (~ 100), then everything works fine. Errors of this type:

[Stage 0: =======> (24 + 26) / 50] [Stage 3:> (0 + 24) / 24] 09/15/17 09:48:35 Error: Error modifier: AssociationError [ akka.tcp: // [e-mail] : 58325] <- [akka.tcp: // [e-mail] : 59441]: Error [Turn off address: akka.tcp: // driverPropsFetcher@10. 131.xx.xxx : 59441] [akka.remote.ShutDownAssociation: Turn off the address: akka.tcp: // [e-mail] : 59441 Called: akka.remote.transport.Transport $ InvalidAssociationException: the remote system completed the merge because it completed. ]

What is the reason for this error? How can I solve it?

Answer 1

The magazine seems like a noise.

This issue also occurs in Spark 1.5.0 on CDH 5.5.0, and Cloudera says below.

http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_rn_spark_ki.html

When using Spark on YARN, the driver reports erroneous error messages. The Spark driver reports erroneous error messages, for example:

Error in ErrorMonitor: AssociationError [akka.tcp: // sparkDriver @ ...] → [akka.tcp: // sparkExecutor @ ...]: Error [Error in association with [akka.tcp: // sparkE xecutor @ .. .]] [akka.remote.EndpointAssociationException: Association failed with [akka.tcp: // sparkExecutor @ ...]]

Workaround: Add the following property to the Spark log4j configuration file: log4j.logger.org.apache.spark.rpc.akka.ErrorMonitor = FATAL. See Configuring Spark Application Logging Properties.