ERROR SparkContext: error initializing SparkContext

Question

I am using spark-1.5.0-cdh5.6.0 . tried the sample application (scala) command:

> spark-submit --class com.cloudera.spark.simbox.sparksimbox.WordCount --master local /home/hadoop/work/testspark.jar 

The following error appeared:

  ERROR SparkContext: Error initializing SparkContext. java.io.FileNotFoundException: File file:/user/spark/applicationHistory does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:424) at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:100) at org.apache.spark.SparkContext.<init>(SparkContext.scala:541) at com.cloudera.spark.simbox.sparksimbox.WordCount$.main(WordCount.scala:12) at com.cloudera.spark.simbox.sparksimbox.WordCount.main(WordCount.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 

Answer 1

Spark has a feature called the "history server" that allows you to view historical events after SparkContext dies. This property is set by setting spark.eventLog.enabled to true .

You have two options: either specify a valid directory for storing the event log using the spark.eventLog.dir configuration spark.eventLog.dir , or simply set spark.eventLog.enabled to false if you do not need it.

You can learn more about this on the Spark Configuration page.