Geek Answers Handbook

Exception in thread "main" java.lang.NoClassDefFoundError: org / apache / spark / rdd / RDD

Please note that I am a better dataminer than a programmer. I am trying to run examples from the book "Advanced Analytics Using Spark" by Sandy Ryza (these code examples can be downloaded from https://github.com/sryza/aas "), and I encountered the following problem. When I open this project in Intelij Idea and trying to start it, I get the error "Exception in thread" main "java.lang.NoClassDefFoundError: org / apache / spark / rdd / RDD" Does anyone know how to solve this problem?
Does this mean what am i using the wrong spark version?

At first, when I tried to run this code, I got the error "Exception in thread" main "java.lang.NoClassDefFoundError: scala / product", but I solved it by setting scala -lib to compile in maven, I use Maven 3.3.9 , Java 1.7.0_79 and scala 2.11.7, spark 1.6.1. I tried both Intelij Idea 14 and 15 different versions of java (1.7), scala (2.10) and spark, but to no avail. I also use windows 7. My SPARK_HOME and Path parameters are set, and I can execute the spark shell from the command line.

+4
source share
1 answer

-master sparkshell, , . Hadoop, . , (:///), HDFS (hdfs://)

:

REPL , , .

, , . JAR , -jars, maven .

, simplespark:

maven

cd simplesparkproject/
mvn package

jar

spark-shell --master local[2] --driver-memory 2g --jars ../simplesparkproject-0.0.1.jar ../README.md

:

val myApp = com.cloudera.datascience.MyApp

, , pom.xml. , . , pom.xml.

<!--<scope>provided</scope>-->

, . .

: maven 3.0.5 Java 7+. maven 3.3.X .

+1

More articles:


All Articles