Use spark in sbt project in intellij

Question

Here is the build.sbt file:

name := "scalaChartTest" version := "1.0" scalaVersion := "2.11.7" //libraryDependencies += "org.jfree" % "jfreechart" % "1.0.19" // //libraryDependencies += "org.scalafx" %% "scalafx" % "8.0.0-R4" // //libraryDependencies += "com.github.wookietreiber" %% "scala-chart" % "latest.integration" libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.4.1" 

And after the error, I received the error message:

 15:56:30 SBT project import [warn] Multiple dependencies with the same organization/name but different versions. To avoid conflict, pick one version: [warn] * org.scala-lang:scala-compiler:(2.11.0, 2.11.7) [warn] * org.scala-lang:scala-reflect:(2.11.2, 2.11.7) [warn] * jline:jline:(0.9.94, 2.12.1) [warn] * org.scala-lang.modules:scala-parser-combinators_2.11:(1.0.1, 1.0.4) [warn] * org.scala-lang.modules:scala-xml_2.11:(1.0.1, 1.0.4) [warn] * org.slf4j:slf4j-api:(1.6.4, 1.7.10) 

What is wrong here?

Answer 1

You have indirect dependencies on the libraries mentioned in the warning. There is a conflict, since the version is indirectly different from what is indicated in your sbt file (in this case, probably the version of scala). The conflict is automatically resolved using sbt (choosing one of the versions depending on your configuration). However, the selected version may not be the version that you intend to use, so this is a warning.

In your case, this is probably not a problem. Although, if you want, you can explicitly exclude indirect dependencies:

 libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.4.1" excludeAll ( ExclusionRule(organization = "org.scala-lang"), ExclusionRule("jline", "jline"), ExclusionRule("org.slf4j", "slf4j-api") ) 

Answer 2

I learned from the spark programming guide that spark 1.4.1 is dependent on scala 2.10.x, so I change build.sbt to:

 name := "scalaChartTest" version := "1.0" scalaVersion := "2.10.5" libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.4.1" 

And the warning disappeared.