Geek Answers Handbook

Spark Streaming with Kafka 2.0.0 Dependencies

I am trying to use kafka 0.8 theme with spark-streaming2.0.0, I am trying to identify the required dependencies that I tried to use with these dependencies in the build.sbt file

libraryDependencies += "org.apache.spark" %% "spark-streaming_2.11" % "2.0.0"

when I run the sbt package, I get unresolved dependencies for all three of these jars,

But these banks exist

https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-8_2.11/2.0.0

Help debug this problem, I am new to Scala, so please let me know if I do not do something right

+5
source share
1 answer

The problem is that you are specifying the version of Scala, as well as using %% , which is trying to determine which version of Scala you are using.

Or delete one % :

 libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.0.0" libraryDependencies += "org.apache.spark" % "spark-streaming_2.11" % "2.0.0" libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.0.0"

Or uninstall the Scala version:

 libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.0.0" libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka-0-8" % "2.0.0"
+10
source

More articles:


All Articles