How to override dependency on a specific task in sbt

Question

I want to redefine project dependency in a specific Task. I have an sbt multiproject that uses spark.

lazy val core = // Some Project val sparkLibs = Seq( "org.apache.spark" %% "spark-core" % "1.6.1" ) val sparkLibsProvided = Seq( "org.apache.spark" %% "spark-core" % "1.6.1" % "provided" ) lazy val main = Project( id = "main", base = file("main-project"), settings = sharedSettings ).settings( name := "main", libraryDependencies ++= sparkLibs, dependencyOverrides ++= Set( "com.fasterxml.jackson.core" % "jackson-databind" % "2.4.4" ) ).dependsOn(core) 

When I try to make a fat can to send yarn to my cluster, I use the https://github.com/sbt/sbt-assembly task. But in this case, I want to use sparkLibsProvided instead of sparkLibs something like:

 lazy val sparkProvided = (project in assembly).settings( dependencyOverrides ++= sparkLibsProvided.toSet ) 

How can I override this dependency correctly?

Answer 1

You can create a new project, which is a dedicated project for creating your spark uber jar with the flag provided:

 lazy val sparkUberJar = (project in file("spark-project")) .settings(sharedSettings: _*) .settings( libraryDependencies ++= sparkLibsProvided, dependencyOverrides ++= Set( "com.fasterxml.jackson.core" % "jackson-databind" % "2.4.4" ) ) 

And when you get together in sbt, first go to the specified project:

 sbt project sparkUberJar sbt assembly 

Answer 2

This can be easily achieved with a key provided specifically for what you want:

 assemblyExcludedJars in assembly := { val cp = (fullClasspath in assembly).value cp filter { _.data.getName == "spark-core-1.6.1.jar" } } 

However, this approach is considered hacked, and it would be better if you could manage to split your configuration into subprojects, as this is also warned in the official documentation here :

If you need to tell sbt-assembly to ignore the JAR, you are probably wrong. task assembly captures JARs from your path to the project class. First try setting the class path first.