Third-Party Shading

Question

I am currently facing the problem of deploying uber-jar in a Spark Streaming application, where there are congruent JAR files with different versions that cause a spark to rule out runtime exceptions. This library contains TypeSafe Config .

After many attempts, my solution was to defer the shadowing of the provided dependency so that it would not collide with the JAR provided by Spark at runtime.

Therefore, I went to the documentation for sbt-assembly and when shading I saw the following example:

 assemblyShadeRules in assembly := Seq( ShadeRule.rename("org.apache.commons.io.**" -> " shadeio.@1 ") .inLibrary("commons-io" % "commons-io" % "2.4", ...).inProject ) 

Trying to obscure com.typesafe.config , I tried applying the following solution to my build.sbt :

 assemblyShadeRules in assembly := Seq( ShadeRule.rename("com.typesafe.config.**" -> " shadeio.@1 ").inProject ) 

I assumed that he should have renamed any TypeSafe Config link in my project. But that does not work. It corresponds to several classes in my project and forces them to be removed from uber jar. I see this when I try to run sbt assembly :

 Fully-qualified classname does not match jar entry: jar entry: ***/Identifier.class class name: **/Identifier.class Omitting ***/OtherIdentifier.class. Fully-qualified classname does not match jar entry: jar entry: ***\SparkBaseJobRunner$$anonfun$1.class class name: ***/SparkBaseJobRunner$$anonfun$1.class 

I also tried using:

 assemblyShadeRules in assembly := Seq( ShadeRule.rename("com.typesafe.config.**" -> " shadeio.@1 ") .inLibrary("com.typesafe" % "config" % "1.3.0") 

This completed the uber JAR build process, but did not have the desired runtime effect.

I'm not sure I fully understand what effect shading has in my build process using sbt.

How can I obscure com.typesafe.config in my project, so when I call the library at runtime, Spark will load my shaded library and avoid the collision caused by versioning?

I am running sbt-assembly v0.14.1

Answer 1

Turns out it was a bug in sbt-assembly , where the hatch was completely broken on Windows. This caused the source files to be removed from the uber JAR, and for the tests to fail, because the specified classes are not available.

I created a pull request to fix this . Starting with version 0.14.3 SBT, the shader function works correctly. All you have to do is upgrade to the appropriate version in plugins.sbt :

 addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3") 

To obscure a specific JAR in your project, you do the following:

 assemblyShadeRules in assembly := Seq( ShadeRule.rename("com.typesafe.config.**" -> " my_conf.@1 ") .inLibrary("com.typesafe" % "config" % "1.3.0") .inProject ) 

This will rename the assembly com.typesafe.config , which will be packaged inside my_conf . Then you can view it with jar -tf on your assembly (irrelevant parts are omitted for brevity):

 ***> jar -tf myassembly.jar my_conf/ my_conf/impl/ my_conf/parser/ 

Edit

I wrote a blog post describing the problem and the process that led to this for anyone interested in a more detailed explanation.