JAR HELL in the IDE (intelliJ), mvn-test and mvn-package: how to unify dependency movement in all settings?

Question

A simple case for me: I use 2 very popular but conflicting scala libraries: Spark SQL and JSON4s. Each of them depends on different versions of the Jackson XML parser.

Since no one else uses JSON4 except my own program, a simple solution is to move the org.json4s.jackson and com.fasterxml.jackson to new locations.

However, the maven-shade plugin can only do this during the packaging process, since before that all tests and IDEs are executed. This still causes all tests to fail, regardless of whether to run in mvn-test or scala in the IDE.

Is there a plugin that can collectively manage package transfer policies for all three cases: JAR packaging / mvn-test / IDE-run? I searched the Internet all day and did not find an answer.

Answer 1

We solved the problem by recompiling (and fixing) to align the libraries. Everything related to the shade will contribute to one version over another, which means that you can fix in one library that you could split into another.

And for those comments from the OSGi camp, yes, the fact that OSGi was intended to be fixed, but this does not work so well in the context of Spark :)

Have you considered downgrading one of the two versions to align resources? Often one of the libraries manages to release the first one, leaving the second one a little bit ... Sometimes the answer can be found when starting a sequential, but older version, which is consistent with the "last to be released" project.