Apache Spark Maven Release and Application Dependencies

Question

I have to follow this tutorial to create a uber jar for my Apache Spark app with maven.

I installed all Spark dependencies in pom with <scope>provided</scope> . This works very well, but now when I run the application locally, I get an error for missing Spark dependencies.

At the moment, I had to remove the provided tag from pom.

How can I make the above spark dependencies only when creating an application for release?

I use Intellij as an IDE for application development.

Answer 1

You can create separate Maven profiles.

It is best to have a dependencyManagment section in the POM where you specify the versions, and then in profiles you will only have groupId + artifactId + scope

For instance:

 <profile> <id>dev</id> <activation> <activeByDefault>false</activeByDefault> </activation> <dependencies> <!-- Here Spark deps without provided --> <dependency> <groupId>...</groupId> <artifactId>...</artifactId> <scope>compile</scope> </dependency> </dependencies> </profile> <profile> <id>prod</id> <dependencies> <!-- Here Spark deps with provided --> <dependency> <groupId>...</groupId> <artifactId>...</artifactId> <scope>provided</scope> </dependency> </dependencies> </profile>