Geek Answers Handbook

Do we still have to make a fat jar to send work to Spark 2.0.0?

The Spark 2.0.0 note release says that:

Spark 2.0 no longer needs a container for collecting fat.

  • Does this mean that we no longer need to make a fat jar to submit work?

  • If so, how? Therefore, the documentation here is not updated.

+6
source share
1 answer

Does this mean that we no longer need to make a fat bank provide jobs?

Unfortunately not. You still need to create a JAR for Uber to deploy Sparks.

The title of the release notes is very misleading. The actual meaning is that Spark itself, as a dependency, is no longer compiled into the uber JAR, but acts like a regular JAR application with dependencies. You can see it in more detail @ SPARK-11157 , which is called “Allow sparks to be created without assemblies”, and read the article entitled “Replacing the spark assembly with old good jars” , which describes the pros and cons of deploying Spark not like a few huge JAR (Core, Streaming, SQL, etc.), but as several JARs with a relatively regular size, containing the code and the lib/ directory with all the related dependencies.

If you really need details, this pull request touches on several key parts.

+8
source

More articles:


All Articles