Geek Answers Handbook

What are AssemblyKeys used for and how to import them?

In a recent popular sbt application https://github.com/databricks/reference-apps I found a line that required me

import AssemblyKeys._

This line does not compile in SBT or in my IntelliJ IDEA.

What is import needed for and why is it needed?

+7
scala sbt apache-spark
source share
3 answers

Another answer from @mfirry pretty much answers what part of the assembly (definition) import AssemblyKeys._ brings. This is the sbt-assembly plugin, which (quoting the plugin docs ):

Create a complete JAR of your project with all its dependencies.

It is necessary for the plugin to do its job.

You may ask yourself why I need a plugin at all.

Since you did not refer to an application requiring import and, therefore, to a plug-in, and I did not consider examples, I can only guess by Databricks that the commercial object behind Apache Spark uses examples.

To deploy the application in the Apache Spark cluster, you need to build the entire application and configure the workers so that they can access the binary files (with the necessary dependencies, namely with banks). That the purpose of the sbt-assembly plugin is to pack everything and offer one jar of the application that Spark can use. See Standalone Applications in the Apache Spark documentation for a separate case (without a cluster environment):

For sbt to work correctly, you need to use the SimpleApp.scala and simple.sbt layout in accordance with the typical directory structure. When this is in place, we can create a JAR package containing the application code, then use the spark-submit script function to run our program.

You can read Cluster Mode Overview to get an overview of how to deploy a Spark application on a Spark cluster.

+2
source share

In the latest version, we no longer need to import AssemblyKeys._

Check update guide here

+4
source share

Take a look at the sbt-assembly plugin.

You basically need

  • add the following line to the file: assembly.sbt (or really any *.sbt file) in the project folder in your sbt project:

     addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

  • put the import line at the beginning of your build.sbt (in the directory where the project subdirectory is located).

     import AssemblyKeys._

With the above files, the main structure of the project should look like this:

 ➜ myProjectName tree . |-- build.sbt `-- project `-- assembly.sbt 1 directory, 2 files
+2
source share

More articles:


All Articles