Maven Docker Cache Dependencies

Question

I am trying to use docker to automate the maven build. The project I want to build takes about 20 minutes to download all the dependencies, so I tried to create a docker image that will cache these dependencies, but it does not seem to save it. My docker file

FROM maven:alpine RUN mkdir -p /usr/src/app WORKDIR /usr/src/app ADD pom.xml /usr/src/app RUN mvn dependency:go-offline 

The image builds and it loads everything. However, the resulting image is the same size as the base maven:alpine image, so it does not seem to cache dependencies in the image. When I try to use the image in the mvn compile , it goes through the full 20 minutes of reloading everything.

Is it possible to create a maven image that caches my dependencies so that they don't load every time I use the image to complete the build?

I run the following commands:

 docker build -t my-maven . docker run -it --rm --name my-maven-project -v "$PWD":/usr/src/mymaven -w /usr/src/mymaven my-maven mvn compile 

I understand that everything RUN does during the docker build process becomes part of the resulting image.

Answer 1

Usually there are no changes to the pom.xml file, but only some other changes to the source code when you try to start building a Docker image. Under such circumstances, you can do this:

FYI:

 FROM maven:3-jdk-8 ENV HOME=/home/usr/app RUN mkdir -p $HOME WORKDIR $HOME # 1. add pom.xml only here ADD pom.xml $HOME # 2. start downloading dependencies RUN ["/usr/local/bin/mvn-entrypoint.sh", "mvn", "verify", "clean", "--fail-never"] # 3. add all source code and start compiling ADD . $HOME RUN ["mvn", "package"] EXPOSE 8005 CMD ["java", "-jar", "./target/dist.jar"] 

So the key is:

1. add pom.xml file.

2. then mvn verify --fail-never load the maven dependencies.

3. Then add all your source files and start compiling ( mvn package ).

If there are changes in the pom.xml file or you run this script for the first time, docker will execute 1 → 2 → 3. If there are no changes in the pom.xml file, Docker will skip step 1 pom.xml and do 3 directly.

This simple trick can be used in many other package management situations (gradle 、 yarn pm npm pip).

Edit:

You should also consider using mvn dependency:resolve or mvn dependency:go-offline respectively as other comments & answers suggest.

Answer 2

It turns out that the image I use as the base has a parent image that defines

 VOLUME "$USER_HOME_DIR/.m2" 

see: https://github.com/carlossg/docker-maven/blob/322d0dff5d0531ccaf47bf49338cb3e294fd66c8/jdk-8/Dockerfile

As a result, during the build, all files are written to $USER_HOME_DIR/.m2 , but since this is expected to be volume, none of these files will be saved with the container image.

There is currently no way in Docker to unregister this volume definition, so you need to create a separate maven image rather than use the official maven image.

Answer 3

@Kim is closest, but not quite yet. I do not think adding --fail-never is correct, even through this to do the job.

The verify command calls a lot of plugins to execute, which is the problem (for me) - I don't think they should be executed when all I want to do is install the dependencies! I also have multi-module build and javascript build, so this complicates the setup even more.

But running just verify not enough, because if you run install in the following commands, more plugins will be used - which means more dependencies to download - maven refuses to download them otherwise. Relevant reading: Maven: introduction to the assembly life cycle

You must find which properties disable each plugin and add them one at a time so that they do not break your build.

 WORKDIR /srv # cache Maven dependencies ADD cli/pom.xml /srv/cli/ ADD core/pom.xml /srv/core/ ADD parent/pom.xml /srv/parent/ ADD rest-api/pom.xml /srv/rest-api/ ADD web-admin/pom.xml /srv/web-admin/ ADD pom.xml /srv/ RUN mvn -B clean install -DskipTests -Dcheckstyle.skip -Dasciidoctor.skip -Djacoco.skip -Dmaven.gitcommitid.skip -Dspring-boot.repackage.skip -Dmaven.exec.skip=true -Dmaven.install.skip -Dmaven.resources.skip # cache YARN dependencies ADD ./web-admin/package.json ./web-admin/yarn.lock /srv/web-admin/ RUN yarn --non-interactive --frozen-lockfile --no-progress --cwd /srv/web-admin install # build the project ADD . /srv RUN mvn -B clean install 

but some plugins are not so easy to miss - I'm not a maven expert (so I don’t know why he ignores the cli option - this may be a mistake), but the following works as expected for org.codehaus.mojo:exec-maven-plugin

 <project> <properties> <maven.exec.skip>false</maven.exec.skip> </properties> <build> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>exec-maven-plugin</artifactId> <version>1.3.2</version> <executions> <execution> <id>yarn install</id> <goals> <goal>exec</goal> </goals> <phase>initialize</phase> <configuration> <executable>yarn</executable> <arguments> <argument>install</argument> </arguments> <skip>${maven.exec.skip}</skip> </configuration> </execution> <execution> <id>yarn run build</id> <goals> <goal>exec</goal> </goals> <phase>compile</phase> <configuration> <executable>yarn</executable> <arguments> <argument>run</argument> <argument>build</argument> </arguments> <skip>${maven.exec.skip}</skip> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> 

pay attention to the explicit <skip>${maven.exec.skip}</skip> - other plugins select this from the cli parameters, but not this one (neither -Dmaven.exec.skip=true nor -Dexec.skip=true work itself by oneself)

Hope this helps

Answer 4

Similarly with @Kim's answer, but I am using dependency:resolve solve the mvn command. So here is my complete Dockerfile:

 FROM maven:3.5.0-jdk-8-alpine WORKDIR /usr/src/app # First copy only the pom file. This is the file with less change COPY ./pom.xml . # Download the package and make it cached in docker image RUN mvn -B -f ./pom.xml -s /usr/share/maven/ref/settings-docker.xml dependency:resolve # Copy the actual code COPY ./ . # Then build the code RUN mvn -B -f ./pom.xml -s /usr/share/maven/ref/settings-docker.xml package # The rest is same as usual EXPOSE 8888 CMD ["java", "-jar", "./target/YOUR-APP.jar"] 

Answer 5

I do not think the other answers here are optimal. For example, the mvn verify performs the following steps and does much more than just resolve dependencies:

validate - confirm the correctness of the project and get all the necessary information

compile - compile the source code of the project

test - test the compiled source code using a suitable unit testing environment. These tests should not require code to be packaged or deployed.

package - take the compiled code and pack it into a distributed format such as JAR.

check - perform any checks on the results of integration tests to ensure compliance with the quality criteria

All of these steps and their associated goals do not need to be run if you want to resolve dependencies.

If you want to allow dependencies only, you can use dependency:go-offline target:

 FROM maven:3-jdk-12 WORKDIR /tmp/example/ COPY pom.xml . RUN mvn dependency:go-offline COPY src/ src/ RUN mvn package 

Answer 6

After several days of struggle, I managed to do this caching later using an intermediate container, and I would like to summarize my conclusions here, since this topic is so useful and often appears on the Google search home page:

1. Kim's answer only works under a certain condition: pom.xml cannot be changed, plus Maven regularly updates daily by default
2. The mvn dependency: go-offline -B --fail-never has such a flaw, so if you need to get fresh code from a repo, it is likely that Maven will run a full scan every time
3. Mounting the volume also does not work, because we need to resolve dependencies during image build
4. Finally, I have a compatible workable solution (may not work with others):
   • First create an image to resolve all the dependencies (not an intermediate image)
   • Create another Dockerfile with an intermediate image, with dockerfile examples:

 #docker build -t dependencies . From ubuntu COPY pom.xml pom.xml RUN mvn dependency:go-offline -B --fail-never 

 From dependencies as intermediate From tomcat RUN git pull repo.git (whatsoever) RUN mvn package 

The idea is to keep all the dependencies in another image that Maven can use immediately

There may be other scenarios that I have not yet encountered, but this solution makes it a little easier for me to load 3GB of garbage every time I can not imagine why Java has become such a bold whale in today's meager world

Answer 7

I had this problem only litle. There are many solutions on the Internet, but the one that worked for me simply mounts the volume for the maven modules directory:

 mkdir /opt/myvolumes/m2 

then in the docker file:

 ... VOLUME /opt/myvolumes/m2:/root/.m2 ... 

There are better solutions, but not so simple.

This blog post adds an extra mile, helping you cache everything:

https://keyholesoftware.com/2015/01/05/caching-for-maven-docker-builds/

Answer 8

There are two ways to cache maven dependencies:

1. Run "mvn verify" as part of container execution, DO NOT create and make sure that you are mounting .m2 from the volume.

   This is effective, but it doesn’t work well with the cloud assembly and several slave slaves.

2. Use the "dependency cache container" and update it periodically. Here's how to do it:

   but. Create a Dockerfile that copies pom and create standalone dependencies:

    FROM maven:3.5.3-jdk-8-alpine WORKDIR /build COPY pom.xml . RUN mvn dependency:go-offline 

   b. Create it periodically (for example, at night) as "Deps: latest"

   from. Create another Dockerfile to actually build the system according to the commit (preferably using multi-stage) - and make sure that these are FROM Deps.

Using this system, you will have fast, rebuildable assemblies with a fairly good cache.

Answer 9

Using BuildKit

Starting with Docker v18.03 , you can use BuildKit instead of the volumes that were mentioned in other answers. This allows you to mount caches that may persist between assemblies, and you can avoid loading the contents of the corresponding .m2/repository each time.

Assuming Dockerfile is at the root of your project:

 # syntax = docker/dockerfile:1.0-experimental FROM maven:3.6.0-jdk-11-slim AS build COPY . /home/build RUN mkdir /home/.m2 WORKDIR /home/.m2 USER root RUN --mount=type=cache,target=/root/.m2 mvn -f /home/build/pom.xml clean compile 

target=/root/.m2 mounts the cache at the specified location in the Dockerfile with the image maven image docs .

To build, you can run the following command:

 DOCKER_BUILDKIT=1 docker build --rm --no-cache . 

More information about BuildKit can be found here .

Answer 10

If dependencies are loaded after the container is already inserted, you need to commit the changes to this container and create a new image with the loaded artifacts.