SBT from memory with subprojects

Question

We use SBT 0.13 and Java 8 JVM for CircleCI to create a Play application with several subprojects. Sometimes we ran into memory problems at CircleCI, where it interrupted our build because it exceeded 4 GB of memory usage.

Yesterday I added a new subproject to our assembly, and almost all assemblies fail due to a memory problem. It seems that adding subprojects also increases the amount of memory used for our build.

I tried a few things to reduce the memory load:

• Add _JAVA_OPTIONS: "-Xms512m -Xmx2048" to circle.yml , as described in the CircleCI Documentation Pages . (I noticed from the journal that the JVM really selects this option.)
• Add the -mem to the SBT call.
• Add concurrentRestrictions in Global += Tags.limit(Tags.Test, 1) to the top of the SBT file to make sure that at least the memory is not used all at once.

All these measures seem to have helped, but I have not yet found a final solution to this problem.

What else can I do to keep SBT memory usage under control?

EDIT . Our project has 5 subprojects: about 14,000 lines of Scala code (as well as 21,000 lines of Java code that we “inherited”). External memory usually (but not always) occurs when performing static analysis using FindBugs: we use this together with the FindSecurityBugs plugin to find security problems.

Answer 1

Here are two problems that are confused:

• CI circle does not select values ​​for memory limits

• Excess memory SBT

The first issue should be addressed with CircleCI documentation / examples. To explore why you use so much memory, you can run your sbt locally with memory limits below 4g (i.e. 2g). You will find yourself in one of these two cases:

• Your tests use too much memory, possibly due to memory leaks. Due to java.lang.OutOfMemoryError: GC overhead limit exceeded exit JVM. You must start the assembly locally using the profiler and see what causes the problem (database connections not closed?)

• The tests use too much memory due to the ability of SBT to dynamically load classes: in SBT, you can completely reload the class inside one JVM (for example, you can start the console, load the class, edit the file, recompile and restart the console and reload the class). As described in the Oracle documentation, there are no restrictions for Maximum MetaSpace in Java 8, and you must set it so that your heap + metaspasm <4GB. See https://blogs.oracle.com/poonam/entry/about_g1_garbage_collector_permanent