SBT freezes dependency resolution

Question

We have a multi-project assembly SBT 0.13.0 with 17 projects: 1 sheet project, 15 modules that depend on the sheet (but not from each other) and 1 aggregator, which depends on 15 modules.

Here's a very rough idea of ​​what the Build.scala file looks like:

 val deps: Seq[Setting[_]] = Seq(libraryDependencies ++= Seq( "com.foo" % "foo" % "1.0.0", "com.bar" % "bar" % "1.0.0" )) val leaf = Project("leaf").settings(deps:_*) val module1 = Project("module1").dependsOn(leaf).settings(deps:_*) val module2 = Project("module2").dependsOn(leaf).settings(deps:_*) ... val module15 = Project("module15").dependsOn(leaf).settings(deps:_*) val aggregator = Project("aggregator)".dependsOn( module1, module2, ... module15 ).settings(deps:_*) 

All of these projects list exactly the same set of external dependencies as libraryDependencies . For some reason, when we run the update command in the aggregator, it takes about a minute for the project (~ 15 minutes in total!), Although there is no new dependency for permission or download.

Even worse, we recently added another dependency, and now the update command causes SBT to bloat up to ~ 5 GB of memory and sometimes freezes completely during resolution. How do we debug this?

We tried YourKit to profile it, and it might be a herring to read, but so far the only thing we see is the sbt.MultiLogger class spent over time in a call to BufferedOutputStream.flush .

Answer 1

If some of your external dependencies are actually your own libraries, redirected to the local repository, and their version is set to "last", then this hang is expected. The reason is that ivy is trying to use all repositories for all dependencies when the "latest" version is required. Since your libraries are not placed in public repositories, then checking for them in public repositories ends with a timeout (apparently this is an ivy problem).

I tried to replicate your installation and created an sbt project with a sheet, 15 modules, an aggregator, and several external dependencies. Everything is solved pretty quickly. You can check it out on

https://github.com/darkocerdic/sbt-multiproject-resolving