What is a good way to organize projects with shared dependencies in Mercurial?

Question

Currently, I am moving from an outdated version control system and moving my team project to Mercury. As one example of the types of code that I am moving, I have a solution with 25+ Visual Studio projects containing several separate areas of the application that depend on the common code. Looking back at the stack overflow, the closest question I found was this one , but it just mentioned version control. I am looking for some additional tips on specific implementation methods for using Mercurial to manage these dependencies.

A simplified view of the dependencies looks something like this: (This is just for illustration and example, the actual dependencies are much more complicated, but similar in nature.)

Common Lib 1 / | \ ---- | ----- / | \ \ App 1 Common Lib 2 \ App 2 / | \ \ ------- | ------ | / | \| App 3 App 4 App 5 

Common Lib modules will be common code - it will be a DLL or SO or some other library that will be used between all applications simultaneously - both at compile time and at run time. Otherwise, the applications could work independently of each other.

I have a couple of goals with setting up my mercury repositories:

• Give each significant application or component group its own repository.
• Make each repository standalone.
• Make the total amount of the project autonomous.
• Simple creation of the entire code base at once. (in the end, all these programs and libraries fall into one installer.)
• Keep it simple.

Another point is that I have a server installed, where I have separate repositories for each of these projects.

I see a couple of ways to lay out these projects.

1. Create a Shell repository containing everything.

This will use url based subrepos (for example, in .hgsub, I would do something like App1 = https://my.server/repo/app1 .) Laid out, it would look like this:

 +---------------------------+ | Main Repository | | | +---------------------+ | | +-| Build | | | | +---------------------+ | | | +---------------------+ | | +-| Common Lib 1 | | | | +---------------------+ | | | +---------------------+ | | +-| Common Lib 2 | | | | +---------------------+ | | | +---------------------+ | | +-| App 1 | | | | +---------------------+ | | | +---------------------+ | | +-| App 2 | | | | +---------------------+ | | | +---------------------+ | | +-| App 3 | | | | +---------------------+ | | | +---------------------+ | | +-| App 4 | | | | +---------------------+ | | | +---------------------+ | | +-| App 5 | | | +---------------------+ | +---------------------------+ 

Each main folder in the shell repository will contain a subrepo, one for each area of ​​the project. Dependencies will be relative: for example, since App 4 needs Common Lib 2, it will simply use relative paths to reference the shared library.

The advantages of this approach:

• Each library is flushed once and only once.
• Mercurial confirmations guarantee that the same version of the library will be used automatically in all projects, since there is only one version of this submode in the project.
• Easy to find every resource.

The disadvantages of this approach are:

• I can’t work in the application myself. For example, if I am working on App 2 and it needs to change shared libraries, all other applications will need to make these changes right now.
• If I bring out the App repository, I must find out (or find out) what other dependent repositories he will require by hand if I want to build it.
• Dependencies are not very separated - it would be tempting to insert a new function anywhere, as it was easy to get all the functions.

2. Dependent sub-packages must be fully contained.

In this approach, each application will have its own repository (as before), but this time also contain sub-repositories: one for its own source and one for each dependent sub-repository. Then the common repository will contain each of these project repositories and knows how to build the whole solution. It will look like this:

 +-----------------------------------------------------------------------+ | Main Repository | | +--------------------+ +--------------------+ +--------------------+ | | | Build | | Common Lib 1 | | Common Lib 2 | | | +--------------------+ | | +--------------+ | | | +--------------+ | | | | +-| Lib 1 Source | | | +-| Common Lib 1 | | | | | +--------------+ | | | +--------------+ | | | | | | | +--------------+ | | | | | | +-| Lib 2 Source | | | | | | | +--------------+ | | | +--------------------+ +--------------------+ | | +--------------------+ +--------------------+ +---------------------+ | | | App 1 | | App 2 | | App 3 | | | | | +--------------+ | | | +--------------+ | | | +--------------+ | | | | +-| Common Lib 1 | | | +-| Common Lib 1 | | | +-| Common Lib 2 | | | | | | +--------------+ | | | +--------------+ | | | +--------------+ | | | | | +--------------+ | | | +--------------+ | | | +--------------+ | | | | +-| App 1 Source | | | +-| App 2 Source | | | +-| App 3 Source | | | | | +--------------+ | | +--------------+ | | +--------------+ | | | +--------------------+ +--------------------+ +---------------------+ | | +--------------------+ +--------------------+ | | | App 4 | | App 5 | | | | | +--------------+ | | | +--------------+ | | | | +-| Common Lib 2 | | | +-| Common Lib 1 | | | | | | +--------------+ | | | +--------------+ | | | | | +--------------+ | | | +--------------+ | | | | +-| App 4 Source | | | +-| Common Lib 2 | | | | | +--------------+ | | | +--------------+ | | | +--------------------+ + | +--------------+ | | | | +-| App 5 Source | | | | | +--------------+ | | | +--------------------+ | +-----------------------------------------------------------------------+ 

Pros:

• Each application can be created independently, independently of each other.
• Dependent library versions can be tracked for each application, not globally. To add a new dependency requires the explicit act of inserting subrepo into the project.

Minuses:

• In the final build, each application can use a different version of the shared library. (You may need to write tools to synchronize common lib. Eww subflows.)
• If I want to build the whole source, I end up deleting shared libraries several times. In the case of Common Lib 1, I would have to pull it out eight (!) Times.

3. Do not include dependencies at all as subrepos - add them as part of the assembly.

This approach will be similar to approach 1, except that shared libraries will be pulled out only as part of the assembly. Each application should know which repositories it needs and place them in a common place.

Pros:

• Each application can be created independently.
• Conventional libraries will only need to be pulled out once.

Minuses:

• We need to keep track of the library versions that are currently used by each application. This duplicates subrepo functions.
• We need to create the infrastructure to support this, which means that more things are needed in the script assembly. Ugh.

4. What else?

Is there any other way to handle it? Better? What methods have you tried and succeeded, what methods have you tried but hated? I'm leaning toward 1 now, but the lack of application independence, when it should be able to, is really bothering me. Is there a way to get a good separation of method 2 without massive duplication of the collage to maintain traction and code dependencies, while there is no need to write scripts to process it (as in option 3)?

Answer 1

Dependency management is an important aspect of project organization, in my opinion. You have described in detail the various solutions based on the Subrepos Mercurial function, and I agree with all the pros / cons that you gave.

I think SCMs are not suitable for dependency management. I prefer to have a dedicated tool for this (this would be your No. 3 decision).

My current project is in Java. It was created with Apache Ant , and I first configured Apache Ivy as a dependency management tool. In the end, the setup consisted of some Ivy configuration files in a shared directory and one XML file that lists the dependencies for each project module. Ivy can be triggered by Ant targets, so I added two new actions in each module: "resolve dependencies" and "expand the built-in artifact." Deployment adds the result of buid (called an artifact) to the shared directory. Dependency resolution means transitive resolution of module dependencies and copying of allowed artifacts in the "lib" folder of module sources.

This solution applies to the C ++ project, since Ivy is not specific to Java dependency management: anything can be artifacts. In C ++, artifacts created by a module will be:

• a so / dll at runtime
• header files at compile time.

This is not an ideal solution: Ivy is not easy to configure, you still need to tell your build script what dependencies to use, and you do not have direct access to the dependency sources for debugging purposes. But you have independent SCM repositories.

In our project, we then switched the Ant + Ivy form to Apache Maven , which takes care of both the structure and dependency management. Artifacts are deployed to Apache Archiva instead of a shared folder. This is a huge improvement, but it will only work well for Java projects.

Answer 2

What you want to do is each project in its own directory, as in (1). Then you mark the working versions of your dependencies and save the tag in some file for assembly, for example:

 App1 / .dependencies:
 CommonLib1 tag-20100515
 CommonLib2 tag-20100510

 App2 / .dependencies:
 CommonLib1 tag-20100510
 CommonLib2 tag-20100510

You then use build scripts to create libraries based on a specific tag and include these built-in libraries as derived objects for your applications. If build time is a problem, you can have a tagged version that is used for those libraries that were previously created and saved somewhere.

Note (design principles are the same when developing a database schema, object model, or product assembly):

• Do not link to code in other projects (encapsulation breaks)
• You do not have multiple copies of the libraries in your repository (modularity)

Answer 3

We solved a similar problem using subversion.

Each application and each Common Lib have their own repositories.

Each application has a Libs directory that contains dependent dlls.

therefore, the application receives a Common Lib update if a new dll set is provided.

however, updating the lib folder is not trivial because dependent sub-dlls must match the correct version.