Team Build is now Painly Slow

Question

We run into performance issues with our Team Foundation Build Server implementation, and I'm running out of ideas on how to speed up the process. We have already added several PropertyGroup elements to improve performance in a few steps (SkipClean, SkipLabel, SkipInitializeWorkspace), but I think we need to carry out a serious restructuring to fix the situation. Here is our setup:

• We have about 40 web applications that are very different from each other, but run away with a bunch of common assemblies
• Each of these web applications has its own solution;
• Each of these web applications has 10 to 25 common assemblies;
• There is a build definition that contains all the solutions that run at each check-in in the trunk;

And here are the main problems we face

• During the assembly, it will build each general assembly as many times as it refers to, and not build once and use for each application
• Copying a file to a drop directory is much slower. It must be through a network resource and will not accept the local path.
• Each of so many assemblies, one or more output files gets "locked" and causes the assembly to crash, even if the compilation is fine.
• And one more thing - I also tried separate build definitions, but that would also force another workspace to get the latest version. I would prefer the build server to contain one version of the assembly trunk.

Over the past few months, we have become lethargic and ignored this problem, but now the assembly time is more than an hour to an hour and a half.

I play with the idea of ​​learning and switching to Cruise Control for more control that I would have. Does anyone disagree with this?

Any help is most appreciated. Thanks!

Answer 1

So, here is what I did, and I got the build up to 9 minutes. For the number of projects that I collect, I am fine with that.

• Created a solution that contains all shared libraries and all web pages. There was a lot of extra work at this point because I had to fix a bunch of links that were outdated code or were otherwise stored in several places.
• What ultimately saved the most time was to perform a file system move rather than a network copy for all output files. Since our location is actually located on the build server, that makes sense.

To complete the transition, I simply redefine the CoreDropBuild target in the TFSBuild.proj file:

<Target Name="CoreDropBuild" Condition=" '$(SkipDropBuild)'!='true' and '$(IsDesktopBuild)'!='true' " DependsOnTargets="$(CoreDropBuildDependsOn)" > <Exec Command="move $(BinariesRoot)\Release d:\BuildOutput\$(BuildNumber)\Release"/> </Target> 

Answer 2

First, it looks like all your web applications are contained in the same Team Project. If true, divide them into logical groups. Typically, one Team Project should consist of one deployment model.

Secondly, share the general assemblies in your own Team Project. After moving, you have a couple of options, you can branch out either the source or compiled DLLs for the team projects that they need. They can have their own unit tests, and you can expand the assembly of commands to automatically merge with a successful test if you are so inclined.

To summarize, you need to simplify your build strategy.

Answer 3

Do you really need to build everything in every web application? If the general assemblies have not changed, why build them again and again?

Here is a thought:

• Let each web application have its own \ lib folder.
• Put each shared assembly in the lib folder.
• Let the web application only reference shared assemblies from its local lib folder.
• Check it all out.

Now the assembly should not start if something has not changed, and the assembly will not include general assemblies.

• There should be a central folder containing all shared assemblies.
• Any change here should apply to all local \ lib folders.
• Finally, any shared assembly should be copied to the central folder whenever they change.

The idea here is that the general build project only knows about the central folder. This will simplify any post-assembly actions necessary to copy the assembly.

The central folder must be managed so that any changes are copied to the entire web application that references it.

Answer 4

Speaking of personal experience in the CruiseControl offering, keep in mind the continuous integration of the framework. It will not solve all your problems out of the box (component assemblies, shooting for each change in components and serialized assemblies, although it will significantly improve the work). In order to get what you need, you will need a certain configuration (and possibly even customization), so be prepared to spend some time. Of course, in the long run, you will get many years of gain if your build time decreases - if you can no longer ignore the problem, it is worth spending some time on the best solution for CI.

Keep in mind that any CI effort is only as good as the policies that you have. We had huge political voids when it came to version marking, release, dependencies, beta versions of binary files, archive builds ... and many other issues that we did not even consider at that time.

Also, be prepared to devote at least some resources to maintaining things. This is not a full time job (and I like to do this, as it constantly improves the process). Our settings led us to a 2-hour monolithic assembly of our first product with over 400 components in 20 products that were built in parallel on several machines in about 20 minutes, so it’s worth it.