Slow complex assemblies and hudson vs electric cloud

Question

Is hudson the right tool for complex C ++ builds?

I have a C ++ build that takes about 4 hours. Compilation and packaging take about 1/2 time, and testing consumes the other half. We currently use the home system, but some of them go to hudson, as we use it for all our java builders.

My problem is that continuous integration is not very ... continuous at 4-hour intervals. I want a tool that allows me to parallelize the assembly in an understandable way.

Hudson did a great job with small builds or java builds where I sit on top of a large maven project, but I don't think it will scale well for complex C ++ builds.

What were your experiences?

Answer 1

It looks like you have a few questions here:

• Should I use a CI server to control my build in C ++? The answer is definitely YES. Your homegrown system may be adequate, but it is not standard, its distribution is probably difficult, and its maintenance distracts from the work that you really pay.
• Is Hudson the right choice for my project? He will probably do his job, and he has the advantage that you are already on your site. However, you specifically point out that you need a tool that supports parallelization well, and I don't think Hudson really matches the bill. The problem is that Hudson was not designed with parallelism in mind. See, Presenting the build process in Hudson is a “task” that is just a few steps that are performed in sequence — validation, compilation, testing, package, etc. It is not possible to run these steps in parallel. Now you can get around this by simulating your process with several tasks. Each work is completely independent, therefore, of course, they can be launched in parallel; you can use something like a Locks and Latches plugin to coordinate tasks, but the whole process is more complicated than it should be, and kind of clumsy - instead of a single task representing one launch of the build process, you have several unrelated tasks, at best related with each other through a naming convention.
• Can Electric Cloud Help? Again, an explicit YES. Electric Cloud offers ElectricCommander, a CI server with parallel support built from the start. As with Hudson, the task is used to represent the assembly process, but the steps in the task can be easily started in parallel (just check the "parallel" checkbox at these stages), so you do not need to resort to adding, ons and kludges: one start of the assembly process - This is one task with as many parallel steps as you like.
• Will the correct CI server "come back" back to my integration? The CI server will bring you so far. The fact is that the CI server can provide you with coarse-grained parallelism - so with a little work you can configure it to run the packaging in parallel with the tests, for example. With a bit more work, you can split your test phase into several independent parts that can be run in parallel.
  You did not specify a lot of details, but assume that your assembly is 90 minutes of compilation, 30 minutes of packaging, and 2 hours of tests, which can be divided into four 30-minute fragments. Suppose you can do packaging and testing at the same time. This would bring your 4-hour process up to 2 hours. At this point, the “long pole” in your process is the compilation phase, and although you can break it down manually into parts that can run in parallel with your CI server, the truth is that the CI server is simply not the right tool for this job.
  the best option is to use a build tool that can give you automatic fine-grained parallelism at compile time. For example, if you are already using gmake, you can try gmake -j 8 to run 8 compilations immediately. If your makefiles are clean and your dependencies are correct, and you have a powerful build server, this can give you a pretty good performance boost. You can also use ElectricAccelerator, another product from Electric Cloud, specifically designed to speed up this part of the build process, even for builds that cannot use gmake -j due to incorrect or incomplete dependencies.

Hope this helps.

Answer 2

Can you break the assembly into several parts?

You note that there are several separate parts to the assignment. A general guide with Hudson is to do part of the assembly in one job, test in another, pack in another, etc.

You can compile the code in Job A and archive the output, and then tell Job B to copy these artifacts from Job A and run tests on them. Meanwhile, another task A can be started, due to further fixing in the original repository.

Answer 3

It seems to me that the problem is with the build process (make files ?, msbuild?), And not with Hudson. Hudson is just about to complete the build process just like a user from the command line. Is it possible to optimize the build process?

Even if a 4-hour build process is inevitable, Hudson can help because you can connect an unlimited number of slave machines that can run multiple assemblies in parallel, given sufficient hardware power.