Parallel compilation of delphi projects through MSBuild

I have a script that will compile all projects (about 50) of my solution, for example, the following

msbuild "myProjName.dproj" /t:build /p:config="Release" /fileLogger /flp:ErrorsOnly /nologo 

This works fine, but compiles forever. To speed up the build process, I tried to use the full potential of our modern multi-core machines using the '/ maxcpucount' switch described here: http://msdn.microsoft.com/library/bb651793.aspx

I get about the same compilation time on my 4 core dev processor. There is no championship.

Apparently, this can only work when projects need to create dependencies. Other “workers” will then build these dependency projects in parallel as the main project. So I tried to create a project group in delphi and add all my projects to it, and then run the msbuild command for that .groupproj, but it is still as slow as ever.

Has any of you reached to create several projects at the same time with msbuild? If so, can you give me an explanation?

Thanks!

+8
delphi msbuild delphi-xe6
source share
3 answers

RAD Studio XE4 applies the following rule, but it can also apply to earlier or later versions. Also, dependencies defined in .groupproj will not be respected by this method. There were no cross-project dependencies in .groupproj, which I tried to parallelize, so I did not understand how to do this.

When you create a .groupproj file with MSBuild using a Build , Clean or Make target, the assembly does not start in parallel because these targets use the CallTarget task to accomplish other goals, but CallTarget does not execute its goals in parallel.

To build separate projects in parallel, an MSBuild project must use one MSBuild task to create multiple projects at the same time. Objectives should be defined as follows:

  <Target Name="Build"> <MSBuild Projects="@(Projects)" BuildInParallel="true"/> </Target> <Target Name="Clean"> <MSBuild Targets="Clean" Projects="@(Projects)" BuildInParallel="true"/> </Target> <Target Name="Make"> <MSBuild Targets="Make" Projects="@(Projects)" BuildInParallel="true"/> </Target> 

Add them to .groupproj, then remove the other <Target> directives as well as the <Import> directive. ( CodeGear.Group.Targets defines some goals for building projects in the correct order and for creating dependencies when you request to create only a subset of projects, but it overrides the Build , Clean and Make goals defined in .groupproj.) Note that this allows Create only all projects, not just a subset.

BuildInParallel was added in MSBuild 3.5. However, since .groupproj files do not specify the ToolsVersion attribute, MSBuild will use the MSBuild Task, as defined in version 2.0, which does not support BuildInParallel . There are two ways to fix this:

  • Add ToolsVersion="3.5" (or later) to the root <Project> element of your .groupproj file.
  • Launch MSBuild with the /toolsversion:3.5 command line /toolsversion:3.5 (or /tv:3.5 for short) ( /toolsversion overrides ToolsVersion specified in all project files.)

After that, start MSBuild with /maxcpucount (or /m ), and your projects should build in parallel. However, RAD Studio does not handle the project group correctly, so you can provide the file with a different extension so that it makes it clear that this is not a standard RAD Studio project group (any extension that ends with proj ).

The following XSLT stylesheet performs the conversion described above:

 <?xml version="1.0" encoding="utf-8"?> <xsl:stylesheet version="1.0" exclude-result-prefixes="msbuild" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" xmlns:msbuild="http://schemas.microsoft.com/developer/msbuild/2003" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:output method="xml" indent="yes"/> <xsl:template match="//msbuild:Project"> <xsl:copy> <xsl:attribute name="ToolsVersion">3.5</xsl:attribute> <xsl:apply-templates select="@* | node()"/> <Target Name="Build"> <MSBuild Projects="@(Projects)" BuildInParallel="true"/> </Target> <Target Name="Clean"> <MSBuild Targets="Clean" Projects="@(Projects)" BuildInParallel="true"/> </Target> <Target Name="Make"> <MSBuild Targets="Make" Projects="@(Projects)" BuildInParallel="true"/> </Target> </xsl:copy> </xsl:template> <xsl:template match="//msbuild:Target"> <!-- Do not copy --> </xsl:template> <xsl:template match="//msbuild:Import"> <!-- Do not copy --> </xsl:template> <xsl:template match="@* | node()"> <xsl:copy> <xsl:apply-templates select="@* | node()"/> </xsl:copy> </xsl:template> </xsl:stylesheet> 

You can apply this stylesheet with MSBuild (4.0 or later: XslTransformation was added to MSBuild 4.0) using this project file (where groupproj2parallel.xslt is the XSLT file above):

 <?xml version="1.0" encoding="utf-8"?> <Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> <Target Name="Build" Inputs="$(InputPaths)" Outputs="$(OutputPaths)"> <XslTransformation XmlInputPaths="$(InputPaths)" XslInputPath="groupproj2parallel.xslt" OutputPaths="$(OutputPaths)" /> </Target> </Project> 

You need to explicitly specify InputPaths and OutputPaths on the command line using /p:InputPaths="..." /p:OutputPaths="..." or by specifying them in the Properties parameter of MSBuild . (Alternatively, you can simply copy the file names to the project file.)


The goal definitions provided by MSBuild projects for C # and Visual Basic handle dependencies using the <ProjectReference> elements defined in the project files instead of defining the dependencies in the solution file. Delphi.dproj files and C ++ Builder.cbproj files do not support this, because the underlying CodeGear.Common.Targets does not reuse the mechanism defined in Microsoft.Common.Targets for <ProjectReference> .

+5
source share

There are two ways to create Delphi projects: MSBuild or DCC32.exe . MSBuild recommends encapsulating all configuration parameters as project files ( dproj and groupproj ).

However, there are additional ones overhead that use MSBuild compared to the usual old DCC32.exe . In addition, using MSBuild to create a Delphi Project Group ( .groupproj ) does not bring any benefits for multi-core CPUs. Build performance is the same as single-core CPU.

Here are my statistics for creating 290 dproj files in one groupproj :

 MSBuild a `groupproj` contains 290 `dproj` on 2C/4T CPU: ~100s MSBuild a `groupproj` contains 290 `dproj` on 4C/8T CPU: ~100s MSBuild 290 `dproj` run in multi-threads on 2C/4T CPU: ~121s MSBuild 290 `dproj` run in multi-threads on 4C/8T CPU: ~50s DCC 290 `dproj` run in multi-threads on 2C/4T CPU: ~37s DCC 290 `dproj` run in multi-threads on 4C/8T CPU: ~24s 

From the readings, it can be concluded that MSBuild introduces additional overhead compared to DCC32 . To take full advantage of the available cores and CPU threads, DCC32 is the path to sacrifice a user-friendly design encapsulation design for .DPROJ .

A MSBuild script to create Delphi groupproj in parallel is available at https://github.com/ccy/msbuild.delphi.parallel

+2
source share

A little off topic: you can try the fastdcc part of the IDE fix pack to get faster builds: http://andy.jgknet.de/blog/ide-tools/ide-fix-pack/ For example, I have build time 1 minute to 22 seconds!

0
source share

All Articles