MSBuild - how to create multiple files and projects in parallel

I have a large Visual Studio 2010 solution file ( .sln ) that contains a large number of projects ( ~50 ). Each project consists of approximately 20 .cpp and .h files. Building an entire project takes about 2 hours on a fast Intel i7 machine with all the dependencies cached locally on a fast SSD.

I am trying to replicate an existing experiment that I found in this section , and you need to clarify how:

  • Get the concurrency I'm looking for to work in Visual Studio (with GUI / IDE).
  • Get these changes while also working with automatic command line MSBuild.exe through MSBuild.exe .

First, do the /m and /maxcpucount match? I read an article about the /m option , which seems to create more projects in parallel. This seems to be the same option that I set using the following steps in the GUI:

  • Tools
    • Options
    • Projects and solutions
    • Build and run
      • Maximum number of parallel project builds

There is another /MP option that I can find in the GUI via:

  • Projects
    • Properties
    • Configuration properties
    • C / c ++
      • General
      • Multi-processor Compilation

Please correct me if I am mistaken about this, but this, apparently, assumes that the project will create several .cpp files in parallel, but there is no way to specify how many, unless I have to manually set it in the text box (t ie: /MP 4 or /MP4 .

It also seems that in order for /MP work, I need to disable minimal rebuild ( Enabled Minimal Rebuild: No (/GM-) .: Enabled Minimal Rebuild: No (/GM-) ), and also not work with precompiled headers. I worked on releasing precompiled headers by making precompiled headers a special project that is first created before all other projects.

To the question: how can I implement these parameters for parallel projects as part of the solution, and the parallel file ( .cpp ) builds as part of the projects, using MSBuild through the command line and confirming that they work as expected? Are my assumptions correct? My ultimate goal is to verify that the project is built up to the total number of cores used, or perhaps even overload it if my build process is related to I / O binding and not CPU binding. I can do this quite easily on Linux through:

 NUMCPUS=`grep -c '^processor' /proc/cpuinfo` NUMJOBS="$(echo "$NUMCPUS*1.5" | bc)" NUMJOBS="$(printf '%.0f' $NUMJOBS)" alias pmake='time nice make -j$NUMJOBS' 

This allows me to queue 12 build tasks on my 8-core PC right away, and make automatically creates up to 12 .cpp files in parallel, processing implicit build / dependency rules, etc. m is trying to do the same in Visual Studio Professional and MSBuild. When my builds are related to I / O bindings, this approach was best while keeping all cores at ~100% .

Thanks.

+7
parallel-processing visual-studio visual-studio-2010 multiprocessing msbuild
source share
2 answers

The /MP switch for the compiler ( cl.exe ) and /m for msbuild.exe is really the way to go. Here are some relevant documentation :

The / MP switch causes the compiler to create one or more copies on its own, each in a separate process. Then these copies compile the source files at the same time. Optional argument: The maximum number of processes that the compiler can create. If you omit the processMax argument, the compiler extracts the number of efficient processors on your computer from the operating system and creates a process for each processor. The / MP switch is not compatible with some compiler options and language functions. If you use an incompatible compiler option with the / MP option, the compiler generates warning D9030 and ignores the / MP option

Examples Suppose you specify the following command line:

cl / MP7 a.cpp b.cpp c.cpp d.cpp e.cpp

In this case, the compiler uses five processes because it is less than five source files and no more than seven processes. Alternatively, suppose your computer has two efficient processors and you specify the following command line:

cl / MP a.cpp b.cpp c.cpp

In this case, the operating system reports two processors; therefore, the compiler uses two processes in its calculation. As a result, the compiler will build using two processes, since it is less than two processes and three source files.

So, you noticed that all this is related to the compiler, and not a word about msbuild, since the compiler is a stand-alone tool. In other words: suppose you open 2 command windows, and in both of them a command like cl /MP3 a.cpp b.cpp c.cpp at the same time, you must start 6 compiler processes at the same time. What behavior you are doing and which can also do with msbuild.

Exiting msbuild /? :

/ maxcpucount [: n] Specifies the maximum number of concurrent processes to build with. If the switch is not used, the default value is used: 1. If the switch is used without a value, then MSBuild will use up to the number of processors on the computer. (Short form: / m [: n])

Also more relevant information here , in particular the part that says

BuildInParallel - an optional logical parameter in the MSBuild task. If the BuildInParallel parameter is set to true (its default value), several workflows are generated to create as many projects as possible, possibly time. For proper operation, the / maxcpucount switch must be set to a value greater than 1, and the system must have at least dual core or two or more processors.

Now, if you just call msbuild /m my.vcxproj in a typical project, this will not do anything special, because there is only one project to build. However, if inside the msbuild file there is a call to the MsBuild task, for example

 <MsBuild Projects="@(ListOfProjects)" BuildInParallel="True"/> 

and the / m option is passed to msbuild, then msbuild will spawn several other msbuild processes in accordance with the rules outlined above. And coincidence is exactly what happens when building a solution with msbuild. Msbuild first converts the solution to the actual msbuild file, which invokes the MsBuild task with a list of projects in the solution. Since such several projects can be built in parallel, several msbuild processes are created for this. In turn, if these are C ++ projects with the / MP option set, each of these projects will create several compiler processes giving a maximum of <max num cl processes> * <max num msbuild instances> parallel compilations. It depends on your build, which is optimal if you want to play with the parameters (in order to do this efficiently, each of your projects will import a common property sheet in which you set the / MP parameter, which you will need to edit all of them separately) and look at the time reports that were created during the work of the collector. I always leave the arguments for / m and / MP by default, because I do not want other developers using my projects to have possible suboptimal configurations, and also because they easily allocate CPUs.

+6
source share

You can achieve this on the command line by calling:

 MSBuild /m:PARALLEL_PROJECT_COUNT /p:CL_MPCount=PARALLEL_FILE_COUNT 

The number of resulting processes will be:

MSBuild = PARALLEL_PROJECT_COUNT Cl = PARALLEL_PROJECT_COUNT * PARALLEL_FILE_COUNT

+2
source share

All Articles