Processing parallel elements in parallel

Question

I have an ItemGroup and I want to process all its elements in parallel (using a custom job or .exe).

• I could write my / exe task to accept the entire ItemGroup and process its elements in parallel. However, I want this concurrency to work in conjunction with the MSBuild parameter /maxCpuCount, as otherwise I could end up /maxCpuCount.
• This topic says that there is no way.
• My tests show that MSBuild /maxCpuCountonly works for creating various projects, not elements (see code below)

How can I process items from ItemGroup in parallel?
Is there a way to create a custom parallel task in combination with MSBuild Parallel support?

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Target Name="Build" >
    <!-- Runs only once - I guess MSBuild detects it the same project -->
    <!--<MSBuild Projects="$(MSBuildProjectFullPath);$(MSBuildProjectFullPath)" Targets="Wait3000" BuildInParallel="true" />-->

    <!-- Runs in parallel!. Note that b.targets is a copy of the original a.targets -->
    <MSBuild Projects="$(MSBuildProjectFullPath);b.targets" Targets="Wait3000" BuildInParallel="true" />

    <!-- Runs sequentially -->
    <ItemGroup>
      <Waits Include="3000;2000"/>
    </ItemGroup>
    <Wait DurationMs="%(Waits.Identity)" />
  </Target>

  <Target Name="Wait3000">
    <Wait DurationMs="3000" />
  </Target>

  <UsingTask TaskName="Wait" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll" >
    <ParameterGroup>
      <DurationMs ParameterType="System.Int32" Required="true" />
    </ParameterGroup>
    <Task>
      <Code Type="Fragment" Language="cs">
        Log.LogMessage(string.Format("{0:HH\\:mm\\:ss\\:fff}  Start  DurationMs={1}", DateTime.Now, DurationMs), MessageImportance.High);
        System.Threading.Thread.Sleep(DurationMs);
        Log.LogMessage(string.Format("{0:HH\\:mm\\:ss\\:fff}  End    DurationMs={1}", DateTime.Now, DurationMs), MessageImportance.High);
      </Code>
    </Task>
  </UsingTask>
</Project>   

Answer 1

I know this is old, but if you have a few minutes, try to use the task again MSBuild. Using the metadata elements of the reserved element Propertiesand / or AdditionalProperties* will solve the problem described in the sample code ("It starts only once - I suppose MSBuild will detect the same project").

MSBuild ItemGroup MSBuild ( /maxCpuCount). BuildTargetsInParallel MSBuild .

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <Target Name="Build" >
    <ItemGroup>
      <Waits Include="3000;2000"/>
    </ItemGroup>

    <ItemGroup>
      <ProjectItems Include="$(MSBuildProjectFullPath)">
        <Properties>
          WaitMs=%(Waits.Identity)
        </Properties>
      </ProjectItems>
    </ItemGroup>
    <MSBuild Projects="@(ProjectItems)" Targets="WaitSpecifiedMs" BuildInParallel="true" />
  </Target>

  <Target Name="WaitSpecifiedMs">
    <Wait DurationMs="$(WaitMs)" />
  </Target>

</Project>

* " " MSBuild.

Answer 2

, , .

TPL, ForEach.

public bool ForEach<T>(IEnumerable<T> enumerable, Action<T> action, int max = -1)
{
    return enumerable != null && Parallel.ForEach(enumerable, new ParallelOptions { MaxDegreeOfParallelism = max }, (e, s) =>
    {
        if (Canceled)
            s.Stop();
        if (s.ShouldExitCurrentIteration)
            return;
        action(e);
        Interlocked.Increment(ref _total);
    }).IsCompleted;
}

, .NET, , , , , MSDeploy, SSRS DoS 20 IP- zip, , , 1. , maxCpuCount Environment.ProcessorCount %NUMBER_OF_PROCESSORS%, -, , .. .

private void Engine(object host)
{
    var type = host.GetType();
    if (type.FullName != "Microsoft.Build.BackEnd.TaskHost")
    {
        Log.Warn("[Host] {0}", type.AssemblyQualifiedName);
        return;
    }

    var flags = BindingFlags.NonPublic | BindingFlags.Instance;
    var taskLoggingContext = type.GetProperty("LoggingContext", flags).GetValue(host, null);
    var targetLoggingContext = taskLoggingContext.GetType().GetProperty("TargetLoggingContext", flags).GetValue(taskLoggingContext, null);

    ProjectTask = taskLoggingContext.GetType().GetProperty("Task", flags).GetValue(taskLoggingContext, null).To<ProjectTaskInstance>();
    ProjectTarget = targetLoggingContext.GetType().GetProperty("Target", flags).GetValue(targetLoggingContext, null).To<ProjectTargetInstance>();

    var entry = type.GetField("requestEntry", flags).GetValue(host);
    var config = entry.GetType().GetProperty("RequestConfiguration").GetValue(entry, null);

    Project = config.GetType().GetProperty("Project").GetValue(config, null).To<ProjectInstance>();
    Properties = Project.Properties.ToDictionary(p => p.Name, p => p.EvaluatedValue);

: ForEach:

public class Transform : Task
{
    [Required]
    public ITaskItem[] Configs { get; set; }

    protected override void Exec()
    {
        //...

        ForEach(Configs, i =>
        {
            //...
        }, Environment.ProcessorCount);

        //...
    }