TPL Parallel.For with lengthy tasks

Question

I want to use the parallel task library (TPL) in F # to perform many (> 1000) lengthy tasks. Here is my current code:

Parallel.For(1, numberOfSets, fun j -> //Long running task here ) 

When I start this, it seems that .NET launches all tasks at once and constantly bounces between them. What would be better if he stayed on the task until it was done before moving on to the next. This minimizes context switching.

Is there a way to provide a hint to the scheduler? I know you can give hints, but I can’t find clear examples or the planner already knows about it, and it’s just my opinion that there are too many context switches. Thanks for the help!

Answer 1

We had a similar problem - using C # instead of F #, but the libraries are the same. The solution was to limit the degree of parallelism:

 ParallelOptions parallelOptions = new ParallelOptions(); parallelOptions.MaxDegreeOfParallelism = 16; Parallel.For(0, n, parallelOptions, i => { . . . }); 

16 worked well for our tasks - you should experiment to find out which value is better in your case.

Answer 2

In my experience, for a large number of tasks it is better to bind MaxDegreeOfParallelism to Environment.ProcessorCount .

Here is a similar code snippet for @Mimo in F # syntax:

 let options = ParallelOptions() options.MaxDegreeOfParallelism <- Environment.ProcessorCount * 2 Parallel.For(0, n, options, (fun i -> (* Long running task here *))) |> ignore 

Since you work with parallel programming in F #, please take a look at the excellent book “Parallel Programming Using Microsoft.NET” , in particular the chapter “Parallel Loops” . @Tomas translated his samples into F # and they are available here .

Answer 3

Looking at the source, the following code fragment appears: number of workers:

 // initialize ranges with passed in loop arguments and expected number of workers int numExpectedWorkers = (parallelOptions.EffectiveMaxConcurrencyLevel == -1) ? Environment.ProcessorCount : parallelOptions.EffectiveMaxConcurrencyLevel; 

As far as I can tell, with the default task scheduler and the default ParallelOptions parameter, this is the value of Environment.ProcessorCount , so it is strange that you get a different behavior by pointing MaxDegreeOfParallelism to the processor counter yourself. I suggest you debug it to make sure that there really is a difference (you can print Thread.ManagedThreadId in the long run of the task).