Is it possible to execute some asynchronous / waiting inside some .NET code Parallel.ForEach ()?

Question

Given the following code, is it ok to do async/await inside a Parallel.ForEach ?

eg.

 Parallel.ForEach(names, name => { // Do some stuff... var foo = await GetStuffFrom3rdPartyAsync(name); // Do some more stuff, with the foo. }); 

or are there any keys that I need to know about?

EDIT: No idea if this compiles, by the way. Just a pseduo code .. thinking out loud.

Answer 1

No, it makes no sense to combine async with Paralell.Foreach .

Consider the following example:

 private void DoSomething() { var names = Enumerable.Range(0,10).Select(x=> "Somename" + x); Parallel.ForEach(names, async(name) => { await Task.Delay(1000); Console.WriteLine("Name {0} completed",name); }); Console.WriteLine("Parallel ForEach completed"); } 

What result do you expect?

 Name Somename3 completed Name Somename8 completed Name Somename4 completed ... Parallel ForEach completed 

This is not what will happen. He will output:

 Parallel ForEach completed Name Somename3 completed Name Somename8 completed Name Somename4 completed ... 

Why? Because when ForEach hits await first, the method really returns, Parallel.ForEach does not know that it is asynchronous and it has reached completion !. The code after await is executed as a continuation in another thread, not a "Paralell processing thread"

Stephen Tube addressed this here

Answer 2

The closest alternative could be the following:

 static void ForEach<T>(IEnumerable<T> data, Func<T, Task> func) { var tasks = data.Select(item => Task.Run(() => func(item))); Task.WaitAll(tasks.ToArray()); } // ... ForEach(names, name => GetStuffFrom3rdPartyAsync(name)); 

Ideally, you should not use a blocking call like Task.WaitAll if you can make a whole chain of methods with async calls, "everything is okay down" in the current call stack:

 var tasks = data.Select(item => Task.Run(() => func(item))); await Task.WhenAll(tasks.ToArray()); 

Also, if you are not doing CPU work inside GetStuffFrom3rdPartyAsync , Task.Run may be redundant:

 var tasks = data.Select(item => func(item)); 

Answer 3

From the name, I assume that GetStuffFrom3rdPartyAsync is I / O bound. The Parallel class is designed specifically for processor-bound code.

In an asynchronous world, you can run several tasks, and then (asynchronously) wait for all of them to complete using Task.WhenAll . Since you start with a sequence, it is easiest to project each element onto an asynchronous operation, and then wait for all these operations:

 await Task.WhenAll(names.Select(async name => { // Do some stuff... var foo = await GetStuffFrom3rdPartyAsync(name); // Do some more stuff, with the foo. })); 

Answer 4

As @Sriram Sakthivel pointed out, there are some problems using Parallel.ForEach with asynchronous lambdas. Stephen Tub ForEachASync can do the equivalent. He talks about it here , but here is the code:

 public static class Extensions { public static Task ForEachAsync<T>(this IEnumerable<T> source, int dop, Func<T, Task> body) { return Task.WhenAll( from partition in Partitioner.Create(source).GetPartitions(dop) select Task.Run(async delegate { using (partition) while (partition.MoveNext()) await body(partition.Current); })); } } 

It uses the Partitioner class to create a load balancer ( doco ) and allows you to specify how many threads you want to run with the dop parameter. to see the difference between it and Parallel.ForEach . Try the following code.

  class Program { public static async Task GetStuffParallelForEach() { var data = Enumerable.Range(1, 10); Parallel.ForEach(data, async i => { await Task.Delay(1000 * i); Console.WriteLine(i); }); } public static async Task GetStuffForEachAsync() { var data = Enumerable.Range(1, 10); await data.ForEachAsync(5, async i => { await Task.Delay(1000 * i); Console.WriteLine(i); }); } static void Main(string[] args) { //GetStuffParallelForEach().Wait(); // Finished printed before work is complete GetStuffForEachAsync().Wait(); // Finished printed after all work is done Console.WriteLine("Finished"); Console.ReadLine(); } 

if you run GetStuffForEachAsync , the program waits for the completion of all work. If you run GetStuffParallelForEach , the Finished line will be printed before completion.