How to get multiple threads to handle the same IEnumerable result?

Question

I have a method that returns an IEnumerable<string> , which of course is handled with yield return <string>; . I want several threads to handle the result of this, of course, without repeating it and not being thread safe. How can I achieve this?

 var result = GetFiles(source); for (int i = 0; i < Environment.ProcessorCount; i++) { tasks.Add(Task.Factory.StartNew(() => { ProcessCopy(result); })); } Task.WaitAll(tasks.ToArray()); 

However, this seems to produce repetitions:

 C:\Users\esac\Pictures\2000-06\DSC_1834.JPG C:\Users\esac\Pictures\2000-06\DSC_1835.JPG C:\Users\esac\Pictures\2000-06\.picasa.ini C:\Users\esac\Pictures\2000-06\DSC_1834.JPG C:\Users\esac\Pictures\2000-06\DSC_1835.JPG C:\Users\esac\Pictures\2000-06\.picasa.ini C:\Users\esac\Pictures\2000-06\DSC_1834.JPG C:\Users\esac\Pictures\2000-06\DSC_1835.JPG C:\Users\esac\Pictures\2000-06\.picasa.ini C:\Users\esac\Pictures\2000-06\DSC_1834.JPG C:\Users\esac\Pictures\2000-06\DSC_1835.JPG 

Answer 1

You can easily do this using the Parallel.ForEach method.

Write a simple Parallel.ForEach loop

Each iteration will be queued in the task manager. The loop will exit when all iterations have been completed.

 var result = GetFiles(source); Parallel.ForEach(result, current => { ProcessCopy(current); }); Console.WriteLine("Done"); 

Answer 2

You need to select a series of elements for each call to ProcessCopy() - right now you are passing each thread a complete listing of the files - remember that the IEnumerable you pass has a method called GetEnumerator() - only when this method is called (which foreach does for you under the hood) returns a real counter, with which you can list items one by one. Since you pass in IEnumerable , each thread calls GetEnumerator() and therefore lists all the files.

Instead, do something like this so that each ProcessCopy() processes a single file:

 foreach(string file in GetFiles(source)) { string fileToProcess = file; tasks.Add(Task.Factory.StartNew(() => { ProcessCopy(fileToProcess); })); } Task.WaitAll(tasks.ToArray()); 

I would not worry about the number of processors - let the TPL and the thread pool determine the number of threads for optimal performance.

Answer 3

Why not use a simple LINQ query to accomplish what you want?

 var tasks = from f in GetFiles(source) select Task.Factory.StartNew(() => { ProcessCopy(f); }); Task.WaitAll(tasks.ToArray()); 

Behind the scenes, TPL handles all the icky Environment.ProcessorCount stuff for you anyway.