How to use multithreading in a For loop

Question

I want to achieve the following requirement; suggest some solution.

string[] filenames = Directory.GetFiles("C:\Temp"); //10 files for (int i = 0; i < filenames.count; i++) { ProcessFile(filenames[i]); //it takes time to execute } 

I wanted to implement multithreading. For example, there are 10 files. I wanted to process 3 files at a time (custom, say maxthreadcount ). Thus, 3 files will be processed in 3 threads from the for loop, and if any thread completes execution, it must select the next element from the for loop. You must also ensure that all files are processed before it completes the for loop.

Please suggest a better approach.

Answer 1

This will do the job in .net 2.0:

 class Program { static int workingCounter = 0; static int workingLimit = 10; static int processedCounter = 0; static void Main(string[] args) { string[] files = Directory.GetFiles("C:\\Temp"); int checkCount = files.Length; foreach (string file in files) { //wait for free limit... while (workingCounter >= workingLimit) { Thread.Sleep(100); } workingCounter += 1; ParameterizedThreadStart pts = new ParameterizedThreadStart(ProcessFile); Thread th = new Thread(pts); th.Start(file); } //wait for all threads to complete... while (processedCounter< checkCount) { Thread.Sleep(100); } Console.WriteLine("Work completed!"); } static void ProcessFile(object file) { try { Console.WriteLine(DateTime.Now.ToString() + " recieved: " + file + " thread count is: " + workingCounter.ToString()); //make some sleep for demo... Thread.Sleep(2000); } catch (Exception ex) { //handle your exception... string exMsg = ex.Message; } finally { Interlocked.Decrement(ref workingCounter); Interlocked.Increment(ref processedCounter); } } } 

Answer 2

Try

 Parallel.For(0, filenames.Length, i => { ProcessFile(filenames[i]); }); 

MSDN

It is available only with .Net 4. Hope this is acceptable.

Answer 3

Take a look at Joe Albahari's Consumer Queue Example . This should be a good starting point for what you are trying to accomplish.

Answer 4

You can use ThreadPool .

Example:

 ThreadPool.SetMaxThreads(3, 3); for (int i = 0; i < filenames.count; i++) { ThreadPool.QueueUserWorkItem(new WaitCallback(ProcessFile), filenames[i]); } static void ProcessFile(object fileNameObj) { var fileName = (string)fileNameObj; // do your processing here. } 

---

If you use ThreadPool elsewhere in your application, then this will not be a good solution, as it will be available in your application.

You can also grab another thread pool implementation like SmartThreadPool

Answer 5

Instead of starting a thread for each file name, queue the file names and then run three threads to process them. Or, since the main thread is now free, start two threads and let the main thread work on it too:

 Queue<string> MyQueue; void MyProc() { string[] filenames = Directory.GetFiles(...); MyQueue = new Queue(filenames); // start two threads Thread t1 = new Thread((ThreadStart)ProcessQueue); Thread t2 = new Thread((ThreadStart)ProcessQueue); t1.Start(); t2.Start(); // main thread processes the queue, too! ProcessQueue(); // wait for threads to complete t1.Join(); t2.Join(); } private object queueLock = new object(); void ProcessQueue() { while (true) { string s; lock (queueLock) { if (MyQueue.Count == 0) { // queue is empty return; } s = MyQueue.Dequeue(); } ProcessFile(s); } } 

Another option is to use a semaphore to control how many threads work:

 Semaphore MySem = new Semaphore(3, 3); void MyProc() { string[] filenames = Directory.GetFiles(...); foreach (string s in filenames) { mySem.WaitOne(); ThreadPool.QueueUserWorkItem(ProcessFile, s); } // wait for all threads to finish int count = 0; while (count < 3) { mySem.WaitOne(); ++count; } } void ProcessFile(object state) { string fname = (string)state; // do whatever mySem.Release(); // release so another thread can start } 

The first will work slightly better, because you do not have the overhead of starting and stopping the stream for each processed file name. The second, however, is much shorter and cleaner, and makes full use of the thread pool. You probably won't notice the difference in performance.

Answer 6

Can set maximum threads without ParallelOptions parameters

Parallel method. (Int32, Int32, ParallelOptions, Action)

ParallelOptions.MaxDegreeOfParallelism

Answer 7

 var results = filenames.ToArray().AsParallel().Select(filename=>ProcessFile(filename)).ToArray(); bool ProcessFile(object fileNameObj) { var fileName = (string)fileNameObj; // do your processing here. return true; }