Consider using Reactive Extensions backport for .NET 3.5. This will allow you to use the parallel task library (TPL). If this is not an option, continue reading.
I would immediately throw your entire site into ThreadPool and see how it works before trying to throttle work items.
If you feel that you need to throttle the number of simultaneous work items in ThreadPool , then you can use the semaphore to throttle them. However, be careful not to block the ThreadPool thread, as this is considered bad practice. Instead, block the queue flow.
int pending = sites.Count; var finished = new ManualResetEvent(false); var semaphore = new Semaphore(5, 5); foreach (string site in sites) { semaphore.WaitOne(); ThreadPool.QueueUserWorkItem( (state) => { try { // Process your work item here. } finally { semaphore.Release(); if (Interlocked.Decrement(ref pending) == 0) { finished.Set(); // This is the last work item. } } }, null); } finished.WaitOne(); // Wait for all work items to complete.
source share