I am trying to implement the .NET 4 class helper / utility, which should retrieve HTML page sources based on a list of URLs for a web testing tool. The solution must be scalable and have high performance.
I have been researching and trying different solutions for many days, but I can’t find the right solution.
Based on my understanding, the best way to achieve my goal would be to use asynchronous web requests running in parallel using TPL.
To have full control over headers, etc., I use HttpWebResponse instead of WebClient, which wraps HttpWebResponse. In some cases, the output should be tied to other tasks, so using TPL tasks may make sense.
What I have so far achieved after many different tests / approaches,
The main synchronous, asynchronous (APM) and parallel (using TPL tasks) solutions are implemented to see the level of performance of various solutions.
To see the performance of an asynchronous parallel solution, I used the APM, BeginGetResponse and BeginRead approach and ran it in Parallel.ForEach. Everything works fine and I am pleased with the performance. Somehow I feel that using a simple Parallel.ForEach is not the way to go, and for example, I don't know how to use the task chain.
, APM-, TaskCompletionSource APM. , , , , - 6-10, 2-3 500 URL-.
, , . , , 2-3 , . , , , / , (6-8 ) , , .
:
IEnumerable<Task> DoExample(string input)
{
var aResult = DoAAsync(input);
yield return aResult;
var bResult = DoBAsync(aResult.Result);
yield return bResult;
var cResult = DoCAsync(bResult.Result);
yield return cResult;
…
}
Task t = Iterate(DoExample("42"));
, System.Net.ServicePointManager.DefaultConnectionLimit , ThreadPool.RegisterWaitForSingleObject
, helper/utility html-, :
, APM, TaskCompletionSource , , , .
# Windows, , , , -, , .
, .