Scaling multiple HttpWebRequests?

Question

I am creating a server application that must continuously execute many HTTP requests to a couple of other servers. Currently, I basically configure about 30 threads and run HttpWebRequests continuously in each thread, providing a throughput of about 30 requests per second.

I really install ServicePoint ConnectionLimit in app.config so that is not a limiting factor.

I need to dramatically scale it. At least I need even more processor power, but I wonder if I will get any benefits using the asynchronous methods of the HttpWebRequest object (for example: .BeginGetResponse ()), as opposed to creating threads and using synchronous methods (for example :. GetResponse ()) for these threads.

If I switch using async methods, I obviously have to significantly change my application, so I wonder if anyone can comment before going and transcoding everything, in case I go out for lunch.

Thanks!

Answer 1

If you are on Windows NT, the System.Net.Sockets.Socket class always uses I / O ports for async operations. And HTTPWebRequest in asynchronous mode uses asynchronous sockets and therefore will use IOCP.

Without detailed benchmarking, it’s hard to say whether our bottleneck is inside the HttpWebRequest or up the stack in your application or on the remote side of the server. But of course, asyncc will certainly give you better performance, because it will eventually use IOCP under covers. And redefining the application for async is not so difficult.

So, I would suggest changing the application architecture to asynchronous first. Then see how much maximum bandwidth you get. Then you can start benchmarking and find out where the bottleneck is, and remove it.

Answer 2

The fastest result for me is using 75 threads performing httpwebrequest synchronization. About 140 requests per second on a Windows 2003 server, connection 4core 3ghz, 100mb.

Async Httprequest / winsock stuck around 30-50 req / sec. Did not test winsock synchronization, but I assume that this will give you the same result as httpwebrequest.

Tests were against 1,200,000 blogs.

The last month struggled with this, so it would be interesting to know if someone managed to squeeze more out of .net?

EDIT

New test: it turned out 350req / sec with the xfserver iocp component. Used a bunch of threads with one instance each before any big result. There were some really annoying bugs in the "client side" of the lib that complicated the implementation, and then the "server side". Not what you ask for and not recommended, but some kind of step.

Next: the previous winsock test did not use 3.5 SocketAsyncEventArgs, which will be next.

ANSWER

The answer to your question is no, it will not be worth the effort. The async HttpWebRequest methods unload the main stream when stored in the background, this does not improve the number / scalability of requests. (at least not in 3.5, maybe different in 4.0?)

However, you should pay attention to creating your own shell around asynchronous sockets / SocketAsyncEventArgs, where iocp works, and, possibly, implement a start / end template similar to HttpWebRequest (for the simplest implementation in the current code). The improvement is really huge.