After doing HttpWebRequests for some time, the result ends

Question

I have an application that provides websites for information. It looks like 20-45 minutes after creating the HttpWebRequests, a bunch of returned timeouts. One thing we do is attach an anonymous BindIPDelegate function to give the request a specific IP address, since we go around 150 IP addresses.

I am setting up an HttpWebRequest object with the following settings.

• User Agent Setup
• Set Keep-Alive to false so that the IP address is not reused
• Set TimeOut to 60,000 (60 seconds)
• Set ReadWriteTimeout to 60,000 (60 seconds)
• Setting proxy server to null
• Set Accept To /
• Configure CookieContainer on the new CookieContainer
• Piplined to true setting
• Configure automatic decompression for deflation and gzip

The application uses .NET 4.0 and runs on Windows Server 2008 R2.

This definitely looks like something to the /TCP/.NET application, because if I restart the application it works fine again. It also seems more or less similar to the fact that the timeout is just in line, waiting for a local port or something like that.

Any ideas?

Answer 1

You do not say much about the code that you actually use to execute queries, but, in any case, here are my guesses:

• You use BeginGetResponse()/EndGetResponse() with a callback, and the callback takes too much time (or blocks!). This can lead to a deadlock in threadpool if you issue a lot of requests in a short amount of time.

• Since you do not use reuse of connections, and, again, if requests are executed very quickly and without interruptions, you may run out of sockets (last time I tried ~ 3k to the interface in windows). If setting KeepAlive to true fixes your problem, this is it.

• You do not call Dispose()/Close() in the HttpWebRequest or HttpWebResponse or the stream you receive from the response. This might work a bit until you reach limit 2 (from MSDN docs) or 6 (default configuration file) in the application’s configuration settings for (system.net/connectionManagement/add[address= "*", maxconnection = "6"] ) An easy way to check if this is a problem is to set the limit to 1 and see if the problem happens earlier than before.

Btw, setting KeepAlive to false and Pipelined to true, makes no sense.

Answer 2

I would suggest that this is due to ThreadPool issues.

Answer 3

Could this be IDS at the remote end, thinking you are an attacker and blocking you?

Answer 4

I assume that perhaps not all objects are located correctly, and some TCP ports remain open. Try to see which objects implement IDisposable. At the very least, the result of GetResponse and GetResponseStream are IDisposables and should be removed correctly.

Answer 5

it’s easier to show an example of what I had in mind in the comments, not my own work, but the guys from microsoft are doing such a nice job that I pass you the link.

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.begingetrequeststream.aspx

If you are doing heavy data entry via http, I always suggest taking a look at the callback mechanisms.

Also make sure you close these damned httpWebRequest objects. Wrap everything in bubble wrap using liberal “using” statements.

multithreaded operations: the default connection is established by default for a single host. This parameter can be changed. If the maximum number of connections is used, then the HttpWebRequest (request / response) operations will be queued until a connection slot is available.

The article I came up with when referring to web services can also affect your problem, as the reasons are very similar, heres link:

http://support.microsoft.com/kb/821268

Answer 6

Try adding the following to your app.config under the configuration tag. I think this solved a similar problem that I encountered when repeating http connections many times:

  <system.net> <defaultProxy enabled="false"> </defaultProxy> <connectionManagement> <remove address="*"/> <add address="*" maxconnection="1000" /> </connectionManagement> </system.net> 

Edit: I think the defaultProxy tag was really a critical tag.