Getting the "underlying connection has been closed" in HttpWebRequest

Question

I have an application written in VB.NET ( NOT asp.net, this is a Windows Console application). I am trying to call a url (html page) and return the response in a string. The answer is direct JSON, no html tags. It opens with { and closes with } .

I am creating an HttpWebRequest object in order. Then call req.GetResponse() . As soon as I do this, I get the error The underlying connection was closed: The connection was closed unexpectedly. I searched for search queries and checked stackoverflow, and tried everything I found (which is very important for WCF service configurations that don't apply).

Here is my code:

 Public Function GetJSON(ByRef d As db.Device) As Boolean Try d.Url = "http://" & d.IpAddress & ini.doc.<svc>.<url>.Value Dim req As HttpWebRequest = HttpWebRequest.Create(d.Url) // req.Accept = "*/*" // req.Timeout = 30000 // req.ReadWriteTimeout = 30000 // req.KeepAlive = False // req.UseDefaultCredentials = True // req.CachePolicy = HttpWebRequest.DefaultCachePolicy // req.Proxy = HttpWebRequest.DefaultWebProxy // req.ProtocolVersion = New System.Version(1, 0) Dim rsp As HttpWebResponse = req.GetResponse() Return True Catch ex As Exception Log(ex.Message) Return False Finally rsp = Nothing req = Nothing End Try End Function 

Commented lines (incorrect comment style, but SO will figure it out correctly) is all that I have tried so far, based on what I found on the Internet. None of them fixed it. I have validated the constructed URL; if I give the exact URL in my browser, it will return exactly the correct expected answer.

I tried scrolling it ... and I see the actual expected, full data is returned in the shark's output, and then a few lines, and then a red line that says: http > 51943 [RST] Seq=1607 Win=0 Len=0 , which is The last line displayed before .NET throws an error.

I also tried to enable System.Net trace / logging on a message here on SO, and in the output file, which I see similarly, all expected JSON data is returned, but after it returns, it throws these lines into the .NET trace log :

 System.Net.Sockets Verbose: 0 : [7040] Exiting Socket#60467532::Receive() -> 1605#1605 System.Net.Sockets Verbose: 0 : [7040] Socket#60467532::Receive() System.Net.Sockets Verbose: 0 : [7040] Data from Socket#60467532::Receive System.Net.Sockets Verbose: 0 : [7040] 00000000 : : System.Net.Sockets Verbose: 0 : [7040] Exiting Socket#60467532::Receive() -> 0#0 System.Net.Sockets Verbose: 0 : [7040] Socket#60467532::Dispose() System.Net Error: 0 : [7040] Exception in the HttpWebRequest#27806816:: - The underlying connection was closed: The connection was closed unexpectedly. System.Net Error: 0 : [7040] Exception in the HttpWebRequest#27806816::GetResponse - The underlying connection was closed: The connection was closed unexpectedly. 

Any ideas where to go to try and figure it out? We read this data from some sensors for environmental monitoring, and they gave us this url for use.

Two things that really scare and confuse me about this are that a) it works fine when called in a browser
b) WireShark and .NET tracing shows that all actual data is indeed IS returned, and the structure excludes AFTER receiving all the data for some reason!

WebException itself is very little used because its InnerException is null and its status just says "ConnectionClosed {8}"

Thanks in advance!

UPDATE 08/18 1130: I also tried to use only System.Net.WebRequest , not HttpWebRequest . It also made no difference.

UPDATE 08/18 1222: I just tried switching my code instead of using [Http]Web[Request|Response] to darken the WebClient object and using its DownloadString() method. This, however, also causes the same exact error.

UPDATE 08/18 1230: Tried to use My.Computer.Network.DownloadFile() - also gets the same closed connection error.

Answer 1

Can you post all the contents of the trace log to pastebin.com and post the link here?

You can get this exception because the server can say in the "Content-Length" header that it sends N bytes of the entity, but actually sends less than N bytes and closes the connection.

Answer:

Thanks for the data. From the tracelog and wirehark traces, it seems that the server is not sending response headers and is sending data directly. This is a violation of the HTTP protocol. This is why the client throws an exception.

Answer 2

Here's how I earned it ... Thanks to feroz, to point me in the right direction!
(marked points)

 Public Function GetJSON(ByRef d As db.Device) As Boolean Try Dim tcp = New TcpClient() tcp.Connect(d.IpAddress, 80) Dim ns = tcp.GetStream() Dim req As Byte() = System.Text.Encoding.ASCII.GetBytes( "GET /getdata.htm HTTP/1.1" & vbCrLf & vbCrLf ) ns.Write(req, 0, req.Length) Dim rsp(2048) As Byte, rcv As Integer Do rcv = ns.Read(rsp, 0, rsp.Length) d.JSON &= System.Text.Encoding.ASCII.GetString(rsp, 0, rcv) Loop Until rcv = 0 tcp.Close() Return True Catch ex As Exception Log(ex.Message) Return False End Try End Function 

Answer 3

It helps me. Hope this helps someone too.

Immediately after creating the HttpWebRequest object, add this line.

System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Ssl3;

This means that Secure Socket Layer (SSL) 3.0 as a security protocol

http://support.microsoft.com/kb/915599

Answer 4

Check request delay. If it is more than a few ms, then you should turn off the Nagle algorithm

  ServicePointManager.UseNagleAlgorithm = false; 

Answer 5

I had the same problem regarding the query WITHOUT the body. In my case, installing ContentLength in ZERO fixed the problem.

Answer 6

In my case, the problem was that we used "http: // .." as the business address instead of "https: //" ...