C # NetworkStream.Read oddity

Question

Can anyone point out the flaws in this code? I am extracting HTML with TcpClient. NetworkStream.Read () never ends when talking to the IIS server. If I use the Fiddler proxy instead, it works fine, but when I go directly to the target server, the .read () loop will not exit until the connection exceptions fail, for example, “the remote server closed the connection” .

internal TcpClient Client { get; set; } /// bunch of other code here... try { NetworkStream ns = Client.GetStream(); StreamWriter sw = new StreamWriter(ns); sw.Write(request); sw.Flush(); byte[] buffer = new byte[1024]; int read=0; try { while ((read = ns.Read(buffer, 0, buffer.Length)) > 0) { response.AppendFormat("{0}", Encoding.ASCII.GetString(buffer, 0, read)); } } catch //(SocketException se) { } finally { Close(); } 

Update

In the debugger, I immediately see the whole answer going through , and is added to my StringBuilder (response). It just seems that the connection does not close when the server sends a response, or my code does not detect it.

Conclusion As mentioned here, it is best to use the protocol offers (in the case of HTTP, the Content-Length header) to determine when the transaction is completed. However, I found that not all pages have content lengths. So now I am using a hybrid solution:

• For ALL transactions, set the Connection request header to “close” so that the server is not encouraged to open the socket. This improves the chances that the server will close the connection when it answers your request.

• If Content-Length set, use it to determine when the request will complete.

• Otherwise, set the NetworkStream RequestTimeout property to a large but reasonable value, for example, 1 second. Then, loop on NetworkStream.Read() until a) time out, or b) you read fewer bytes than you requested.

Thanks to everyone for their excellent and detailed answers.

Answer 1

Not sure if this is useful or not, but with HTTP 1.1, the basic connection to the server may not close, so maybe the stream doesn't close either? The idea is that you can reuse the connection to send a new request. I think you should use content length. Use the WebClient or WebRequest classes instead.

Answer 2

Unlike the NetworkStream.Read documentation, a stream received from TcpClient does not just return 0 for the number of bytes counted when there is no data available - it is blocked.

If you look at the documentation for TcpClient , you will see this line:

The TcpClient class provides simple methods for connecting, sending, and receiving stream data over a network in synchronous blocking mode.

Now I assume that if your Read call is blocked, this is because the server decided not to send any data back. This is likely due to the fact that the original request does not pass properly.

My first suggestion would be to exclude StreamWriter as a possible cause (e.g., buffering / coding nuances) and write directly to the stream using the NetworkStream.Write method. If this works, make sure that you are using the correct settings for StreamWriter .

My second sentence will not depend on the result of calling Read to break the loop. The NetworkStream class has a DataAvailable property designed for this. The correct way to record a receive cycle:

 NetworkStream netStream = client.GetStream(); int read = 0; byte[] buffer = new byte[1024]; StringBuilder response = new StringBuilder(); do { read = netStream.Read(buffer, 0, buffer.Length); response.Append(Encoding.ASCII.GetString(buffer, 0, read)); } while (netStream.DataAvailable); 

Answer 3

Read the answer until you reach double CRLF. You now have the response headers. Parse the headers to read the Content-Length header, which will count the number of bytes left in the response.

Here is a regular expression that can capture the Content-Length header.

David Updated Regex

 Content-Length: (?<1>\d+)\r\n 

Content length

Note

If the server did not configure this header correctly, I would not use it.

Answer 4

I might be wrong, but it looks like your Write call is writing (under the hood) to stream ns (via StreamWriter ). Later you read from a single thread ( ns ). I do not quite understand why you are doing this?

In any case, you may need to use Seek in the stream to go to the place where you want to start reading. I would suggest that it tends to end after writing. But, as I said, I'm not sure if this is a useful answer!

Answer 5

Two sentences ...

• Have you tried to use the DataAvailable property for NetworkStream? It should return true if there is data to read from the stream.

 while (ns.DataAvailable) { //Do stuff here } 

2. Another option would be to change ReadTimeOut to a low value so you don't end up blocking. This can be done as follows:

 ns.ReadTimeOut=100;