HttpWebRequest, BeginGetResponse not called until endOfStream

Question

I’m working on a client that uses a web service to get some events moving forward - the web service is designed in such a way that after the client executes the subscription command, it will return some interesting events and continue to do so while the client stays in touch .

When sending a command, the service responds (immediately) to the original response with these headers

Keep-Alive: timeout=5, max=98 Connection: Keep-Alive Transfer-Encoding: chunked 

and then keeps the connection open until it expires (after 30 seconds if the client does not send some keep-alive data)

Since this is a combination of POST + to read answer + while keeping the connection open until endOFStream, it seems that I should use HttpWebRequest with BeginGetRequestStream (before POST) and BeginGetResponse to read and respond to the response.

My problem is that the BeginGetResponse callback is not called until the input stream is actually closed by the server / service (after 30 seconds), despite the fact that AllowReadStreamBuffering is set to false.

The document should say this in AllowReadStreamBuffering:

The AllowReadStreamBuffering property affects the invocation of the BeginGetResponse calling method. When the AllowReadStreamBuffering property is true, the callback occurs after the entire stream has been loaded into memory. When the AllowReadStreamBuffering property is false, the callback is raised as soon as the stream is readable, which may be before all data.

I saw several suggestions that no matter what AllowReadStreamBuffering is selected for, HttpWebRequest will not call BeginGetResponse until the buffer is full - but I could not find anything in this document.

Does anyone have an idea on how to control this buffering behavior or maybe suggest a different approach that I should try when working with this kind of web service?

The relevant snippets of the code I'm currently using are as follows:

 public void open() { string url = "http://funplaceontheinternet/webservice"; HttpWebRequest request = WebRequest.CreateHttp(url); request.Method = "POST"; request.Credentials = new NetworkCredential("username", "password"); request.CookieContainer = new CookieContainer(); request.AllowReadStreamBuffering = false; request.BeginGetRequestStream(new AsyncCallback(GetRequestStreamCallback), request); } void GetRequestStreamCallback(IAsyncResult result) { Debug.WriteLine("open.GetRequestStreamCallback"); HttpWebRequest webRequest = (HttpWebRequest)result.AsyncState; // End the stream request operation Stream postStream = webRequest.EndGetRequestStream(result); // Create the post data byte[] byteArray = Encoding.UTF8.GetBytes(_xmlEncodedSubscribeCommand); // Add the post data to the web request postStream.Write(byteArray, 0, byteArray.Length); postStream.Close(); // Start the web request webRequest.BeginGetResponse(new AsyncCallback(BeginGetResponseCallback), webRequest); } void BeginGetResponseCallback(IAsyncResult result) { HttpWebRequest request = (HttpWebRequest)result.AsyncState; HttpWebResponse response = null; if (request != null) response = (HttpWebResponse)request.EndGetResponse(result); else Debug.WriteLine("request==null :-("); if (response != null) { using (var reader = new StreamReader(response.GetResponseStream())) { while (!reader.EndOfStream) { string line = reader.ReadLine(); Debug.WriteLine("BeginGetResponseCallback - received: " + line); } Debug.WriteLine("BeginGetResponseCallback - reader.EndOfStream"); } } else Debug.WriteLine("response==null :-("); } 

Answer 1

You mentioned that a service is a web service, but not a platform. If this is a “normal” web service, I assume XML is a transport format.

If so, I suspect that the problem may be that this communication style does not really lend itself to streaming. Server-side web service infrastructure may not create a SOAP envelope and payload until all data is available. If you wanted a stream like this, you could better use some user services on the server rather than web services.

Do you know that the server does indeed transmit the response? (e.g. confirmed something like wirehark?)

If you really want to use the web service, I suggest you fill out a request when the first events will be available, and not wait for a timeout. This will still lead to a reduction in the latency that I suppose you are trying to get.