Streaming HTTP with gzip buffered StreamReader?

Question

Fight to find anyone who is experiencing a similar problem or something similar.

I am currently consuming a stream via http (json) that has a GZip requirement, and I am experiencing a delay from sending data when reader.ReadLine() reads it. I was suggested that this could be related to decoding, storing data in a buffer?

This is what I have now, it works great except for the delay.

 HttpWebRequest request = (HttpWebRequest)WebRequest.Create(endPoint); request.Method = "GET"; request.PreAuthenticate = true; request.Credentials = new NetworkCredential(username, password); request.AutomaticDecompression = DecompressionMethods.GZip; request.ContentType = "application/json"; request.Accept = "application/json"; request.Timeout = 30; request.BeginGetResponse(AsyncCallback, request); 

Then inside the AsyncCallback method, I have:

 HttpWebRequest request = result.AsyncState as HttpWebRequest; using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result)) using (Stream stream = response.GetResponseStream()) using (StreamReader reader = new StreamReader(stream, Encoding.UTF8)) { while (!reader.EndOfStream) { string line = reader.ReadLine(); if (string.IsNullOrWhiteSpace(line)) continue; Console.WriteLine(line); } } 

It just sits on reader.ReadLine() until more data is received, and then even cancels some of it. There are also new symbols of new life, which are often read right away when he decides to read something.

I tested a stream running side by side with the curl command running, the curl command receiving and decompressing the data is perfectly fine.

Any insight would be awesome. Thanks,

Dan

EDIT No luck using buffer size on streamreader.

 new StreamReader(stream, Encoding.UTF8, true, 1) 

EDIT Also no luck updating .NET 4.5 and using

 request.AllowReadStreamBuffering = false; 

Answer 1

Update . This seems to have problems for extended periods of time with higher volume levels, and should only be used on a small volume where the buffer affects the functionality of the application. Since then I have returned to StreamReader .

So this is what I came up with. This works without delay. This is not provided by buffering using GZip automatic decompression.

 using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result)) using (Stream stream = response.GetResponseStream()) using (MemoryStream memory = new MemoryStream()) using (GZipStream gzip = new GZipStream(memory, CompressionMode.Decompress)) { byte[] compressedBuffer = new byte[8192]; byte[] uncompressedBuffer = new byte[8192]; List<byte> output = new List<byte>(); while (stream.CanRead) { int readCount = stream.Read(compressedBuffer, 0, compressedBuffer.Length); memory.Write(compressedBuffer.Take(readCount).ToArray(), 0, readCount); memory.Position = 0; int uncompressedLength = gzip.Read(uncompressedBuffer, 0, uncompressedBuffer.Length); output.AddRange(uncompressedBuffer.Take(uncompressedLength)); if (!output.Contains(0x0A)) continue; byte[] bytesToDecode = output.Take(output.LastIndexOf(0x0A) + 1).ToArray(); string outputString = Encoding.UTF8.GetString(bytesToDecode); output.RemoveRange(0, bytesToDecode.Length); string[] lines = outputString.Split(new[] { Environment.NewLine }, new StringSplitOptions()); for (int i = 0; i < (lines.Length - 1); i++) { Console.WriteLine(lines[i]); } memory.SetLength(0); } } 

Answer 2

Maybe there is something for C. Evenhuis Delayed ACK, but I have a weird gut that feels StreamReader causing headaches ... you can try something like this:

 public void AsyncCallback(IAsyncResult result) { HttpWebRequest request = result.AsyncState as HttpWebRequest; using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result)) using (Stream stream = response.GetResponseStream()) { var buffer = new byte[2048]; while(stream.CanRead) { var readCount = stream.Read(buffer, 0, buffer.Length); var line = Encoding.UTF8.GetString(buffer.Take(readCount).ToArray()); Console.WriteLine(line); } } } 

EDIT: Here's the full harness I used to test this theory (maybe the difference in your situation will pop up on you)

(LINQPad ready)

 void Main() { Task.Factory.StartNew(() => Listener()); _blocker.WaitOne(); Request(); } public bool _running; public ManualResetEvent _blocker = new ManualResetEvent(false); public void Listener() { var listener = new HttpListener(); listener.Prefixes.Add("http://localhost:8080/"); listener.Start(); "Listener is listening...".Dump();; _running = true; _blocker.Set(); var ctx = listener.GetContext(); "Listener got context".Dump(); ctx.Response.KeepAlive = true; ctx.Response.ContentType = "application/json"; var outputStream = ctx.Response.OutputStream; using(var zipStream = new GZipStream(outputStream, CompressionMode.Compress)) using(var writer = new StreamWriter(outputStream)) { var lineCount = 0; while(_running && lineCount++ < 10) { writer.WriteLine("{ \"foo\": \"bar\"}"); "Listener wrote line, taking a nap...".Dump(); writer.Flush(); Thread.Sleep(1000); } } listener.Stop(); } public void Request() { var endPoint = "http://localhost:8080"; var username = ""; var password = ""; HttpWebRequest request = (HttpWebRequest)WebRequest.Create(endPoint); request.Method = "GET"; request.PreAuthenticate = true; request.Credentials = new NetworkCredential(username, password); request.AutomaticDecompression = DecompressionMethods.GZip; request.ContentType = "application/json"; request.Accept = "application/json"; request.Timeout = 30; request.BeginGetResponse(AsyncCallback, request); } public void AsyncCallback(IAsyncResult result) { Console.WriteLine("In AsyncCallback"); HttpWebRequest request = result.AsyncState as HttpWebRequest; using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result)) using (Stream stream = response.GetResponseStream()) { while(stream.CanRead) { var buffer = new byte[2048]; var readCount = stream.Read(buffer, 0, buffer.Length); var line = Encoding.UTF8.GetString(buffer.Take(readCount).ToArray()); Console.WriteLine("Reader got:" + line); } } } 

Output:

 Listener is listening... Listener got context Listener wrote line, taking a nap... In AsyncCallback Reader got:{ "foo": "bar"} Listener wrote line, taking a nap... Reader got:{ "foo": "bar"} Listener wrote line, taking a nap... Reader got:{ "foo": "bar"} Listener wrote line, taking a nap... Reader got:{ "foo": "bar"} Listener wrote line, taking a nap... Reader got:{ "foo": "bar"} Listener wrote line, taking a nap... Reader got:{ "foo": "bar"} 

Answer 3

This may be related to the Delayed ACK in conjunction with the Nagle algorithm. This happens when the server sends several small responses in a row.

On the server side, the first response is sent, and subsequent fragments of the response data are sent only when the server received the ACK from the client or until there is enough data for a large packet to be sent (Nagle algorithm).

On the client side, the first response bit is received, but the ACK is not sent immediately - since traditional applications have request-response-request-response-response behavior, it assumes that it can send the ACK along with the next request - which does not happen in your case.

After a fixed amount of time (500 ms?), He decides to send the ACK in any case, as a result of which the server will send the following packets to which it has accumulated sofar.

The problem (if this is really the problem you are facing) can be fixed on the server side at the socket level by setting the NoDelay property, disabling the Nagle algorithm. I think you can also disable it in the operating system.

You can also temporarily disable the Delayed ACK (I know that Windows has a registry entry) on the client side to make sure that this is really a problem without changing anything on your server. ACK latency prevents DDOS attacks, so be sure to restore your settings later.

Submitting keepalives less often may also help, but you will still have a chance of a problem.