Stream.CopyToAsync with progress reporting - progress is reported even after copying is complete

Question

I created simple console applications that download files from the Internet.
Since I had problems with WebClient , I decided to write my application using HttpClient.

I basically execute a request to read the headers, and then using ReadAsStreamAsync I get a stream that I copy to a local file using CopyToAsync .

I found an extension method for a stream that supports IProgress:

 public static class StreamExtensions { public static async Task CopyToAsync(this Stream source, Stream destination, IProgress<long> progress, CancellationToken cancellationToken = default(CancellationToken), int bufferSize = 0x1000) { var buffer = new byte[bufferSize]; int bytesRead; long totalRead = 0; while ((bytesRead = await source.ReadAsync(buffer, 0, buffer.Length, cancellationToken)) > 0) { await destination.WriteAsync(buffer, 0, bytesRead, cancellationToken); cancellationToken.ThrowIfCancellationRequested(); totalRead += bytesRead; //Thread.Sleep(10); progress.Report(totalRead); } } } 

My application works, but I get incorrect progress information.
For example, when I upload 2 files, I see this in the output window:

 file1.tmp 60.95% file2.tmp 98.09% file1.tmp 60.98% file2.tmp 98.21% file2.tmp 98.17% file2.tmp 98.25% file1.tmp 61.02% file2.tmp 98.41% file2.tmp downloaded file2.tmp 98.29% file2.tmp 98.37% file1.tmp 61.06% file2.tmp 89.27% file2.tmp 89.31% file2.tmp 98.33% file2.tmp 98.45% file2.tmp 98.48% file1.tmp 61.10% file1.tmp 61.14% file2.tmp 98.52% file1.tmp 61.22% file2.tmp 98.60% file2.tmp 98.56% file1.tmp 61.30% file2.tmp 98.88% file2.tmp 90.44% file1.tmp 61.53% file2.tmp 98.72% file1.tmp 61.41% file1.tmp 61.73% file2.tmp 98.80% file1.tmp 61.26% file1.tmp 61.49% file1.tmp 61.57% file1.tmp 61.69% ... file1.tmp 99.31% file1.tmp 98.84% file1.tmp 98.80% file1.tmp 99.04% file1.tmp 99.43% file1.tmp 99.12% file1.tmp 99.00% file1.tmp downloaded file1.tmp 100.00% file1.tmp 98.73% file1.tmp 98.88% file1.tmp 99.47% file1.tmp 99.98% file1.tmp 99.90% file1.tmp 98.96% file1.tmp 99.78% file1.tmp 99.99% file1.tmp 99.74% file1.tmp 99.59% file1.tmp 99.94% file1.tmp 98.49% file1.tmp 98.53% ALL FILES DOWNLOADED file1.tmp 99.55% file1.tmp 98.41% file1.tmp 99.62% file1.tmp 98.34% file1.tmp 99.66% file1.tmp 98.69% file1.tmp 98.37% 

As you can see, I received information that file2 was uploaded, but I still get a progress report from CopyToAsync , the same with file1.

because of this, I sometimes get this strange console output:

Ideally, I would like to be sure than when I call:

 await streamToReadFrom.CopyToAsync(streamToWriteTo, progress, source.Token,0x2000); Debug.WriteLine(filename+" downloaded"); 

after receiving this debugging information, no progress is reported (file uploaded). I thought await would solve my problem, but it is not.

How can i fix this? As a workaround, I add CopyToAsync to CopyToAsync just before the progress report.

Below is my current code:

 using System; using System.Collections.Generic; using System.Diagnostics; using System.IO; using System.Linq; using System.Net.Http; using System.Threading; using System.Threading.Tasks; namespace AsyncDownloadTest { class Program { private const string LocalPath = @"D:\TEMP"; static void Main() { try { var filesToDownlad = new List<Tuple<string, string>> { new Tuple<string, string>("file1.tmp", "http://ipv4.download.thinkbroadband.com/10MB.zip"), new Tuple<string, string>("file2.tmp", "http://ipv4.download.thinkbroadband.com/10MB.zip") }; _consolePosition = -1; Console.CursorVisible = false; Parallel.ForEach(filesToDownlad, new ParallelOptions { MaxDegreeOfParallelism = 4 }, doc => { DownloadFile(doc.Item2,doc.Item1).Wait(); }); Debug.WriteLine("ALL FILES DOWNLOADED"); Console.CursorVisible = true; } catch (Exception e) { Console.WriteLine(e); Console.ReadLine(); } } private static readonly object ConsoleLock = new object(); private static int _consolePosition; static readonly CancellationTokenSource source = new CancellationTokenSource(); private static async Task DownloadFile(string url, string filename) { int currenctLineNumber = 0; int currectProgress = 0; try { lock (ConsoleLock) { _consolePosition++; currenctLineNumber = _consolePosition; } long fileSize = -1; IProgress<long> progress = new Progress<long>(value => { decimal tmp = (decimal)(value * 100) / fileSize; if (tmp != currectProgress && tmp > currectProgress) { lock (ConsoleLock) { currectProgress = (int)tmp; Console.CursorTop = currenctLineNumber; Console.CursorLeft = 0; Console.Write("{0,10} - {2,11} - {1,6:N2}%", filename, tmp, "DOWNLOADING"); } Debug.WriteLine("{1} {0:N2}%", tmp, filename); } }); using (HttpClient client = new HttpClient()) { using (HttpResponseMessage response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead, source.Token)) { response.EnsureSuccessStatusCode(); if (response.Content.Headers.ContentLength.HasValue) fileSize = response.Content.Headers.ContentLength.Value; if (response.Content.Headers.ContentDisposition != null) { var tmp = response.Content.Headers.ContentDisposition.FileName.Replace("\"", ""); Debug.WriteLine("Real name: {0}",tmp); } using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync()) { using (Stream streamToWriteTo = File.Open(Path.Combine(LocalPath, filename), FileMode.Create, FileAccess.Write)) { await streamToReadFrom.CopyToAsync(streamToWriteTo, progress, source.Token,0x2000); Debug.WriteLine(filename+" downloaded"); lock (ConsoleLock) { Console.CursorTop = currenctLineNumber; Console.CursorLeft = 0; var oldColor = Console.ForegroundColor; Console.ForegroundColor = ConsoleColor.Green; Console.Write("{0,10} - {2,11} - {1,6:N2}%", filename, 100, "SUCCESS"); Console.ForegroundColor = oldColor; } } } } } } catch (Exception e) { Debug.WriteLine(e.Message); lock (ConsoleLock) { Console.CursorTop = currenctLineNumber; Console.CursorLeft = 0; var oldColor = Console.ForegroundColor; Console.ForegroundColor = ConsoleColor.Red; Console.Write("{0,10} - {2,11} - {1,6:N2}%", filename, currectProgress, "ERROR"); Console.ForegroundColor = oldColor; } } } } public static class StreamExtensions { public static async Task CopyToAsync(this Stream source, Stream destination, IProgress<long> progress, CancellationToken cancellationToken = default(CancellationToken), int bufferSize = 0x1000) { var buffer = new byte[bufferSize]; int bytesRead; long totalRead = 0; while ((bytesRead = await source.ReadAsync(buffer, 0, buffer.Length, cancellationToken)) > 0) { await destination.WriteAsync(buffer, 0, bytesRead, cancellationToken); cancellationToken.ThrowIfCancellationRequested(); totalRead += bytesRead; Thread.Sleep(10); progress.Report(totalRead); } } } } 

Answer 1

Your problem is here:

 new Progress<long> 

The Progress<T> class always calls its callbacks in the SynchronizationContext - in this case, it is the SynchronizationContext thread pool. This means that when the progress report code calls Report , it simply queues the callback to the thread pool. Thus, you can see them out of order (or still a little after the download is complete).

To fix this, you can create your own IProgress<T> implementation:

 //C#6.0 public sealed class SynchronousProgress<T> : IProgress<T> { private readonly Action<T> _callback; public SynchronousProgress(Action<T> callback) { _callback = callback; } void IProgress<T>.Report(T data) => _callback(data); } //older version public sealed class SynchronousProgress<T> : IProgress<T> { private readonly Action<T> _callback; public SynchronousProgress(Action<T> callback) { _callback = callback; } void IProgress<T>.Report(T data) { _callback(data); } } 

Then replace the line

 IProgress<long> progress = new Progress<long>(value => 

from

 IProgress<long> progress = new SynchronousProgress<long>(value => 

Answer 2

As requested by OP, I show how to make my program with TPL data stream in the comments. This is actually a fairly simple conversion. First add a link to the NuGet System.Threading.Tasks.Dataflow package. Then just change the main function to

 static void Main() { try { var filesToDownlad = new List<Tuple<string, string>> { new Tuple<string, string>("file1.tmp", "http://ipv4.download.thinkbroadband.com/10MB.zip"), new Tuple<string, string>("file2.tmp", "http://ipv4.download.thinkbroadband.com/10MB.zip") }; _consolePosition = -1; Console.CursorVisible = false; var downloadBlock = new ActionBlock<Tuple<string, string>>(doc => DownloadFile(doc.Item2, doc.Item1), new ExecutionDataflowBlockOptions {MaxDegreeOfParallelism = 4}); foreach (var file in filesToDownlad) { downloadBlock.Post(file); } downloadBlock.Complete(); downloadBlock.Completion.Wait(); Debug.WriteLine("ALL FILES DOWNLOADED"); Console.CursorVisible = true; } catch (Exception e) { Console.WriteLine(e); Console.ReadLine(); } } 

If you do this using a program with a synchronization context and want to wait for completion and publication instead of performing synchronous operations, you can do

 static async Task Example() { try { var filesToDownlad = new List<Tuple<string, string>> { new Tuple<string, string>("file1.tmp", "http://ipv4.download.thinkbroadband.com/10MB.zip"), new Tuple<string, string>("file2.tmp", "http://ipv4.download.thinkbroadband.com/10MB.zip") }; _consolePosition = -1; Console.CursorVisible = false; var downloadBlock = new ActionBlock<Tuple<string, string>>(doc => DownloadFile(doc.Item2, doc.Item1), new ExecutionDataflowBlockOptions {MaxDegreeOfParallelism = 4}); foreach (var file in filesToDownlad) { await downloadBlock.SendAsync(file); } downloadBlock.Complete(); await downloadBlock.Completion; Debug.WriteLine("ALL FILES DOWNLOADED"); Console.CursorVisible = true; } catch (Exception e) { Console.WriteLine(e); Console.ReadLine(); } } 

Please note that this does not fix your "ALL DOWNLOAD FILES" problem, which were launched at an early stage. You need to use Stephen's solution to fix this . This is all fixing a potential deadlock if this code worked in a situation where there could be a SynchronizationContext in the calling thread.