Will a large System.IO.MemoryStream file lead to a significant increase in application memory usage?

Question

I am creating a library that allows the user to download files from a URL. One of the options I'm considering allows the user to specify the expected MD5 checksum for the file; library's GetFile function (string url) ensures that the checksum for the loaded stream matches the one specified by the user.

Knowing that the NetworkStream returned by HttpWebResponse.GetResponseStream () is not searchable, I found a way to duplicate the stream thanks to the answers to this question: How can I read Two Http response streams in C #? . Before I went any further, I wanted to find out what would be the consequences of this duplication of memory; Unfortunately, numerous searches on Google and MSDN have come to naught.

The library does not impose restrictions on the size of the downloaded file. My question is if a user selects a 2 GB file, is the implementation of MemoryStream in .NET 2.0 smart enough to use PageFile and RAM efficiently enough so that the system does not start scanning due to a VM failure? In addition, Jon Skeet commented on another question that gave me something he was thinking about - he claimed that even after disposing of the MemoryStream, the memory is not 100% free. How and when can I guarantee that the memory will indeed be released? Will it be released based on system requirements (and need)?

Thanks Manoj

Answer 1

You save it to a file, right? Why not keep it piece by piece by updating the hash as you go, and then just check the hash at the end? I don’t think you need to read the answer twice and not buffer it. As another answer indicates that it will fail if you still get 1 GB.

Remember that, like the current size of a MemoryStream , anytime it needs to grow, you end up with a (temporarily) new array plus an old array in memory at the same time. Of course, this would not be a problem if you knew the length of the content in advance, but it would still be better to write it to disk and the hash when you go.

Answer 2

MemoryStream is supported by an array. Even if you have a 64-bit OS, it will not work more than 1 GB, as the infrastructure will not allocate a larger array.

Answer 3

Afaik managed CLR heap does not allocate anything more than 2 GB, and MemoryStream is supported by a live, contiguous byte []. A large bunch of objects do not allocate descriptors larger than 2 GB, not even on x64.

But for storing the entire file in memory, just for calculating the hash seems rather low-tech. You can calculate the hash as you get bytes, a piece of the piece. After each I / O completion, you can hash the received bytes, and then send the record to a file. In the end, you computed the hash and uploaded the file, huraay.

BTW, If you are looking for code for managing files, avoid any pattern that contains the words ReadToEnd ...

 class Program { private static AutoResetEvent done = new AutoResetEvent(false); private static AsyncCallback _callbackReadStream; private static AsyncCallback _callbackWriteFile; static void Main(string[] args) { try { _callbackReadStream = new AsyncCallback(CallbackReadStream); _callbackWriteFile = new AsyncCallback(CallbackWriteFile); string url = "http://..."; WebRequest request = WebRequest.Create(url); request.Method = "GET"; request.BeginGetResponse(new AsyncCallback( CallbackGetResponse), request); done.WaitOne(); } catch (Exception e) { Console.Error.WriteLine(e.Message); } } private class State { public Stream ReponseStream { get; set; } public HashAlgorithm Hash { get; set; } public Stream FileStream { get; set; } private byte[] _buffer = new byte[16379]; public byte[] Buffer { get { return _buffer; } } public int ReadBytes { get; set; } public long FileLength {get;set;} } static void CallbackGetResponse(IAsyncResult ar) { try { WebRequest request = (WebRequest)ar.AsyncState; WebResponse response = request.EndGetResponse(ar); State s = new State(); s.ReponseStream = response.GetResponseStream(); s.FileStream = new FileStream("download.out" , FileMode.Create , FileAccess.Write , FileShare.None); s.Hash = HashAlgorithm.Create("MD5"); s.ReponseStream.BeginRead( s.Buffer , 0 , s.Buffer.Length , _callbackReadStream , s); } catch (Exception e) { Console.Error.WriteLine(e.Message); done.Set(); } } private static void CallbackReadStream(IAsyncResult ar) { try { State s = (State)ar.AsyncState; s.ReadBytes = s.ReponseStream.EndRead(ar); s.Hash.ComputeHash(s.Buffer, 0, s.ReadBytes); s.FileStream.BeginWrite( s.Buffer , 0 , s.ReadBytes , _callbackWriteFile , s); } catch (Exception e) { Console.Error.WriteLine(e.Message); done.Set(); } } static private void CallbackWriteFile(IAsyncResult ar) { try { State s = (State)ar.AsyncState; s.FileStream.EndWrite(ar); s.FileLength += s.ReadBytes; if (0 != s.ReadBytes) { s.ReponseStream.BeginRead( s.Buffer , 0 , s.Buffer.Length , _callbackReadStream , s); } else { Console.Out.Write("Downloaded {0} bytes. Hash(base64):{1}", s.FileLength, Convert.ToBase64String(s.Hash.Hash)); done.Set(); } } catch (Exception e) { Console.Error.WriteLine(e.Message); done.Set(); } } } 

Answer 4

I am sure you will get an OutOfMemoryException. An easy way to try is to try to read the ISO DVD image or something in memory using the memory stream. If you can read all this, then everything will be fine. If you get an exception, well, that's all.