What is the correct way to manage large IStreams?

Question

I am writing a C # class library to transfer large amounts of data through COM automation using IStream. It uses a call to CreateStreamOnHGlobal to create a stream, and methods in System.Runtime.InteropServices.COMTypes.IStream to work with it.

My question is, when transferring large amounts of data, the best way to maintain control over memory? Downloading 100MB + files to memory seems wasteful, and the client application will have to wait until this process is complete before downloading anything.

My plan was to create a stream with a sufficient size and write to it several times. Before writing the next piece of data, go back to the beginning and overwrite, starting from the beginning. I think about it correctly, and is there a better way to solve this problem?

Answer 1

Can you take a look at this Microsoft article on big data and streams. In one of my projects, we transfer the same stream down to the client, and since we never do buffering and maintain it as a stream, until the client receives it (through the response stream for the Internet) or the file stream on the windows app, we were able to leave very few snapshots of memory on the server.

Try to see if WCF and TransferMode = Streamed in conjunction with MTOM encoding support your requirements.

Big data and streaming => http://msdn.microsoft.com/en-us/library/ms733742.aspx

Answer 2

Consider using a file created with the FILE_ATTRIBUTE_TEMPORARY and FILE_FLAG_DELETE_ON_CLOSE attributes. Write something there. Windows will try to save it to disk cache if the memory does not run out. It will self-destruct when you close the handle or when your program terminates (or works!). I found out about it here .

Answer 3

For temporary local storage of large amounts of data, a memory mapped file is usually a good idea. He transfers responsibility for managing Windows memory, which best knows how to do this. If you do not specify a file, it will be displayed in the swap file and will never go to the physical disk if it is not needed.

If you are using .NET 4.0, you have a managed API in memory mapped files.

Answer 4

Each stream implementation is implemented in a buffer. In most cases, you will have 100% control over the size of the buffer. You can adjust the buffer size to adjust the balance of performance and memory. If you do not open hundreds of threads at the same time, I offer you the widest possible stream size:

var readStream = new System.IO.File.OpenRead (sourceFilePath); var writeStream = new System.Io.File.Create (destinationFile);

byte [] buffer = new bytes [ bufferSize ]; int readLength; while ((readLength = readStream.Read (0,0, BufferSize ))> 0) {
writeStream.Write (buffer, 0, readLength); }

writeStream.Close (); readStream.Close ();

Answer 5

When sending large messages using the Windows Communication Foundation (WCF), it is often desirable to limit the amount of memory used to buffer these messages. One possible solution is the message body stream (provided that the bulk of the data is in the body). However, some protocols require buffering the entire message. Reliable messaging and security are two such examples.

Another possible solution is to split the large message into smaller messages called chunks, send these chunks one fragment at a time, and recreate the large message on the receiving side. The application itself can perform this operation and de-chunking or use its own channel for this. A sample channel channel shows how you can use your own protocol or multi-level channel to perform block and decomposition of arbitrarily large messages.

The request is provided for download http://msdn.microsoft.com/en-us/library/aa717050.aspx