The presence of several simultaneous authors (without reading) in one file. Is it possible to run .NET in .NET?

Question

I am developing a multi-segment file loader. To accomplish this task, I am currently creating as many temporary files on disk as I have segments (they are fixed in number during file download). In the end, I simply create a new file f and copy the contents of all segments to f .

I was wondering if this is not the best way to do this. My idealization consists of the initial creation of f in full-size form, and then various streams are written directly to their part. There should not be any interaction between them. We can assume that any of them will start its starting point in the file, and then it will fill in the information sequentially in the file until the task is completed.

I heard about memory files ( http://msdn.microsoft.com/en-us/library/dd997372(v=vs.110).aspx ), and I wonder if they are the solution to my problem or not.

thanks

Answer 1

Using a memory mapped API is absolutely doable, and it will probably work quite well, because some tests will be recommended.

If you want to find a possible alternative implementation, I have the following suggestion.

• Create a static stack data structure in which load streams can pop out each segment of the file immediately after it is downloaded.

• Ask a separate thread to listen for push notifications on the stack. Send segments of the stack files and save each segment in the target file in one stream method.

Following the above pattern, you have separated the loading of file segments and saving to a regular file, placing the stack container between them.

Depending on the stack processing implementation, you may be able to implement this with very little thread blocking, which will provide maximum performance.

The upside of this is that you have 100% control over what is happening and a solution that may be more portable (if this should ever be a problem).

The stack decoupling pattern you make can also be implemented fairly universally and can even be reused in the future.

The implementation of this is not so complicated and probably on the same level as the implementation that should be performed around the api mapped memory.

Good luck ...

/ Anders

Answer 2

Yes, it is possible, but the only precaution you need to have is to ensure that none of the two streams are recorded in one place in the file, otherwise the contents of the file will be incorrect.

  FileStream writeStream = new FileStream(destinationPath, FileMode.OpenOrCreate, FileAccess.Write, FileShare.Write); writeStream.Position = startPositionOfSegments; //REMEMBER This piece of calculation is important // A simple function to write the bytes ... just read from your source and then write writeStream.Write(ReadBytes, 0 , bytesReadFromInputStream); 

After each Write we used writeStream.Flush(); so that buffered data is written to a file, but you can change as per your requirement.

Since you already have code that loads file segments in parallel. The only change you need to make is just to open the file stream as described above, and instead of creating many segment files locally, just open the stream for one file.

startPositionOfSegments very important and computes it perfectly, so that neither of the two segments overwrites the downloaded uploaded bytes to the same place in the file, otherwise it will give an incorrect result.

The above procedure works fine, but it can be a problem if the size of your segment is too small (we also encountered this, but after increasing the size of the segments it became fixed). If you encounter any exception, you can also synchronize only the Write part.

Answer 3

The answers posted so far, of course, relate to your question, but you should also consider the fact that multi-threaded I / O entries are unlikely to give you any performance benefits.

The reason for multi-threaded downloads is obvious and has dramatic results. However, when you try to combine files, remember that you have several threads that control the mechanical head on regular hard drives. With SSDs, you can get better performance.

If you use a single stream, you significantly exceed the recording capacity on the SEQUENTIAL hard drive. This is, by definition, the fastest way to write to conditional discs.

If you think otherwise, I would be interested to know why. I would rather focus on tuning the write performance of a single stream, playing with buffer sizes, etc.