Is OS file buffering dangerous?

Question

I wrote a download library for my colleague. It writes downloaded data to files.

My colleagues found that the file remains small for a long time, even if 100 MB of data was downloaded.

Therefore, they suggest that I call flush() after each write() so that it does not take up memory to buffer this data.

But I don’t think that there is a lot of 100 MB of virtual memory, and I think that windows have their own reason for buffering a lot of data.

What do you think about it?

Answer 1

I would trust the operating system to personalize myself accordingly.

As for the “rinse immediately so as not to lose data if the power dies” - if the power freezes halfway through the file, can you trust that the data you wrote was OK and resume downloading? If so, it might be worth washing early - but I would weigh the difficulty of renewing against the relative rarity of power outages and just close the file when I read everything. If you see half the recorded file, delete it and download from scratch.

Answer 2

Well, you must first research / debug what is happening. The problem may be elsewhere; for example, Windows Explorer may not update the file size fast enough.

However, you are right, as a rule, if the OS virtual machine system decided to reload the material in RAM, it has every reason for this, and you should not interfere normally. If there is a lot of free memory, it makes sense to use it in the end.

Answer 3

If it were me, I would like all the data to be stored in an unstable place as soon as possible. I would definitely drop threads to make sure that I didn't lose anything in the event of a power failure. You did not indicate whether you need to access the data later, but I assume that it is, otherwise why would you store it? However, to answer the original question - this is not "harmful" for the OS, but you risk losing data.

Answer 4

Flushing at certain intervals / sizes / lines may be good, rather than flushing for each recording. This helps to reduce the amount of memory, and also to ensure that the actual file is updated periodically. For example, you can reset for every 100 lines.

Answer 5

If there is a means to reduce memory requirements with little impact on performance, I would prefer a less greedy version. I may need this memory for something more important, and the size of 100Mb for the bootloader is quite huge.