How to make sure the file was written successfully?

Question

I am adding autosave functions to a graphical Java application. The application periodically autosaves the current document, as well as autosaves it upon exit. When the user starts the application, the autosave file is reloaded.

If the autosave file is damaged in any way (suppose that turning off the power when the file is in the middle of saving will do this?), The user will lose his job. How can I prevent such situations and do my best to ensure that the autosave document is in a consistent state?

To complicate matters even further, to autosave a document, I need to save one .xml file and several .png files. In addition, .png is saved in C code over JNI.

My current strategy is to write each .png with the extension .png.tmp, write an XML file with the extension .xml.tmp, and then rename each file to remove the .tmp part, leaving the .xml to the last. When I start, I upload only the autosave file if I can find the .xml file and ignore the .xml.tmp files. I also do not delete the previous autosave file until the .xml.tmp file for the new document is renamed.

I think that my knowledge of what happens when you write to disk is poor. I know that you can use buffers for reading / writing software when using files, as well as buffers for OS and hardware, and that all these functions need to be cleared. I am confused how I can know for sure when something was really written to disk and what I can do to protect myself. Does the rename operation to ensure that the buffers are flushed?

Answer 1

If the autosave file is damaged in any way (suppose that turning off the power when the file is in the middle of saving will do this?), The user will lose his job. How can I prevent such situations and do my best to ensure that the autosave document is in a consistent state?

To prevent data loss due to a partially recorded autosave file, do not overwrite the autosave file. Instead, write to a new file each time, and then rename it as soon as the file is securely written.

Not to forget that the autosave file was not written correctly:

• Note the exceptions that were reset when the autosave file is written and closed in the event of a disk error, a complete file system, etc.
• Keep the current checksum of the file when it is written, and write it at the end of the file. Then, when you download the autosave file, check that the checksum exists and is correct.

If the check state contains several files, make sure that you write the files in a known order (without overwriting!) And write the checksum to the autosave file after all other files have been safely closed. You might want to create a directory for each checkpoint.

FOLLOW UP

No. I am not saying that renaming always succeeds. However, it is atomic — it either succeeds (and completes), or the file system does not change. So if you do this:

• write "file.new" and close,
• delete a file" ,
• rename the file "file.new" to "file"

then, provided that the first step is successful, you are guaranteed to receive the last “file” on disk. And just add a few steps so that you have a backup of the “file” at any time. (If the 3rd step is unsuccessful, you will remain with the file “file.new” and no “file.” This can be restored manually or automatically by the application the next time you start.)

Also, I'm not saying that records always succeed, or that applications do not crash, or that power never goes out. And the point of the checksum is to allow you to detect cases when this happened, and the autosave file is incomplete.

Finally, it’s nice to have two autosaves if your application gets into a state where its data structures are mixed up and the last autosave as a result is pointless. (The checksum will not protect against this.) Be careful when autosaving when the application crashes for the same reason.

Answer 2

Aside, since you have several different files as part of this single document, consider using either the project directory to hold them together, or use some kind of encapsulation format (e.g. .zip) to put them in a single file.

What you want to do is atomize old backup files with new ones. Unfortunately, I do not believe that Java gives you enough control to do this directly. You also need to find out which operations are atomic in the underlying operating system. I know Linux file systems, so my answer will be biased towards a Java program running on that system. I would be shocked if Windows did not do the same, but I cannot say for sure.

Most Linux file systems (like metadata logs) allow you to rename files atomically. If the system logs out halfway through the rename, it will look as if you had never renamed the file in the first place upon restart. For this reason, the general way to atomically update an existing file F is to write your new data to a temporary file T, and then rename T to F. Any system or application failure before this rename will not affect F, so it will always be consistent .

Of course, before renaming, you need to make sure that your temporary file is consistent. Make sure that all stream buffers for the file are Channel.force() to the OS ( Channel.force() or OutputStream.flush() ) and that the OS buffers are FileOutputStream.getFD.sync() to disk ( FileOutputStream.getFD.sync() ). Of course, if your OS does not disable the write cache on the hard drive itself (this is probably not), there is still a chance that your data may be corrupted. Add a checksum to XML if you really want to be sure. If you are truly paranoid, you should clear the OS and hard drive’s cache and re-read the file to make sure it is consistent. This goes beyond reasonable expectations for ordinary consumer applications.

But only for atomic recording write one file. Your advertisement is more complicated: you have a lot of files to update atomically. For example, I will say that you have two files: img.png and main.xml. I would do one of them:

• A simple solution is to create a directory for each file. You do not need to worry about renaming each individual file, and you can still atomically rename the new backup drive from the old backup directory that you are replacing. That is, if your old backup is bak / img.png and bak / main.xml, write bak.tmp / img.png and bak.tmp / main.xml and rename bak.tmp to bak.
• Call the new auxiliary files something else and let them coexist with the old ones for a while. That is, write img.2.png and main.xml.tmp (which should refer to img.2.png, not img.png) and just rename main.xml.tmp to main.xml. Then remove img.png.
• addition: If you do not have atomic renames, the next best thing applies to # 2. Whenever you save a project, give it a new name (for example, ver342.xml). When you download, just find the very latest XML that is consistent (i.e. checks the checksum). Store about 2 or 3 to be safe. Only delete auto-save if you successfully restored from a later copy.