Insert data into RandomAccessFile and update index

Question

I have a RandomAccessFile in Java where I manage some data. Simplified: At the beginning of the file, I have an index. (An 8-byte value in the dataset that represents the offset where real data can be found).

So, if I want now, where can I find dataset data no 3, for example. I read 8 bytes at offset (2 * 8). (Indexing starts at 0).

The data set itself consists of 4 bytes, which represents the size of the data set, and then all the bytes belonging to the data set.

So this works fine if I always rewrite the whole file.

It is very important here that Dataset no 3 could be written as the first record in the file, so the index is ordered, but not by itself.

If I insert a new dataset, I always add it to the end of the file. But the number of data sets that can be in one file is limited. If I can store 100 datasets in a file, there will always be 100 records in the index. If the offset read from the dataset index is 0, the dataset is new and will be added to the file.

Boo there is one case that still does not work for me. If I read dataset no. 3 from the file, and I add some data to it in my application, and I want to update it in the file, I have no idea how to do this.

If it is the same length as befor, I can just overwrite the old data. But if the new dataset has more bytes than the old, I will have to move all the data in the file that is behind this dataset and update the indexes for these datasets.

Any idea how to do this? Or maybe the best way to manage the storage of these data sets in a file?

PS: Yes, of course, I was thinking about using a database, but this is not applicable for my project. I really need simple files.

Answer 1

You cannot easily insert data in the middle of a file. You will basically need to read all the remaining data, write the “new” data, and then rewrite the “old” data. In addition, you can potentially invalidate the old “slow” (potentially allowing it to be reused later), and then simply write the entire new record to the end of the file. Your file format is not entirely clear to me, but in principle you need to know that you cannot insert (or delete) in the middle of a file.

Answer 2

I have a RandomAccessFile in Java where I manage some data.

Stop here. You have a file. You are currently accessing it through RandomAccessFile in Java. However, your whole question relates to the file itself, not to RandomAccessFile or Java. You have a serious problem with the design of the file, because you assume features such as pasting in the middle of a file that is not on any file system that I have used since 1979.

Answer 3

As others answered, there is no real way to make a file longer / shorter without overwriting the whole. There are some workarounds, and maybe one solution will work in the end.

• Limit all datasets to a fixed length .
• Delete by changing / deleting the index and adding, always adding to the end of the file. Update by deleting the old dataset and adding the new dataset to the end if the new dataset is larger. Compress the file from time to time, effectively deleting “ignored data sets” and moving all valid data sets together (overwriting everything).
• If you cannot limit the data set to a fixed length, and you intend to update the data set to make it longer, you can also leave the pointer at the end of the first part of the data set and continue it later in the file. This way you get the structure as a linked list . If a lot of editing is done, this will also make sense to modify and compress the file.

Most solutions have overhead, but file size is usually not a problem, and, as already mentioned, you can let some method "clear it".

PS: I hope it’s normal to answer such old questions - I could not find anything about it in the help center, and I am relatively new here.