Is there a link / resource on how to create a data file structure?

Possible duplicate:
What are the important points when developing a (binary) file format?

I am going to develop a program that will store data in a file.

The file may be large. The data in the file consists mainly of variable length records. And I need random access to records.

I just want to read some rhesus / books on how to create a data file structure. But I can’t find it yet.

Any suggestion is much appreciated.

+6
source share
4 answers

The problem you are describing is the central theme of database theory.

Any decent text on this should give you some good ideas. Standard text from uni:

Basics of Database Systems - Elmasari and Nava (PDF) (Amazon)

Another approach is to use an array of structured arrays with memory, take a look at my answer to a similar question

Another approach is to use a binary protocol, for example, Google protobuf and “send” your data to a file when recording and “receive”, this when recording.

+2
source

You can find http://decoy.iki.fi/texts/filefd/filefd . This is a common starting point for the methods discussed.

Also look here at SO: What are the important points when developing a (binary) file format?

+3
source

If the answer you're looking for is “what kind of book to read,” I can't help.

If “how to do this” may be good for you, I have some suggestions.

One good solution is the proposal proposed by Srikar; I would just add that instead of SQLI I would use SQLite. This is an open source C library that you can embed in your program. It allows you to store data in the database in the same way as with the SQL statement, but instead call the functions of the C library. In your case, you can store everything in memory and then save the data to disk at the right time.

Link: http://www.sqlite.org

Another option is the old do-it-yourself. I mean: there is nothing complicated in storing your data in a file (if your data is not very structured, but in this case I would choose option No. 1).

You write down a plan of how you want your file structure to be. And you follow this plan both when writing a file to disk, and when reading it overwriting data into memory.

If you have entries n . Write n to disk, then write each entry.

If each record has a variable length, then record the length of each record before recording the record.

You are talking about "random access" in your question. You probably mean that the file is very large and during access you want to read from the disk only the part you are interested in.

If you plan to build an index; this index will report the offset of each element in bytes from the beginning of the file. Save the index at the beginning of the file, and then save the data.

When you read a file, you begin to read the index, get the offset of the data you need, and read this part of the file.

These are very simple examples, just to get an idea ...

Hope they help!

+2
source

Is there a reason why you are not considering placing this data in a persistent database repository such as mysql? this system is built to handle random access to data with proper indexes to speed up data retrieval. Plus, when reading from a file, you will need to read the entire file to get what you want, since there are no indexes or query language.

In addition to this, they have systems to make sure that several running processes can access the same data without distorting the data. He provided data recovery in case of non-compliance.

Thus, simple storage is a simple part, but it does not end there. In the end, you will need to provide all the other solutions. The best use available.

+1
source

All Articles