The fastest small data warehouse in Windows

Question

My application monitors the status of about 1000 objects. These objects are read and written to persistent storage (serialized) without any special order.

Currently, the application uses the registry to store each state of the object. This is nice because:

• It's simple

• It's very fast

• The state of an individual object can be read / written without having to read a larger object (for example, pulling a fragment from a large XML file)

• There is a decent editor (RegEdit) that makes it easy to manipulate individual elements

Having said that, I wonder if there is a better way. SQLite seems like an opportunity, but you do not have the same level of multiple reading / multiple writer that you get with the registry, and there is no easy way to edit existing records.

Any best deals? A bunch of flat files?

Answer 1

If you are starting to experiment with SQLite, you should know that out of the box this may not seem as fast as you would like, but you can quickly do it faster using some of the established optimization tips:

SQLite optimization

Depending on the size of the data and the amount of RAM available, one of the best performance gains will be by installing sqlite to use the all-in-memory database, rather than writing to disk.

For in-memory databases, pass NULL as an argument to the sqlite3_open file name and ensure that TEMP_STORE is defined appropriately

On the other hand, if you tell sqlite to use the hard drive, you will get a similar advantage for your current use of RegEdit to manage program data on the fly.

The way you could model your current RegEdit method with sqlite would be to use the sqlite command line tool to connect to a database on disk. You can run UPDATE statements for SQL data from the command line while your main program is running (and / or while it is paused in break mode).

Answer 2

If what you mean by “multiple readers / multiple writers” is that you save many threads writing to the repository at the same time, SQLite is thread safe (you can have simultaneous SELECTs and simultaneous writes are processed transparently). See [FAQ [1]] and grep for "threadsafe"

[1]: http://www.sqlite.org/faq.html/ FAQ

Answer 3

I doubt that any sane person will go this route these days, however some of what you described can be done using Window Structured / Compound Storage , I only mention this since you are asking about Windows - and this / was the official Windows way for this.

This is how DOC files (but not the new DOCX format) stack up. From MSDN it will be very difficult, but I used it, it is not the worst API in Win32.

• not easy
• it is fast, I would guess that it is faster than the registry.
• The state of an individual object can be read / written without having to read a larger object.
• There is no suitable editor, but there are some real basic things (VC ++ 6.0 had the “DocFile Viewer” in the “Tools” section. (Yes, that it did). I found a few more on the Internet.
• You get a file instead of registry keys.
• You have a geek-cred developer for Windows developers.

Other random thoughts: I think XML is the way to go (despite the random access problem). Heck, .ini files can work. The registry gives you very good grain protection if you need it - people seem to forget about it when a file claim is better. The built-in DB seems redundant if I understand what you are doing.

Answer 4

Do you need to save objects in each change event or just in memory and store it at the end of the work? If so, just download them and serialize at the end, assuming your application has been running for a long time (and you are not sharing this state with another program), then memory will be the winner.

If you have fixed size structures, can you consider using a file with memory mapping and allocating memory from it?

Answer 5

If you are only doing serialization / deserialization of individual objects (without fancy queries), use a btree database like Berkeley DB . This is very fast when storing and retrieving pieces of data by key (I assume that your objects have some identifier that can be used as a key), and access is supported by several processes.