I know that this may have been asked two million times, but I cannot find a golden solution for my specific use case.
I have only one data structure, a map, where the key is a string. The objects of the map are the maps themselves, but this time the values are simple objects / primitives, such as a string, int, double, etc. Therefore, a map of cards. The keys of the innermost map are permanent, i.e. No entries are ever added / deleted from the innermost map, except when they are created. Therefore, it looks like a traditional table, although each row can have arbitrary columns.
I need this data structure to be persistent and replicated.
Here are my requirements:
- Pure Java Solution
- The disk card is used only if it is restarted. Therefore, it is never read from disk, and all recordings are performed by only one application)
- Embedded
- Performance. Important is the UPDATE performance of existing records. UPDATES will occur potentially 100k times per second (but more likely 20-50k per second). As for INSERT / DELETE, they certainly occur, but probably only a few times a day. So I'm not too worried about the performance of INSERT / DELETE.
- Replicated. For stability, I need a copy of the disk to replicate the card. Replication from master to slave does not have to be part of the original transaction, that is, I can sacrifice some ACIDness for performance.
- The number of entries is expected to be 100k-200k, but not much higher. The size of each record is probably 100-200 Kbytes, so there is actually not much data. I assume the total data file size will be below 100 MB, and this is probably a high side estimate.
- The total amount of data is no more than it can always fit into memory. (that’s why I can guarantee that there will be no read of the disk, except during startup)
- My application is not distributed. At any given time, there is only one active process that writes to disk.
- Liberal open source license. (Apache, BSD, LGPL, should be fine)
The application in question never needs to store anything other than the above data structure, that is, it will not have future uncovered need for other persistent data structures. Therefore, this is true for optimization based on this particular data structure.
I looked at the Berkeley DB Java version, but it failed to meet requirement # 6. I looked at TokyoCabinet / KoyotoCabinet, but it did not fulfill requirement No. 1.
So what would you recommend?
source share