I would like to create a JVM (Java / Scala) data structure that can be used to represent and store the contents of arbitrary relational database tables. The data structure should be fast (not too gc-intensive, in terms of caching) and efficient memory, so larger tables can fit in RAM.
One effective solution for storing data is to store each column separately in a primitive array, but I am concerned about the friendliness of the cache, because the elements on the same line are not stored together. A row with N columns will skip the N cache, regardless of how narrow the columns are.
Another solution is to store each line in an array of objects, where each element represents a field and is sent to the correct type when searching, but this requires storing the numeric types in their box, so it is not very memory efficient. And probably this is not so effective for the cache.
Another solution is to put each row of data into a byte array, just like real databases ordering their rows using only as many bytes as needed. This is safe for caching and memory efficiency, but I'm worried about the cost of serializing / de-serializing with every access.
What is the best way?
source
share