I have about 500 million 128-bit integers, adding about 100 M per year. Nothing is ever deleted. Numbers have a uniform distribution on the scale and in time.
Basically, all I need is an add operation, which also returns if a number exists in the database. In addition, I do not want to use too much RAM for this system, so just storing everything in memory is not what I'm looking for.
So far, we have used several MyISAM tables in MySQL, using two large keys as the primary key. This gives us good performance, but I suspect that this is not the right tool for this job. Before breaking up the tables, we had performance problems, and we had a power outage. In addition, the database gives us much more functions that we do not need.
I use Python for Linux, but I am open to suggestions.
A similar question in C ++ .
UPDATE: Comment Marcelo mentioned the Bloom Filter , which seems very promising to me. Since I work with hashes, I already completely abandoned full precision, so this can be a great compromise with performance / performance.