Scalable stl like container for C ++

Question

I need to store a large number of integers. There may be duplicates in the input stream of integers, I just need to store individual among them. I used stl at first, but it exited OutOfMem when the number of input numbers of integers is too large. I am looking for some C ++ container library that let me store numbers with the specified requirement, possibly supported by the ie file, the container should not try to store all the numbers in memory. I do not need to constantly store this data, I just need to find unique values ​​among it.

Answer 1

Take a look at STXXL ; maybe what you are looking for.

Edit: I did not use it myself, but from the documents - you could use it stream::runs_creatorto create sorted runs of your data (as much as you want in memory), then stream::runs_mergerto combine sorted streams, and finally use it stream::uniqueto filter uniqueness.

Answer 2

Since you need more than RAM, you can look memcached

Answer 3

( SQLite)? ?

Answer 4

You should seriously at least try the database before completing it too slowly. All you need is one of the easiest key storage locations. I used to use Berkeley DB, but here is a list of others .