Geek Answers Handbook

I am trying to evaluate the memory requirements for my Dictionary (Of String, Long ())

I have a new dictionary (Of String, Long ()) with 3,125,000 unique (string) keys. I distribute about 1 billion (935,984,413) values ​​(of all lengths) among the keys and populate the long () array for each key.

This works fine and very fast for medium-sized data sets, say, for the 1,500,000 rows of keys and the 500,000,000 long values ​​to be distributed, this takes about 2 hours.

However, for the aforementioned dataset, as soon as I get half of my data, the process is very slow and will never end in the current trend ...

I think I'm running out of memory, the application uses 5 GB of memory, and I believe that now it is limited to my system (8 GB of RAM).

How can I calculate the amount of memory I need for the above situation? The size of the string keys on average about 5 characters.

Thank!

+4
source share
1 answer

LongThe data type is 8 bytes each. For stringit is more difficult. Check out this post by the famous John Skeet.

Quote:

In the current implementation, at least the strings occupy 20+ (n / 2) * 4 bytes (rounding the n / 2 value down)

(Note: in his blog post, he has some updates in this string calculation)

Given your case, each of your 5 characters stringwill cover:

20 + (5/2) * 4 = 20 + 8 = 28 bytes

, , - Long, , string, string (5 ).

, 1 Long, 8 , Long. + , 8 (935,984,413 x 8 = 7,487,875,304).

string, :

28 * 3,125,000 = 87.5 MB

, 7,5 ~ 7,6 string Long()

+4

More articles:


All Articles