Geek Answers Handbook

Redis is the best way to save a large map (dictionary)

What I need to do is keep the one-to-one mapping. A data set consists of a large number of key-value pairs of the same type (10M +). For example, you can use one instance of a HashMap object in Java to store such data.

The first way to do this is to store many key-value pairs, for example:

SET map:key1 value1 ... SET map:key900000 value900000 GET map:key1

The second option is to use a single "hash":

 HSET map key1 value ... HSET map key900000 value900000 HGET map key1

Redis Hashes have some convenient commands ( HMSET , HMGET , HGETALL , etc.) and they do not pollute the key space, so this looks like the best option. However, are there any performance or memory considerations when using this approach?

+7
redis
source share
1 answer

Yes, as Itamar Haber says you should look at the redis memory optimization guide . But you should also keep in mind such things (in a few lines):

  • Prefer HSET besides KEYS. Redis consumes most of the memory only in the key playpen. In simple (and rough) 1 HSETs with 1,000,000 keys consume up to 10x less memory, and then 1,000,000 keys with a single value.
  • Keep the HSET size less than hash-max-zipmap-entries and a valid hash-max-zipmap-value if the main purpose is memory. Remember to understand what hash-max-zipmap-entries and hash-max-zipmap-value . It will also take some time to read about the ziplist.
  • Until you actually have the right to process hash-max-zipmap-entries using the 10M + keys (to slow down access to these keys), you must split one HSET in some slots. For example, you set hash-max-zipmap-entries as 10,000. So, to store 10M + keys, you need 1000+ HSET keys with 10,000 each. For an example, crc32 (key)% maxHsets.
  • Read about strings in redis and use KEY names (in HSET) based on real memory management for this structure. With a simple key length of up to 7 bytes, you spend 16 bytes per key, but an 8-byte key spends 48 bytes. What for? Read about simple dynamic lines .

It may be useful to read:

+9
source share

More articles:


All Articles