What is a good way to manage keys in a keystore?

Question

Trying to define some policy for keys in the key value store (we use Redis). Key space should be:

• Shardable (can enter more servers and distribute a key space between them)

• Namespaced (there should be some mechanism for “grouping” keys together logically, for example, by domain or related concepts)

• Effective (try to use as little space in the key database as possible to provide as much data as possible)

• As few conflicts as possible (avoid keys for two different objects)

The two alternatives that I have reviewed are as follows:

• Use prefixes for namespaces separated by some sort of character (e.g. human_resources:person:<some_id> ). The surface of this is that it is quite scalable and understandable. The downside may be conflicts depending on the delimiter (what if id has a character in it:?) And, possibly, size efficiency (too many nested name spaces can create very long keys).

• Use some data structure (e.g. Ordered Set or Hash) to store namespaces. The main disadvantage of this may be the loss of “fragments”, since the structure for storing namespaces must be in the same database.

Question What would be a good way to manage key space in a fine tuning? Should we use one of these alternatives, or is there some other, better model that we have not considered?

Many thanks!

Answer 1

A common convention in the Redis world is option 1 - that is, namespaces separated by a character, such as a colon. However, namespaces are almost always on the same level. For example: person:12321 instead of human_resources:person:12321 .

How does it work with your 4 recommendations?

Facing . This approach is scalable. Each key can fall into another shard or shard, depending on how you configured it.

Namespace Namespace as a way to avoid collisions works with this approach. However, the namespace as a way of grouping keys does not work. In general, using keys as a way to group data is a bad idea. For example, what if a person moves from department to another? If you change the key, you will have to update all the links - and that gets complicated.

It is best to ensure that the key is never changed for the object. Grouping can be processed externally by creating a separate index.

For example, let's say you want to group people by department, by salary range, by location. Here's how you do it -

• Individual people go in a separate hash with keys persons:12321
• Create a set for each group - For example: persons_by:department - and save only the numeric identifiers for each person in this set. For example, [12321, 43432]. Thus, you get the benefits of the Redis Integer Set

The Effective Method described above is quite memory efficient. To save some more memory, you can compress the keys further from the application side. For example, you can save p:12321 instead of persons:12321 . You should only do this if you have determined by profiling that you need such memory savings. All in all, it's not worth the cost.

Collision Free . It depends on your application. Each user or person must have a primary key that never changes. Use this in your Redis key and you will not have collisions.

You mentioned two problems with this approach, and I will try to address them

What if the identifier has a colon?

This, of course, is possible, but your application design should prevent it. It is better not to allow special characters in identifiers - because they will be used on many systems. For example, an identifier is likely to be part of a URL, and a colon is a reserved character even for URLs.

If you really need to allow special characters in your identifier, you will have to write a small wrapper in code that encodes special characters. URL coding does a great job of this.

Size Efficiency

There is a cost to long keys, however this is not so much. In general, you should worry about the data size of your values, not the keys. If you think the keys consume too much memory, profile the database with a tool such as redis-rdb-tools.

If you determine that key size is a problem and you want to save memory, you can write a small wrapper that overwrites the keys with an alias.