Effectiveness of using redis

Question

I want to load data with 4 columns and 80 million rows into MySQL on Redis so that I can reduce the sampling delay.

However, when I try to download all the data, it gets 5 times bigger.

The original data was 3gb (when exporting to csv format), but when I upload it to Redis, it takes 15 GB ... it is too large for our system.

I also tried different data types -

1) 'table_name: row_number: column_name' → string 2) 'table_name: row_number' → hash

but they are all too much.

Am I missing something?

added)

my data has 4 col - (user id (pk), account, creation time and date)

Answer 1

The most efficient way to store memory is to store the values ​​as a json array and split your keys in such a way that you can store them using a zisp-encoded hash.

• Encode your data using say json array, so you have key = value pairs, such as user:1234567 -> [21,'25-05-2012','14-06-2010'] .
• Divide your keys in two so that the second part has about 100 possibilities. For example, user:12345 and 67
• Store this combo key in a hash like this hset user:12345 67 <json>
• To get user data for user id 9876523, just do hget user:98765 23 and hget user:98765 23 json array
• Be sure to configure hash-max-ziplist-entries and hash-max-ziplist

Instagram has written a great blog explaining this method , so I will skip the explanation why it is effective memory.

Instead, I can tell you about the disadvantages of this technique.

• You cannot access or update a single attribute for a user; you have to rewrite the whole record.
• You will need to retrieve the entire json object always, even if you only need some fields.
• Finally, you must write this logic for key separation, which adds service.

As always, this is a compromise. Define your access patterns and see if such a structure makes sense. If not, you will have to buy more memory.