Providing uniqueness for a sorted set in redis

Question

I am trying to save media objects and restore them for a certain period of time via redis. To do this, I selected the data type of the sorted set. I am adding elements such as:

zAdd: key: media:552672 score: 1355264694 zAdd: key: media:552672 score: 1355248565 zAdd: key: media:552672 score: 1355209157 zAdd: key: media:552672 score: 1355208992 zAdd: key: media:552672 score: 1355208888 zAdd: key: media:552672 score: 1355208815 

If the key is unique to the identifier of the location on which the media was removed, and the estimate is the time the media object was created. And the value is a json_decode media object.

When I switch to using zRevRangeByScore , there will sometimes be duplicate entries. I essentially use Redis as a buffer for the external API, if users make the same API call twice with X seconds, then I will get the results from the cache, otherwise I will add it to the cache, rather than checking to make sure it is already exists because of the definition of a set that does not contain duplicates. Possible known problems: If the attribute of a media object changes between caches, it will appear as a duplicate

Is there a better way to store this data type without checking on the redis client side?

TL; DR; What is the best way to store and retrieve objects in Redis, where you can select a range of objects by timestamp and make sure they are unique?

Answer 1

Lets make sure that we are talking about the same things, so here is the terminology for sorted Redis sets:

 ZADD key score member [score] [member] summary: Add one or more members to a sorted set, or update its score if it already exists 

• key - "name" of the sorted set
• score - score (in our case, timestamp)
• member - string associated with
• A sorted set has many members, each with a score

It looks like you are using the object's encoded JSON string as a member. The item is unique in the sorted set. As you say, if an object changes, it will be added as a new member to the sorted set. This is probably not what you want.

A sorted set is Redis's way of storing data by timestamp, but the member that is stored in the set is usually a "pointer" to another key in Redis.

From your description, I think you need this data structure:

• Sorted set that saves all media using the created timestamp
• String or hash for each unique medium

I recommend storing media objects in a hash, as this provides more flexibility. Example:

 # add some members to our sorted set redis 127.0.0.1:6379> ZADD media 1000 media:1 1003 media:2 1001 media:3 (integer) 3 # create hashes for our members redis 127.0.0.1:6379> HMSET media:1 id 1 name "media one" content "content string for one" OK redis 127.0.0.1:6379> HMSET media:2 id 2 name "media two" content "content string for two" OK redis 127.0.0.1:6379> HMSET media:3 id 3 name "media three" content "content string for three" OK 

There are two ways to retrieve data stored in this way. If you need to get members within specific time intervals (for example: the last 7 days), you will need to use ZREVRANGEBYSCORE to retrieve the members, and then scroll through them to get each hash using HGETALL or similar. See pipelining to find out how you can make a single call to the server loop.

 redis 127.0.0.1:6379> ZREVRANGEBYSCORE media +inf -inf 1) "media:2" 2) "media:3" 3) "media:1" redis 127.0.0.1:6379> HGETALL media:2 1) "id" 2) "2" 3) "name" 4) "media two" 5) "content" 6) "content string for two" 

If you want to get only the last n members (or, for example, the 10 most recent of the last 100), you can use SORT to get the elements. See the sorting documentation for syntax and how to extract different hash fields, limit results, and other parameters.

 redis 127.0.0.1:6379> SORT media BY nosort GET # GET *->name GET *->content1) DESC 1) "media:2" 2) "media two" 3) "content string for two" 4) "media:3" 5) "media three" 6) "content string for three" 7) "media:1" 8) "media one" 9) "content string for one" 

NB: sorting a sorted hash by account ( BY nosort ) only works with Redis 2.6.

If you plan to receive media within the last day, week, month, etc., I would recommend using a separate sorted set for each of them and using ZREMRANGEBYSCORE to delete old members. You can then use SORT on these sorted sets to retrieve data.