Redis publish-subscribe: Is radish guaranteed to deliver a message even under massive stress?

Question

Provided that both the client has signed and the server publishing the message will remain connected, is Redis guaranteed to always provide the published message to the signed client in the end, even in situations where the client and / or server are under severe stress? Or should I plan for the possibility that Redis can sometimes drop messages as things get hot?

Answer 1

Redis absolutely does not provide guaranteed delivery for publication and subscription traffic. This mechanism is based only on sockets and event loops; there is no queue (even in memory). If the subscriber does not listen during publication, the event will be lost for that subscriber.

You can implement some guaranteed delivery mechanisms on top of Redis, but not with the publish and subscribe APIs. List data type in Redis can be used as a queue and as the basis for more advanced queuing systems, but it does not provide multicast capabilities (therefore, it is not published or subscribed).

AFAIK, there is no obvious way to easily implement publication and subscription and guaranteed delivery at the same time as Redis.

Answer 2

Redis does not provide guaranteed delivery using its Pub / Sub engine. Moreover, if the subscriber does not actively listen to the channel, he will not receive messages that would be published.

Earlier, I wrote a detailed article that described how you can use Redis lists in conjunction with BLPOP to implement reliable pub / sub multicast:

http://blog.radiant3.ca/2013/01/03/reliable-delivery-message-queues-with-redis/

For the record here is a high-level strategy:

• When each consumer starts up and prepares to consume messages, he registers by adding himself to the set representing all consumers registered in the queue.
• When manufacturers publish a message in a queue, it:
  • Saves message content in a Redis button
  • Iterates over the set of consumers registered in the queue and pushes the message identifier in the list for each registered user
• Each consumer is constantly looking for a new entry in his consumer-oriented list, and when he arrives, deletes the entry (using the BLPOP operation), processes the message, and proceeds to the next message.

I also made the Java implementation of these principles available open source: https://github.com/davidmarquis/redisq

These principles were used to process about 1000 messages per second from one instance of Redis and two instances of the consumer application, each of which consumes messages with 5 threads.