What are the appropriate settings for ZMQ when sending messages of 500 KB in 64 bytes?

Question

I have an application that periodically needs to send a snapshot of its current status, which will currently be submitted to approximately 500,000 64-byte messages. I had difficulty receiving these messages, which were sent and received quickly and reliably using ZMQ.

I used PUB / SUB on top of tcp to do this at the moment, but I am not bound to either the template or the protocol until it does the job. In my experiments, I focused on playing with sending and getting a high watermark, sending and receiving buffer settings, and adding a few sleeping places to the sending cycle to slow it down a bit. With settings that seemed very generous to me (500K HWM, 10MB buffers), and using only a loopback connection, messages are still not all received sequentially.

I am interested in what are the appropriate settings for these or other settings, and in a broader sense, how to talk about the impact of various settings.

Some additional information that may help provide an appropriate answer:

• Distribution is one for many. The expected number of recipients is about 20.

• Each message is a set of information about another financial instrument, which is observed simultaneously. In my opinion, the arguments can be combined into one large message (the totality of all messages logically makes up one complete snapshot) and for their separate (clients can potentially be interested in only some tools, and I think this will help to filter them more easily).

• The estimated message frequency is basically no faster than every 20 milliseconds, and no slower than 5 seconds. Where I am on land, performance considerations will probably influence (that is, how fast my server can actually download messages and how fast the data transfer rate will be overwhelming for clients).

Answer 1

Let me break it.

First, why does HWM not work:

HWM is not an exact limit, since internal buffers are filled and emptied by two separate threads, and the amount of available space can lag significantly when there is a lot of activity. The 0MQ zmq_setsockopt page says: "0MQ does not guarantee that the socket will receive the same number of ZMQ_SNDHWM messages, and the actual limit may be 60-70% lower depending on the message flow in the socket."

Secondly, why are you losing messages:

As you unload 0.5M (x 20) messages into socket buffers, you accidentally fall into HWM, and the behavior of the PUB socket should then discard messages that it cannot queue.

Third, how to solve this:

There is no reason to split the state into separate messages; the only justification for this would be if the state did not fit into the memory, which is easy to do. Send as multipart (ZMQ_SNDMORE); this creates one effective message, which occupies 1 slot in the outgoing buffer.

Then remove the 500W HWM limit and return the default value (1000), which will be more than sufficient.

Fourth, how to achieve better performance:

Obviously profile and improve your publisher and subscriber code as much as possible; these are the usual bottlenecks.

Then consider some form of message compression if it is sparse, and you can do it without too much processor cost. With 20 subscribers, you usually get more from network costs than you lose from processor cost.

Finally, if you increase the number of subscribers and this is a critical system, look at the PGM multicast, which will effectively eliminate network costs.

Answer 2

After a day of experimental semi-random with various combinations, I came to the following preliminary conclusions:

• Adding sleep statements to my send loop to limit message rates improves reliability with any set of parameters.

• Sending 500,000 messages as frames of a single message instead of 500K individual messages improves reliability.

• Using the epgm protocol rather than tcp achieves greater throughput.

• Using epgm, the multicast option should match the required message rate obtained using sleep operators.

• Increasing the label of water and buffers helps to increase reliability, but you need to increase both parameters and do this both on the client and on the server. If all is not performed in combination, this does not help. You must set it high enough to get any kind of reliability performed with individual messages (as opposed to frames of a single message). In this case, I did not get good results until I had a high water mark set to 1,000,000, and buffers up to 65 MB. (Twice the size of the set of messages I tried to send.) It was much higher than I instinctively thought to try. This event paused 5 seconds between each round of 500 thousand messages. If I bring the interval to 1 second, I had to push them even higher, to 4 times the size of one batch of messages.

• Using epgm, setting a recovery interval does not help much.