Geek Answers Handbook

RabbitMQ: throttling a fast producer against large queues with a slow consumer

We are currently using RabbitMQ, where a continuous super-fast producer is paired with a consumer limited by a limited resource (for example, slow MySQL inserts).

We do not like to queue with x-max-length, since all messages will be dropped or dead as soon as the limit is reached, and we do not want to lose messages.

Adding more consumers is easy, but they will all be limited to one common resource, so this will not work. The problem still remains: how to slow down the manufacturer?

Of course, we could set the flow control flag in Redis, memcached, MySQL or something else that the producer reads as in response to a similar question , or maybe it is better, the producer can periodically check the queue length and throttle itself, but they seem hacked to me.

I basically doubt if I have a fundamental misunderstanding. I expected this to be a common scenario, and so I wonder:

What is the best practice for throttling manufacturers? How is this done with RabbitMQ? Or are you doing it completely differently?

Background

Suppose that the manufacturer really knows how to slow down the work with the correct input. For instance. a hardware sensor or random number generator of equipment that can generate as many events as necessary.

API, . , , , API , "", / , , API , . , , , .

- API TCP, write() select() , , . , RabbitMQ API, , .

+4
1

, . , , ( ), , - .

, , , . ( , ?), ( MQ ).

, . , (, - , ), / (: , CRM). , , , , - .

, ( , , - ) .

IF eventsInTimePeriod/timePeriod > estimatedConsumerBandwidth
THEN LowerRate()
ELSE RiseRate()

, , .. ( ), , , .

, " ?" , : , , AddMessage:

AddMessage(message)
    WHILE(getQueueLength() > maxAllowedQueueLength)
        spin(1000); // or sleep or whatever
    mqAdapter.AddMessage(message)

, - .

+2

More articles:


All Articles