Consuming rabbitmq multi-thread message queue (Python Kombu)

Question

I have one RabbitMQ exchange with one queue. I want to create a daemon that starts several threads and runs through this queue as quickly as possible.

"Work" is associated with communication with external services, so within each consumer there will be a significant blockage. Thus, I want to have multiple threads dealing with messages from the same queue.

I can achieve this by consuming a queue in my main thread and then farming the incoming work into a pool of other threads, but is there a way to start multiple consumers, each in its own threading context?

Answer 1

Take a look at celery - it is designed to queue and process tasks from the AMPQ broker (but also works with other queue servers). It handles multi-processor or multi-threaded concurrency and simplifies the creation and use of tasks.