Semaphore-like mechanism for celery

Question

We are developing a distributed application in Python + Celery for our task queue.

Our application requires that we download emails from a remote Internet provider via IMAP (for example, gmail), and we want this task to be performed in parallel. For this email account you are provided with a limited number of connections for simulation, so we need a way to atomically track our active connections for all downloadable accounts.

I found several examples of atomic locks for Celery using Redis, but not one of them that can track a pool of limited resources like this, and all attempts to implement our own, have led to difficult to debug race conditions, our locks are intermittently never released.

Answer 1

Since celery uses a multiprocessing library for processes, you should be able to use the safe multiprocessing.Semaphore([value]) process.

You need to create a preliminary semaphore and pass it, and you can set a default value equal to the maximum number of concurrent accesses that you want to allow. Then purchase before connecting IMAP and release after disconnecting.