Get employee ID in celery

Question

I want to use Celery to run jobs on a GPU server with four Tesla cards. I manage a celery worker with a pool of four workers, so each card always does one job.

My problem is how to instruct workers on each requirement for one GPU. Currently, I rely on the assumption that workflows should have continuous process identifiers:

device_id = os.getpid() % self.ndevices 

However, I am not guaranteed to always work, i.e. when workflows restart over time. Therefore, ideally, I would like to get the identifier of each employee directly. Can someone tell me if it is possible to inspect an employee from a task or offer another solution for distributing tasks among GPUs?

Answer 1

If you use CELERYD_POOL = 'processes' , the work pool is processed by billiard , which, as it turns out, exposes its process index based on 0:

 from billiard import current_process from celery import task @task def print_info(): # This will print an int in [0..concurrency[ print current_process().index 

index based on 0, and if the worker reboots, it will store its index.

I could not find the documentation regarding the index value though: /