Celery: stuck in endlessly repeating timeouts (UP message timeout)

Question

I defined some tasks with a time limit of 1200:

@celery.task(time_limit=1200) def create_ne_list(text): c = Client() return c.create_ne_list(text) 

I also use the worker_process_init signal to initialize, each time a new process starts:

 @worker_process_init.connect def init(sender=None, conf=None, **kwargs): init_system(celery.conf) init_pdf(celery.conf) 

This initialization function takes a few seconds.

In addition, I use the following configuration:

 CELERY_RESULT_SERIALIZER = 'json' CELERY_TASK_SERIALIZER = 'json' CELERY_ACCEPT_CONTENT = ['json'] BROKER_URL = 'amqp://' CELERY_RESULT_BACKEND = 'amqp://' CELERY_TIMEZONE = 'Europe/Berlin' CELERY_ENABLE_UTC = True 

and run my worker with the following command:

 celery -A isc worker -l info --concurrency=3 

As expected, starting an employee causes the initialization function to be called three times. Now I can send tasks, and they are executed, and everything works fine.

BUT: As soon as the task exceeds the deadline, the worker enters an endless spawning cycle and is killed again due to the excess of the deadline.

 [2014-06-13 09:46:18,978: ERROR/MainProcess] Timed out waiting for UP message from <Worker(Worker-20381, started daemon)> [2014-06-13 09:46:20,000: ERROR/MainProcess] Process 'Worker-20381' pid:18953 exited with 'signal 9 (SIGKILL)' // new worker 20382 getting started, initialization getting triggerd and soon after that --> [2014-06-13 09:46:18,978: ERROR/MainProcess] Timed out waiting for UP message from <Worker(Worker-20382, started daemon)> [2014-06-13 09:46:20,000: ERROR/MainProcess] Process 'Worker-20382' pid:18954 exited with 'signal 9 (SIGKILL)' // and so on.... 

Does anyone have an idea why this is happening?

Answer 1

The answer seems to be that the worker_process_init signal requires that the handler does not block for more than 4 seconds.

http://celery.readthedocs.org/en/latest/userguide/signals.html#worker-process-init

Since my init function takes longer to execute, the worker automatically terminates. After that, it naturally reboots and starts the init function again, which leads to the fact that the worker exits again, etc.