Celery quick cycle to check your tasks?

Question

#ready is populated with some obj
ready = set()
running = []
while ready or running:
    #send all tasks in ready
    while ready:
        #send celery task
        t = ready.pop()
        res = add.delay(t)
        running[res] = t

    while running:
         if breakout:
             break

         for run in running.iterkeys():
             if run.ready():
                 rtask = running[run]
                 result = run.get(interval=0.00001)
                 # put dependent tasks into ready
                 if rtask.depends:
                     for t in rtask.depends:
                         ready.add(t)

                 del running[run]
                 # we have tasks in ready now so break out of running and send out ready jobs
                 breakout = True
                 break

As the tasks get larger, my code has been checking the completion of tasks for too long, which makes sense because it is O (N)

Is there a faster way to check the execution of tasks sent to celery workers?

Answer 1

Why do you really need to check running tasks? You can make logic that does something after completing a task. For example, you can make a decorator.

, - (dict) . uuid, (, True). , - , - . , , False, ( N * (N-1) .