Geek Answers Handbook

Performing two tasks at the same time as celery

I am testing celery in a local environment. My Python file has the following two lines of code:

celery_app.send_task('tasks.test1', args=[self.id], kwargs={})
celery_app.send_task('tasks.test2', args=[self.id], kwargs={})

Looking at the console output, they seem to be executed sequentially one after another. But test2 only works after test1 completes. At least that's what reading console output looks like.

These tasks have no dependencies on each other , so I do not want one task to wait for the completion of another before moving on to the next line.

How can I complete both tasks at the same time?

---- **** -----
--- * ***  * -- Darwin-14.0.0-x86_64-i386-64bit
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         tasks:0x104cd8c10
- ** ---------- .> transport:   sqs://123
- ** ---------- .> results:     disabled
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery
+4
source share
2 answers

There are several ways to achieve this.

1. Single Worker - Single Queue.

$ celery -A my_app worker -l info  -c 2 -n my_worker

, 2 .

2. - .

$ celery -A my_app worker -l info  -c 1 -n my_worker1
$ celery -A my_app worker -l info  -c 1 -n my_worker2

, . , .

3. - .

$ celery -A my_app worker -l info  -c 1 -n my_worker1 -Q queue1
$ celery -A my_app worker -l info  -c 1 -n my_worker2 -Q queue2

, . .

celery_app.send_task('tasks.test1', args=[self.id], kwargs={}, queue='queue1')
celery_app.send_task('tasks.test2', args=[self.id], kwargs={}, queue='queue2')
+8

--autoscale, .

--autoscale AUTOSCALE
                       Enable autoscaling by providing max_concurrency,
                       min_concurrency. Example:: --autoscale=10,3 (always
                       keep 3 processes, but grow to 10 if necessary)

.

celery -A sandbox worker --autoscale=10,0 --loglevel=info
+1

More articles:


All Articles