Geek Answers Handbook

Celery rate_limit not observed when planning with the eta option

I would like to schedule the task at different times in the future. I am trying to use the eta parameter when invoking a task for this. This part works fine on its own, but at the same time I would like to have a speed limit for the task. I can not do both: plan the task for launch in the future, but also get celery to comply with the speed limit.

Here is an example (rate_limit_eta.py)

from celery import Celery
from datetime import datetime, timedelta

app = Celery('rate_limit_eta', broker='amqp://guest@localhost//')

@app.task(rate_limit='4/m')
def my_task():
    print 'execution timestamp: {}'.format(datetime.utcnow())


if __name__ == '__main__':

    now = datetime.utcnow()
    d = now + timedelta(seconds=5)
    print now

    for i in range(10):
        my_task.apply_async(eta=d)

In a loop, when I call 'my_task' with eta = d, tasks are completed as quickly as possible. Rate_limit is not respected.

my_task.apply_async(eta=d)


[2015-06-12 13:45:15,750: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.748498
[2015-06-12 13:45:15,757: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.757757
[2015-06-12 13:45:15,760: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.759992
[2015-06-12 13:45:15,763: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.763614
[2015-06-12 13:45:15,766: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.765996
[2015-06-12 13:45:15,768: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.768424
[2015-06-12 13:45:15,771: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.771079
[2015-06-12 13:45:15,774: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.774561
[2015-06-12 13:45:15,777: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.777425
[2015-06-12 13:45:15,780: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.780207

But if I do not plan the task (i.e. without eta), then the speed limit applies.

my_task.apply_async()

[2015-06-12 13:47:05,844: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:05.843872
[2015-06-12 13:47:05,850: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:05.850502
[2015-06-12 13:47:21,919: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:21.919120
[2015-06-12 13:47:35,937: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:35.937060
[2015-06-12 13:47:51,959: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:51.959174
[2015-06-12 13:48:05,976: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:05.976197
[2015-06-12 13:48:21,998: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:21.998254
[2015-06-12 13:48:36,014: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:36.014837
[2015-06-12 13:48:52,041: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:52.041326
[2015-06-12 13:49:06,057: WARNING/Worker-1] execution timestamp: 2015-06-12 13:49:06.057286

Am I doing something wrong?

+4
source share
1

, . , , - :

if req.eta:
    apply_eta()
else:
    check_if_rate_limited()

, - , ETA 2 , , . , , , ETA.

v3 Celery ( , v3), . , - (?), !

UPDATE: .

+1

More articles:


All Articles