Celery - minimizes memory consumption

Question

We have ~ 300 celeryd processes running under Ubuntu 10.4 64-bit, in standby mode each process takes ~ 19 MB RES, ~ 174 MB VIRT, so it is about 6 GB of standby RAM for all processes. In the active state, the process takes up to 100 MB RES and ~ 300 MB VIRT

Each process uses minidom (xml files - <500kb, simple structure) and urllib.

Quetions - how can we reduce RAM consumption - at least for inactive workers, maybe some celery or python options may help? How to determine which part takes up most of the memory?

UPD:, these are flight search agents, one worker for one agency / date. We have 10 agencies, one user search == 9 dates, so we have 10 * 9 agents for one user search.

Is it possible to start celeryd processes on demand to avoid idle workers (something like MaxSpareServers on apache)?

UPD2: Agent life cycle - send an HTTP request, wait for a response ~ 10-20 seconds, parse xml (takes less than 0.02s), save the result in MySQL

Answer 1

Read this:

http://docs.celeryproject.org/en/latest/userguide/workers.html#concurrency

It looks like you have one worker for each celery. This seems wrong. You should have dozens of workers for each celery. Continue to increase the number of workers (and reduce the number of celery trees) until your system is very busy and very slow.

Answer 2

S. Lott is right. The primary instance consumes messages and delegates them to worker pool processes. It probably doesn't make sense to run 300 pools on the same machine! Try 4 or 5 times the number of processor cores. You can get something by running more than celeryd with several processes each, some people, but you will have to experiment for your application.

See http://celeryq.org/docs/userguide/workers.html#concurrency

In the upcoming release 2.2, we are working on supporting the Eventlet pool, this may be a good alternative for tasks related to IO, which will allow you to run more than 1000 threads with minimal memory overhead, but it is still experimental and errors are fixed for the final release.

See http://groups.google.com/group/celery-users/browse_thread/thread/94fbeccd790e6c04

The release of version 2.2 also has autoscaling support, which adds / removes the process on demand. See Changelog: http://ask.github.com/celery/changelog.html#version-2-2-0 (this changelog has not yet been fully written)

Answer 3

The natural number of workers is close to the number of cores that you have. Workers are there, so intense tasks can efficiently use the entire core. There is a broker so that requests that do not have a worker to process them are stored in the queue. The number of queues may be high, but this does not mean that you also need a large number of brokers. A single broker should be sufficient, or you can fine the queue to one broker per machine, if you later get a quick work queue interaction. [/ P>

Your problem seems unrelated to this. I assume that your agencies do not provide an api message queue, and you must abide by many requests. If so, you need several (highlighting not many) event processes, for example, twisted or node.js.

Answer 4

Use autoscaling. This allows you to increase or decrease the number of workers in each instance of celeryd if necessary. http://docs.celeryproject.org/en/latest/userguide/workers.html#autoscaling