Running multiple celery demons automatically using Jenkins

Question

I have a Ubuntu server configured on 5 different django sites running on it. They are used for testing, so each developer has his own site and database, as well as one site for integrated code, which is updated only when the functions are ready. Jenkins is used to update every site from Github when changes are pushed to the repository.

We recently added Django-Celery to our dependencies to process uploaded files asynchronously. Each site now needs its own celery line, which uses the correct settings (database, download directory, etc.) for this particular site.

I want to restart every celery server when the code changes so that it can receive the latest changes automatically. We have a script update in our git repository that Jenkins runs whenever it updates the site. When I try to start the celery daemon in this script, the celery begins, but ends again at the end of the script.

Here is a copy of my update script:

#!/bin/bash # Delete all *.pyc files find $WORKSPACE -name '*.pyc' | xargs rm # Update the database […] # Run automated tests python code/manage.py test <project> --noinput TEST_STATUS=$? # Refresh this repo public website touch $WORKSPACE/apache/wsgi.py # Restart our celery daemon for this installation /sbin/start-stop-daemon --stop -p $WORKSPACE/../celery.pid echo 'Starting Celery Server' # When run on the command line, this line starts a daemon just fine /sbin/start-stop-daemon --start --background --quiet --oknodo -p $WORKSPACE/../celery.pid -m --exec $WORKSPACE/code/manage.py -- celeryd --logfile=$WORKSPACE/../celery.log echo 'Celery Server Status: '$? exit $TEST_STATUS 

Here's a copy of the celery log during the execution of this script:

 [2011-05-10 20:45:41,286: WARNING/MainProcess] -------------- celery@ip-10-227-139-6 v2.2.6 ---- **** ----- --- * *** * -- [Configuration] -- * - **** --- . broker: djkombu.transport.DatabaseTransport:// guest@localhost :5672/ - ** ---------- . loader: djcelery.loaders.DjangoLoader - ** ---------- . logfile: /var/lib/jenkins/jobs/mpdaugherty-farmforce/workspace/../ celery.log@WARNING - ** ---------- . concurrency: 1 - ** ---------- . events: OFF - *** --- * --- . beat: OFF -- ******* ---- --- ***** ----- [Queues] -------------- . celery: exchange:celery (direct) binding:celery [2011-05-10 20:45:41,333: WARNING/MainProcess] celery@ip-10-227-139-6 has started. [2011-05-10 20:46:28,481: WARNING/MainProcess] celeryd: Warm shutdown (MainProcess) 

Any tips on how I can get the celery demons started by Jenkins to not close? Thank you very much!

Answer 1

One of my colleagues finally did the job. Instead of launching the Celery daemon directly, we use at to schedule it immediately and disconnect from the current shell.

Instead:

 # Restart our celery daemon for this installation /sbin/start-stop-daemon --stop -p $WORKSPACE/../celery.pid echo 'Starting Celery Server' # When run on the command line, this line starts a daemon just fine /sbin/start-stop-daemon --start --background --quiet --oknodo -p $WORKSPACE/../celery.pid -m --exec $WORKSPACE/code/manage.py -- celeryd --logfile=$WORKSPACE/../celery.log 

Change to:

 # Restart our celery daemon for this installation echo "/sbin/start-stop-daemon --stop -p $WORKSPACE/../celery.pid echo 'Starting Celery Server'" | at now # When run on the command line, this line starts a daemon just fine echo "/sbin/start-stop-daemon --start --background --quiet --oknodo -p $WORKSPACE/../celery.pid -m --exec $WORKSPACE/code/manage.py -- celeryd --logfile=$WORKSPACE/../celery.log" | at now