Maximum customers achieved at Heroku and Redistogo Nano

Question

I am using celerybeat on Heroku with RedisToGo Nano addon

There is one web dyno and one working dyno

A celerybeat worker must complete every task every minute.

Problem: whenever I deploy a new commit, reboot dynos and I get this error

2014-02-27T13:19:31.552352+00:00 app[worker.1]: Traceback (most recent call last): 2014-02-27T13:19:31.552352+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 389, in start 2014-02-27T13:19:31.552352+00:00 app[worker.1]: self.reset_connection() 2014-02-27T13:19:31.552352+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 727, in reset_connection 2014-02-27T13:19:31.552352+00:00 app[worker.1]: self.connection = self._open_connection() 2014-02-27T13:19:31.552352+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 792, in _open_connection 2014-02-27T13:19:31.552352+00:00 app[worker.1]: callback=self.maybe_shutdown) 2014-02-27T13:18:23.864287+00:00 app[worker.1]: self.on_connect() 2014-02-27T13:18:23.864287+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 263, in on_connect 2014-02-27T13:18:23.864287+00:00 app[worker.1]: if nativestr(self.read_response()) != 'OK': 2014-02-27T13:18:23.864287+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 314, in read_response 2014-02-27T13:18:23.864287+00:00 app[worker.1]: raise response 2014-02-27T13:18:23.864287+00:00 app[worker.1]: ResponseError: max number of clients reached 2014-02-27T13:19:31.552352+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 272, in ensure_connection 2014-02-27T13:19:31.552352+00:00 app[worker.1]: interval_start, interval_step, interval_max, callback) 2014-02-27T13:19:31.552591+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/__init__.py", line 218, in retry_over_time 2014-02-27T13:19:31.552591+00:00 app[worker.1]: return fun(*args, **kwargs) 2014-02-27T13:19:31.552591+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 162, in connect 2014-02-27T13:19:31.552591+00:00 app[worker.1]: return self.connection 2014-02-27T13:19:31.552591+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 617, in connection 2014-02-27T13:18:23.870811+00:00 app[worker.1]: [2014-02-27 13:18:23,870: ERROR/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection... 

and these magazines go on forever. until I stop both speakers and restart them.

This became a problem because it happens almost every time a new commit is deployed.

Any ideas why this is happening and how to solve it?

Answer 1

The plans of the nano redistogo plan accompany repeated connections at 10 .

The number of redis connections used will depend on the desktop and desktop settings. It looks like you are using > = 5 redis connections for your production stack.

When deploying new code, Heroku creates a completely new stack. This means that you use > = 10 redis connections during deployment.

There are two ways to fix this:

• Increase the maximum number of redistogo allowed connections by moving to a larger plan ($$$)
• Reduce the number of connections used for your stack (reduce the number of concurrency celery or recycle the connections used by your web worker).

This is a simple matter of running out of resources. I would just pay for a larger RedisToGo plan.