Celery mix reset periodically with error message:
[Errno 104] Connection reset by peer
Currently, I am losing the task and everything that was called up until I reset the server. I have 2 questions: (1) What causes the error / how can I prevent it? (2) How can I catch an error when it occurs and repeat my tasks?
I'm running to Hereka. Celery is configured with AMQP (CloudAMQP) as a broker and redis (REDIS_TO_GO) as a backend of results.
Stacktrace:
File "celery/task/trace.py", line 224, in trace_task R = retval = fun(*args, **kwargs) File "myapp/tasks.py", line 310, in mytask1 mytask2.delay(message) File "celery/app/task.py", line 343, in delay return self.apply_async(args, kwargs) File "celery/app/task.py", line 458, in apply_async with app.producer_or_acquire(producer) as P: File "python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "celery/app/base.py", line 256, in producer_or_acquire with self.amqp.producer_pool.acquire(block=True) as producer: File "kombu/connection.py", line 712, in acquire R = self.prepare(R) File "kombu/pools.py", line 57, in prepare p.revive(connection.default_channel) File "kombu/connection.py", line 626, in default_channel self._default_channel = self.channel() File "kombu/connection.py", line 167, in channel chan = self.transport.create_channel(self.connection) File "kombu/transport/amqplib.py", line 324, in create_channel return connection.channel() File "kombu/transport/amqplib.py", line 243, in channel return Channel(self, channel_id) File "kombu/transport/amqplib.py", line 267, in __init__ super(Channel, self).__init__(*args, **kwargs) File "amqplib/client_0_8/channel.py", line 82, in __init__ self._x_open() File "amqplib/client_0_8/channel.py", line 469, in _x_open self._send_method((20, 10), args) File "amqplib/client_0_8/abstract_channel.py", line 76, in _send_method method_sig, args, content) File "amqplib/client_0_8/method_framing.py", line 252, in write_method self.dest.write_frame(1, channel, payload) File "amqplib/client_0_8/transport.py", line 165, in write_frame frame_type, channel, size, payload, 0xce)) File "python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args)
Versions:
Django==1.4.1 django-celery==3.0.9 celery==3.0.9 celery-with-redis==3.0 amqplib==1.0.2 kombu==2.4.7