I have a Django application in Production that works with Celery and Amazon SQS. Every day in celery logs I see that there was an SSL error:
[ERROR/MainProcess] Empty body: SQSError: 599 gnutls_handshake() failed: An unexpected TLS packet was received.
which follows the following error when trying to reconnect to the broker:
[2016-12-14 16:06:28,917: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection... Traceback (most recent call last): File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/celery/worker/consumer/consumer.py", line 318, in start blueprint.start(self) File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/celery/bootsteps.py", line 119, in start step.start(parent) File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/celery/worker/consumer/consumer.py", line 584, in start c.loop(*c.loop_args()) File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/celery/worker/loops.py", line 88, in asynloop next(loop) File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/hub.py", line 284, in create_loop poll_timeout = fire_timers(propagate=propagate) if scheduled else 1 File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/hub.py", line 137, in fire_timers entry() File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/timer.py", line 68, in __call__ return self.fun(*self.args, **self.kwargs) File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/timer.py", line 127, in _reschedules return fun(*args, **kwargs) File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/http/curl.py", line 108, in _timeout_check self._process_pending_requests() File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/http/curl.py", line 132, in _process_pending_requests self._process(curl, errno, reason) File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/http/curl.py", line 178, in _process buffer=buffer, effective_url=effective_url, error=error, File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/promises.py", line 146, in __call__ svpending(*ca, **ck) File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/promises.py", line 139, in __call__ return self.throw() File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/promises.py", line 136, in __call__ retval = fun(*final_args, **final_kwargs) File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/funtools.py", line 100, in _transback return callback(ret) File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/promises.py", line 139, in __call__ return self.throw() File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/promises.py", line 136, in __call__ retval = fun(*final_args, **final_kwargs) File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/funtools.py", line 98, in _transback callback.throw() File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/funtools.py", line 96, in _transback ret = filter_(*args + (ret,), **kwargs) File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/aws/connection.py", line 269, in _on_list_ready raise self._for_status(response, body) boto.exception.SQSError: SQSError: 599 gnutls_handshake() failed: An unexpected TLS packet was received.
Sometimes after this message, the queues are dropped, and I have to restart the celery workers. In general, I'm not sure that Celery tasks are being executed and are being executed correctly after receiving this error.
What is the best way to solve this problem?