Nginx + Gunicorn + Flask & # 8594; permanent 500 errors after X hours of use

Question

I have a Flask application that works with Nginx + Gunicorn, usually without a problem. The application is still under development, so at the moment there is only one request per hour.

The problem is that Gunicorn suddenly crashes after 12-36 hours after the last restart. When this happens, nginx can still serve static files, but whatever it takes, Gunicorn starts to return 500 errors, even though the Gunicorn process is still running on the server. The problem is always fixed within the next 12-36 hours, restarting Gunicorn with sudo supervisorctl restart xxx (no nginx restart required). The problem occurred about 10 times. Is there any way to improve the logging or do something else?

Nginx conf (/ etc / nginx / sites-available / xxx_gunicorn):

 server { listen 80 default_server; listen [::]:80 default_server ipv6only=on; access_log /var/log/nginx/xxx-access.log; server_name 127.0.0.1 www.xxx.yy; location / { proxy_pass_header Server; proxy_set_header Host $http_host; proxy_redirect off; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header X-Scheme $scheme; proxy_connect_timeout 10; proxy_read_timeout 10; proxy_pass http://127.0.0.1:8000/; } location /static { alias /opt/xxx/static/; } } 

supervisor conf fires cannon fire (/etc/supervisor/conf.d/xxx.conf)

 [program:xxx] command = gunicorn xxx:app -b localhost:8000 --debug --log-level debug --log-file /var/log/gunicorn.log --error-logfile /var/log/gunicorn.error.log --workers 2 --worker-connections 1000 --max-requests 100 --timeout 30 directory = /opt/xxx user = ubuntu stdout_logfile = /var/log/gunicorn.log ; Where to write log messages autostart=true autorestart=true redirect_stderr=true 

xxx.py

 ... app = Flask(__name__) ... if __name__ == '__main__': app.debug=True app.run() 

/var/log/gunicorn.error.log : nothing is logged when 500 errors are triggered. No suspicious lines up to 500 errors.

/var/log/gunicorn.log : no suspicious lines.

/var/log/nginx/error.log : no suspicious lines.

/var/log/nginx/xxx-access.log : 500 errors are displayed here:

 80.221.255.134 - - [27/Jan/2015:07:01:50 +0000] "GET / HTTP/1.1" 500 291 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36" 

Answer 1

As I know, gunicorn does not have an option log file, only an access log file and an error log file:

http://gunicorn-docs.readthedocs.org/en/latest/settings.html#logging

But to register errors in your flash application, you need to install python logger in it. For example, this can be done as follows:

 from logging.handlers import WatchedFileHandler @app.before_first_request def setup_logging(): """ Setup logging """ handler = WatchedFileHandler("/var/log/your_flask_app.log") app.logger.addHandler(handler) app.logger.setLevel(logging.INFO)