Elegantly set up Python logging in Django

Question

I have yet to find a way to configure Python logging with Django, which I'm pleased with. My requirements are pretty simple:

• Different log handlers for different events - that is, I want to be able to register in different files.
• Easy access to registrars in my modules. The module should be able to find your registrar without much effort.
• It should be easily applicable to command line modules. Parts of the system are stand-alone command line or daemon processes. Logging should be easy to use with these modules.

My current setup is to use the logging.conf file and do the setup in every module I register from. This is not true.

Do you have a logging setting that you like? Please explain in detail: how do you configure the configuration (do you logging.conf or install it in the code), where / when do you start the loggers, and how do you access them in your modules, etc.

Answer 1

The best way I've found so far is to initialize the logging setting in settings.py - nowhere else. You can either use the configuration file or do it programmatically in stages - it just depends on your requirements. The main thing is that I usually add handlers that I want to the root log, using levels and sometimes logging. Filters for receiving the events that I want for the corresponding files, console, system logs, etc. Of course, you can add handlers to any other registrars too, but in my experience there is no need for this.

In each module, I define a logger using

 logger = logging.getLogger(__name__) 

and use this to register events in the module (and, if I want to differentiate further), use the registrar, which is a descendant of the registrar created above.

If my application will be potentially used on a site that does not configure logging in settings.py, I define NullHandler somewhere like this:

 #someutils.py class NullHandler(logging.Handler): def emit(self, record): pass null_handler = NullHandler() 

and make sure that its instance is added to all the registrars created in the modules of my applications that use logging. (Note: NullHandler is already in the logging package for Python 3.1 and will be in Python 2.7.) So:

 logger = logging.getLogger(__name__) logger.addHandler(someutils.null_handler) 

This is to ensure that your modules play well on a site that does not configure logging in settings.py, and that you do not receive annoying messages "There are no handlers for XYZ log messages" (which warnings about potentially incorrectly configured logging).

Doing so meets your stated requirements:

• You can configure different log handlers for different events, as you are doing now.
• Easy access to registrars in your modules - use getLogger(__name__) .
• Easily applied to command line modules - they also import settings.py .

Update: Please note that from version 1.3 Django now includes support for logging .

Answer 2

I know this is already a resolved answer, but according to django> = 1.3 there is a new logging setting.

The transition from the old to the new is not automatic, so I decided to record it here.

And of course also django doc .

This is the base conf created by default using django-admin createproject v1.3 - mileage can be changed using the latest django versions:

 LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'handlers': { 'mail_admins': { 'level': 'ERROR', 'class': 'django.utils.log.AdminEmailHandler', } }, 'loggers': { 'django.request': { 'handlers': ['mail_admins'], 'level': 'ERROR', 'propagate': True, } } } 

This structure is based on the standard Python logging dictConfig , which dictates the following blocks:

• formatters - the corresponding value will be a dict, in which each key is a formatting identifier, and each value is a dict that describes how to configure the corresponding Formatter instance.
• filters - the corresponding value will be a dict, in which each key is a filter identifier, and each value is a dict that describes how to configure the corresponding filter instance.

• handlers - the corresponding value will be a dict, in which each key is the identifier of the handler, and each value is a dict that describes how to configure the corresponding Handler instance. Each handler has the following keys:

  • class (required). This is the fully qualified name of the handler class.
  • level (optional). Handler level.
  • formatter (optional). The formatting identifier for this handler.
  • filters (optional). A list of filter identifiers for this handler.

I usually do this at least:

• add .log file
• configure my applications to write to this log

It means:

 LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'formatters': { 'verbose': { 'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s' }, 'simple': { 'format': '%(levelname)s %(message)s' }, }, 'filters': { 'require_debug_false': { '()': 'django.utils.log.RequireDebugFalse' } }, 'handlers': { 'null': { 'level':'DEBUG', 'class':'django.utils.log.NullHandler', }, 'console':{ 'level': 'DEBUG', 'class': 'logging.StreamHandler', 'formatter': 'simple' }, # I always add this handler to facilitate separating loggings 'log_file':{ 'level': 'DEBUG', 'class': 'logging.handlers.RotatingFileHandler', 'filename': os.path.join(VAR_ROOT, 'logs/django.log'), 'maxBytes': '16777216', # 16megabytes 'formatter': 'verbose' }, 'mail_admins': { 'level': 'ERROR', 'filters': ['require_debug_false'], 'class': 'django.utils.log.AdminEmailHandler', 'include_html': True, } }, 'loggers': { 'django.request': { 'handlers': ['mail_admins'], 'level': 'ERROR', 'propagate': True, }, 'apps': { # I keep all my of apps under 'apps' folder, but you can also add them one by one, and this depends on how your virtualenv/paths are set 'handlers': ['log_file'], 'level': 'INFO', 'propagate': True, }, }, # you can also shortcut 'loggers' and just configure logging for EVERYTHING at once 'root': { 'handlers': ['console', 'mail_admins'], 'level': 'INFO' }, } 

change

See exception requests are now always logged and Ticket # 16288 :

I updated the above conf example to explicitly enable the correct filter for mail_admins so that by default emails are not sent when debug is True.

You must add a filter:

 'filters': { 'require_debug_false': { '()': 'django.utils.log.RequireDebugFalse' } }, 

and apply it to the mail_admins handler:

  'mail_admins': { 'level': 'ERROR', 'filters': ['require_debug_false'], 'class': 'django.utils.log.AdminEmailHandler', 'include_html': True, } 

Otherwise, django.core.handers.base.handle_uncaught_exception does not send errors to the django.request log if the .DEBUG parameters are True.

If you do not do this in Django 1.5, you will get

DeprecationWarning: you have no filters defined in the mail_admins log handler: adding an implicit filter debug-false-only

but everything will work correctly BOTH in django 1.4 and django 1.5.

** end edit **

This conf is heavily inspired by the conf pattern in the django doc, but adds a portion of the log file.

I also often do the following:

 LOG_LEVEL = 'DEBUG' if DEBUG else 'INFO' ... 'level': LOG_LEVEL ... 

Then in my Python code I always add NullHandler in case no logging protocol is defined. This avoids warnings that a handler is not specified. Especially useful for libraries that are not necessarily called only in Django ( ref )

 import logging # Get an instance of a logger logger = logging.getLogger(__name__) class NullHandler(logging.Handler): #exists in python 3.1 def emit(self, record): pass nullhandler = logger.addHandler(NullHandler()) # here you can also add some local logger should you want: to stdout with streamhandler, or to a local file... 

[...]

 logger.warning('etc.etc.') 

Hope this helps!

Answer 3

Currently, I am using the registration system that I created myself. It uses CSV format for logging.

django-csvlog

This project is not yet complete documentation, but I'm working on it.

Answer 4

We initialize the logging at the top level of urls.py using the logging.ini file.

The logging.ini location logging.ini provided in settings.py , but that’s it.

Each module then performs

 logger = logging.getLogger(__name__) 

To distinguish between test, development, and production instances, we have different logging.ini files. For the most part, we have a "console log" that only goes to stderr with errors. We have an “application log” that uses a regular rolling log file that is sent to the log directory.