Django - Rotating file handler gets stuck when file is maxBytes

Question

I have a problem with RotatingFileHander with Django.

The problem is that when the file size does not exceed the maximum size, it will not create a new file and give an error message when trying to make logger.info ("any message"):

Strange part:

• No one shares registrars, they must have their own journal, celery tasks have their own registrars.

• Logs are run only once at the top of the file (chartLogger = getLogger ...) Different functions in the same file will use the same name

  Logged from file views.py, line 1561 Traceback (most recent call last): File "C:\Python27\lib\logging\handlers.py", line 77, in emit self.doRollover() File "C:\Python27\lib\logging\handlers.py", line 142, in doRollover os.rename(self.baseFilename, dfn) WindowsError: [Error 32] The process cannot access the file because it is being used by another process 

Inside my .py settings, I have:

  LOGGING = { 'version': 1, 'disable_existing_loggers': True, 'formatters' : { 'standard' : { 'format' : '%(asctime)s [%(levelname)s] %(name)s: %(message)s' }, }, 'handlers': { 'celery.webapp' : { 'level' : 'ERROR', 'class' : 'django.utils.log.AdminEmailHandler', }, 'celery' : { 'level' : 'INFO', 'class' : 'logging.handlers.RotatingFileHandler', 'filename' : 'logs/celery.log', 'maxBytes' : 1024*1024*10, # 10MB 'backupCount' : 10, 'formatter' : 'standard', }, 'views.error' : { 'level' : 'ERROR', 'class' : 'django.utils.log.AdminEmailHandler', }, 'views' : { 'level' : 'INFO', 'class' : 'logging.handlers.RotatingFileHandler', 'filename' : 'logs/views.log', 'maxBytes' : 1024*1024*10, # 10MB 'backupCount' : 10, 'formatter' : 'standard', }, }, 'loggers': { 'celery.webapp' : { 'level' : 'ERROR', 'handlers' : ['celery.webapp'], 'propogate' : True, }, 'celery.webapp.task' : { 'level' : 'INFO', 'handlers' : ['celery'], 'propogate' : True, }, 'views.logger' : { 'level' : 'ERROR', 'handlers' : ['views.error'], 'propogate' : True, }, 'views.logger.login' : { 'level' : 'INFO', 'handlers' : ['views'], 'propogate' : True, }, 'views.logger.register' : { 'level' : 'INFO', 'handlers' : ['views'], 'propogate' : True, }, 'views.logger.chartConfigure' : { 'level' : 'INFO', 'handlers' : ['views'], 'propogate' : True, }, 'views.logger.sendEmail' : { 'level' : 'INFO', 'handlers' : ['views'], 'propogate' : True, }, }, } 

I tried to resize different file sizes, but it gets stuck in maxBytes.

Although he said that the process cannot access the file because it is used by some other processes. All logging is fine before it gets maxBytes.

EDIT:

I split the registration between celery and django.

 LOGGING = { 'version': 1, 'disable_existing_loggers': True, 'formatters' : { 'standard' : { 'format' : '%(asctime)s [%(levelname)s] %(name)s: %(message)s' }, }, 'handlers': { 'celery.webapp' : { 'level' : 'ERROR', 'class' : 'django.utils.log.AdminEmailHandler', }, 'celery' : { 'level' : 'INFO', 'class' : 'logging.handlers.RotatingFileHandler', 'filename' : 'logs/celery.log', 'maxBytes' : 1024*1024*10, # 10MB 'backupCount' : 10, 'formatter' : 'standard', }, 'celery_chartConfigure' : { 'level' : 'INFO', 'class' : 'logging.handlers.RotatingFileHandler', 'filename' : 'logs/celery_chartConfigure.log', 'maxBytes' : 1024*1024*10, # 10MB 'backupCount' : 10, 'formatter' : 'standard', }, 'celery_register' : { 'level' : 'INFO', 'class' : 'logging.handlers.RotatingFileHandler', 'filename' : 'logs/celery_register.log', 'maxBytes' : 1024*1024*10, # 10MB 'backupCount' : 10, 'formatter' : 'standard', }, 'views.error' : { 'level' : 'ERROR', 'class' : 'django.utils.log.AdminEmailHandler', }, 'views' : { 'level' : 'INFO', 'class' : 'logging.handlers.RotatingFileHandler', 'filename' : 'logs/views.log', 'maxBytes' : 1024*1024*10, # 10MB 'backupCount' : 10, 'formatter' : 'standard', }, 'views_login' : { 'level' : 'INFO', 'class' : 'logging.handlers.RotatingFileHandler', 'filename' : 'logs/views_login.log', 'maxBytes' : 1024*1024*10, # 10MB 'backupCount' : 10, 'formatter' : 'standard', }, 'views_sendEmail' : { 'level' : 'INFO', 'class' : 'logging.handlers.RotatingFileHandler', 'filename' : 'logs/views_sendEmail.log', 'maxBytes' : 1024*1024*10, # 10MB 'backupCount' : 10, 'formatter' : 'standard', }, 'views_register' : { 'level' : 'INFO', 'class' : 'logging.handlers.RotatingFileHandler', 'filename' : 'logs/views_register.log', 'maxBytes' : 1024*1024*10, # 10MB 'backupCount' : 10, 'formatter' : 'standard', }, 'views_chartConfigure' : { 'level' : 'INFO', 'class' : 'logging.handlers.RotatingFileHandler', 'filename' : 'logs/views_chartConfigure.log', 'maxBytes' : 1024*1024*10, # 10MB 'backupCount' : 10, 'formatter' : 'standard', }, }, 'loggers': { 'celery.webapp' : { 'level' : 'ERROR', 'handlers' : ['celery.webapp'], 'propogate' : True, }, 'celery.webapp.task' : { 'level' : 'INFO', 'handlers' : ['celery'], 'propogate' : True, }, 'celery.webapp.chartConfigure' : { 'level' : 'INFO', 'handlers' : ['celery_chartConfigure'], 'propogate' : True, }, 'celery.webapp.register' : { 'level' : 'INFO', 'handlers' : ['celery_register'], 'propogate' : True, }, 'views.logger' : { 'level' : 'ERROR', 'handlers' : ['views.error'], 'propogate' : True, }, 'views.logger.login' : { 'level' : 'INFO', 'handlers' : ['views_login'], 'propogate' : True, }, 'views.logger.register' : { 'level' : 'INFO', 'handlers' : ['views_register'], 'propogate' : True, }, 'views.logger.chartConfigure' : { 'level' : 'INFO', 'handlers' : ['views_chartConfigure'], 'propogate' : True, }, 'views.logger.sendEmail' : { 'level' : 'INFO', 'handlers' : ['views_sendEmail'], 'propogate' : True, }, }, } 

However, he is still having problems with doRollOver.

Would splitting the logs between celery and Django solve the problem? Since access to the log is not carried out by several processes, but only by Django or Celery.

EDIT 2:

I also make Ajax calls. Will it somehow spawn another process that may interfere with registration?

Answer 1

I think you ran into the problem described in this post: Django write with RotatingFileHandler error

That is, when the Django development server starts, there are actually two processes running.

By default, two Django server processes are started. One of them is the actual server, and the other is to detect changes in the code and restart the server. Therefore, settings.py is imported twice, and therefore, two processes access the log file at the same time.

As you advised, try

 python manage.py runserver --noreload 

Answer 2

As described in other answers python manage.py runserver --noreload will work. But here is another solution that still works with code reloading.

Add this to the end of your .py settings

 if DEBUG and os.environ.get('RUN_MAIN', None) != 'true': LOGGING = {} 

python manage.py runserver starts a python process that starts your server in a python child process. Each time a parent discovers a change, he re-creates a new child. The problem is that the rotation of the log by the child process fails because the parent still has a handle to this file. This solution tells parents that there is no log file.