Python WatchedFileHandler still writing old file after rotation

Question

I used WatchedFileHandler as my python log file handler so that I can rotate my logs using logrotate (on ubuntu 14.04), which, as you know, is what its docs say. My logrotate configuration files look like

 /path_to_logs/*.log { daily rotate 365 size 10M compress delaycompress missingok notifempty su root root } 

Everything seemed to be working fine. I use logstash to send my logs to my elasticsearch cluster and everything is fine. I added a second log file for my debug logs, which rotates but is not tracked by logstash. I noticed that when this file rotates, python just writes to /path_to_debug_logs/*.log.1 and never starts writing to a new file. If I manually tail /path_to_debug_logs/*.log.1 , it instantly switches and starts writing to /path_to_debug_logs/*.log .

It seems REALLY weird to me.

I believe that it happens that logstash always processes my logs without debugging, some of which trigger a transition to a new file after a call to logrotate . If logrotate is called twice without switching, the log.1 file is moved and compressed to log.2.gz, because of which python can no longer log and logs are lost.

It’s clear that there are a bunch of hacking solutions for this (like cronjob, which processes all my logs from time to time), but I feel like I have to do something wrong.

I use WatchedFileHandler and logrotate instead of RotatingFileHandler for a number of reasons, but mainly because after the rotation it will be nice to compress my logs.

UPDATE:

I tried a terrible hack of adding tail manually until the end of my script log configuration change.

 sharedscripts postrotate /usr/bin/tail -n 1 path_to_logs/*.log.1 endscript 

Sure enough that this works most of the time, but occasionally fails sometimes for some clear reason, so this is not a solution. I also tried several less fragile solutions in which I changed the way WatchFileHandler checks to see if the file has changed, but failed.

I am sure that the root of my problem is that the logs are stored on a network drive, which somehow confuses the file system.

I am moving my rotation to python using RotatingFileHandler , but if someone knows the right way to handle this, I would love to know.

Answer 1

Use the copytruncate logrotate parameter. From docs

copytruncate

Trim the original log file in place after creating the copy instead of moving the old log file and possibly creating a new one, it can be used when some program cannot be told to close its log file and thus continue recording (adding) to forever log file forever. Please note that there is very little time between copying a file and truncating it, so some registration data may be lost. When this option is used, the create option will have no effect, as the old log file remains in place.

Answer 2

WatchedFileHandler performs a rollover when a device and / or inode discovery is detected in a log file just before it is recorded. Perhaps a file that is not being viewed by logstash does not see a change in its device / inode? This explains why the handler continues to write.