Continuous loop recording

Question

What would be a good way to create logs (with the python logging module) inside a continuous loop without creating a lot of useless log files? An example is a loop in which a constant enumerates a folder and performs some actions when it sees a file of a certain type. I want to log that the files were not found, or the files were found, but of the wrong type, without constantly registering the same line for each folder check, because it can be run many times per second.

Answer 1

Create a Handler that subclasses any other functionality that you need. Save either the last or all previously registered messages that you no longer want emit :

 def make_filetype_aware_handler(handler_class): class DontRepeatFiletypeHandler(handler_class): def __init__(self, *args, **kwds): super().__init__(*args, **kwds) self.previous_types = set() def emit(self, record): if not record.file_type in self.previous_types: self.previous_types.add(record.file_type) super().emit(record) return DontRepeatFiletypeHandler FiletypeStreamHandler = make_filetype_aware_handler(logging.StreamHandler) logger = logging.getLogger() logger.addHandler(FiletypeStreamHandler(sys.stderr)) logger.debug('Found file of type %(file_type)', file_type='x-type/zomg') 

Answer 2

I understand that you are trying to limit logging to the same message again and again.

If this is your problem, I would create a set of file_types files that you already registered. However, you need to be careful if this goes on forever, you will end up crashing.

 from sets import Set logged = Set() while yourCondition: file_type = get_next_file_type() needToLog = #determine if you need to log this thing if needToLog and (not file_type in logged): logger.info("BAH! " + file_type) logged.add(file_type) 

Answer 3

Record less important events with a lower priority, such as DEBUG. See setLevel and SysLogHandler .

During development, set the level to DEBUG, and as your application matures, set it to more reasonable values, such as INFO or ERROR.

Your application must do something with errors, for example, delete files with the wrong type and / or create missing files; or move the incorrectly configured directories from the job poll to the quarantine location, so your log will not be flooded.

Answer 4

 import logging logger = logging.getLogger(test) # logging to a file hdlr = logging.FileHandler(test.log) formatter = logging.Formatter('%(asctime)s %(filename)s %(lineno)s %(levelname)s % (message)s') hdlr.setFormatter(formatter) logger.addHandler(hdlr) logger.setLevel(logging.DEBUG) 

Then in the loop you should check the file type and the file is present or not. Then add:

 logger.debug('File Type : %s ' % file_type) 

also

 if file_present: logger.debug('File Present : %s ' % present)