Register using elasticsearch-py

Question

I would like to register my python script that uses elasticsearch-py . In particular, I want to have three magazines:

• General log: INFO log and higher in both stdout and file.
• ES log: ES-only file-only messages.
• ES trace log : Advanced ES logging (curling requests and their output, for example) for a file only.

Here is what I still have:

 import logging import logging.handlers es_logger = logging.getLogger('elasticsearch') es_logger.setLevel(logging.INFO) es_logger_handler=logging.handlers.RotatingFileHandler('top-camps-base.log', maxBytes=0.5*10**9, backupCount=3) es_logger.addHandler(es_logger_handler) es_tracer = logging.getLogger('elasticsearch.trace') es_tracer.setLevel(logging.DEBUG) es_tracer_handler=logging.handlers.RotatingFileHandler('top-camps-full.log', maxBytes=0.5*10**9, backupCount=3) es_tracer.addHandler(es_tracer_handler) logger = logging.getLogger('mainLog') logger.setLevel(logging.DEBUG) # create file handler fileHandler = logging.handlers.RotatingFileHandler('top-camps.log', maxBytes=10**6, backupCount=3) fileHandler.setLevel(logging.INFO) # create console handler consoleHandler = logging.StreamHandler() consoleHandler.setLevel(logging.INFO) # create formatter and add it to the handlers formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') consoleHandler.setFormatter(formatter) fileHandler.setFormatter(formatter) # add the handlers to logger logger.addHandler(consoleHandler) logger.addHandler(fileHandler) 

My problem is that INFO es_logger are also displayed on the terminal. In fact, the log messages are saved in the necessary files!

If I delete the logger related part then ES logging works fine, i.e. only saved in the corresponding files. But then I have no other part ... What am I doing wrong with the last part of the settings?

---

Edit

Possible hint: elasticsearch-py sources have a log called logger . Maybe this conflicts with mine? I tried changing the logger name to main_logger in the lines above, but that did not help.

Possible hint 2: If I replaced logger = logging.getLogger('mainLog') with logger = logging.getLogger() , the output format to the es_logger console will change and become identical to the format specified in the fragment.

Answer 1

I think you are struck by the somewhat confusing distribution of the log hierarchy. Everything written in "elasticsearch.trace", which goes through the log level of this registrar, will first be distributed to the "elasticsearch" logger, and then to the root ("") logger. Please note that after the message passes LogLevel to the "elasticsearch.trace" logger, the parent logging levels ("elasticsearch" and root) are not checked, but all messages will be sent to the handler. (The handlers themselves have registration levels that apply.)

Consider the following example to illustrate a problem and a possible solution:

 import logging # The following line will basicConfig() the root handler logging.info('DUMMY - NOT SEEN') ll = logging.getLogger('foo') ll.setLevel('DEBUG') ll.addHandler(logging.StreamHandler()) ll.debug('msg1') ll.propagate = False ll.debug('msg2') 

Output:

 msg1 DEBUG:foo:msg1 msg2 

You see that "msg1" is logged by both the "foo" log and its parent, the root logger (like "DEBUG: foo: msg1"). Then, when propagation is disabled ll.propagate = False to "msg2", the root registrar no longer registers it. Now, if you comment out the first line ( logging.info("DUMMY...") , the behavior will change so that the root logger line is not shown. This is because the logging module level functions above info() , debug() , etc. Configure the root log with the handler when the handler is not yet defined, which is why you see different behavior in your example when you change the root handler by doing logger = logging.getLogger() .

I can’t see in my code that you will do anything in the root logger, but, as you can see, crazy logging.info() or the like, in the program code or the library code can cause a handler to be added.

So, to answer your question, I would set logger.propagate = False for lumberjacks, where it makes sense to you and where you want to distribute, make sure the log level handlers themselves are the way you want them.

Here is an attempt:

 es_logger = logging.getLogger('elasticsearch') es_logger.propagate = False es_logger.setLevel(logging.INFO) es_logger_handler=logging.handlers.RotatingFileHandler('top-camps-base.log', maxBytes=0.5*10**9, backupCount=3) es_logger.addHandler(es_logger_handler) es_tracer = logging.getLogger('elasticsearch.trace') es_tracer.propagate = False es_tracer.setLevel(logging.DEBUG) es_tracer_handler=logging.handlers.RotatingFileHandler('top-camps-full.log', maxBytes=0.5*10**9, backupCount=3) es_tracer.addHandler(es_tracer_handler) logger = logging.getLogger('mainLog') logger.propagate = False logger.setLevel(logging.DEBUG) # create file handler fileHandler = logging.handlers.RotatingFileHandler('top-camps.log', maxBytes=10**6, backupCount=3) fileHandler.setLevel(logging.INFO) # create console handler consoleHandler = logging.StreamHandler() consoleHandler.setLevel(logging.INFO) # create formatter and add it to the handlers formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') consoleHandler.setFormatter(formatter) fileHandler.setFormatter(formatter) # add the handlers to logger logger.addHandler(consoleHandler) logger.addHandler(fileHandler)