What is the correct way to log in a CSV file?

Question

I want to log some information about each individual request, send it to a busy HTTP server in formatted form, use the log module to create something that I don't want:

[I 131104 15:31:29 Sys:34] 

I think about csv format, but I don’t know how to configure it, and python got csv module, but read the manual

 import csv with open('some.csv', 'w', newline='') as f: writer = csv.writer(f) writer.writerows(someiterable) 

since it will open and close the file every time, I am afraid that in this way it will slow down the entire server, what can I do?

Answer 1

Just use python logging .

You can configure the output as you wish; see Changing the format of displayed messages :

To change the format used to display messages, you need to specify the format you want to use:

import logging
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
logging.debug('This message should appear on the console')
logging.info('So should this')
logging.warning('And this, too')

and formatters :

Formatter objects customize the final order, structure, and contents of the log message.

You can easily replace : with , to create an output file in CSV format.

You will find a list of attributes that you can use here: LogRecord attributes .

Answer 2

I agree that you should use the registration module, but you cannot do it correctly with just a format string, as shown in some other answers, since they do not affect the situation when you register a message containing a comma.

If you need a solution that will correctly remove any special characters in the message (or other fields, I suppose), you will need to write your own formatter and install it.

 logger = logging.getLogger() formatter = MyCsvFormatter() handler = logging.FileHandler(filename, "w") handler.setFormatter(formatter) logger.addHandler(handler) logger.setLevel(level) 

You will obviously have to implement the MyCsvFormatter class, which should inherit from logging.Formatter and override the format () method

 class MyCsvFormatter(logging.Formatter): def __init__(self): fmt = "%(levelname)s,%(message)s" # Set a format that uses commas, like the other answers super(MyCsvFormatter, self).__init__(fmt=fmt) def format(self, record): msg = record.getMessage() # convert msg to a csv compatible string using your method of choice record.msg = msg return super(MyCsvFormatter, self).format(self, record) 

Note. I already did something similar before, but have not tested this sample code

Regarding the actual screening of the message, here is one of the possible approaches: Python - write data in csv format as a string (not a file)

Answer 3

As the sloth says, you can easily edit the log delimiter to a comma, thus creating a CSV file.

Working example:

 import logging # create logger lgr = logging.getLogger('logger name') lgr.setLevel(logging.DEBUG) # log all escalated at and above DEBUG # add a file handler fh = logging.FileHandler('path_of_your_log.csv') fh.setLevel(logging.DEBUG) # ensure all messages are logged to file # create a formatter and set the formatter for the handler. frmt = logging.Formatter('%(asctime)s,%(name)s,%(levelname)s,%(message)s') fh.setFormatter(frmt) # add the Handler to the logger lgr.addHandler(fh) # You can now start issuing logging statements in your code lgr.debug('a debug message') lgr.info('an info message') lgr.warn('A Checkout this warning.') lgr.error('An error writen here.') lgr.critical('Something very critical happened.') 

Answer 4

I do not think this is the best idea, but it is doable and fairly simple. Manually buffer your journal. Keep logs in a specific place and write them to a file from time to time. If you know that your server will be constantly busy, flush your buffer when it reaches a certain size. If there can be big gaps in use, I would say that a better call (or a better process, check why threads suck and slow down applications) with an infinite (theoretically, of course) sleep / flash cycle. Also, do not forget to create some kind of hook that will buffer the buffer when the server is interrupted or does not work (maybe signals? Or just try / besides the main function - there are even more ways to do this), so you won’t lose loose buffer data about unexpected exit.

I repeat - this is not a good idea, this is the first thing that occurred to me. You might want to consult logging implementations from Flask or some other webapp platform (AFAIR Flask also has a CSV log).