Geek Answers Handbook

Writing to Python files only for gzipped files

I create a service in which I register plain text formatted logs from several sources (one file per source). I am not going to rotate these magazines, as they should be forever.

To make these forever smaller files, I hope I can gzip them on the fly. Because they are log data, files are compressed very well.

What is a good approach in Python for writing text files that only support append files, so that later recording can resume when the service is turned on and off? I'm not worried about losing a few lines, but if the gzip container itself breaks and the file becomes unreadable, that is not.

In addition, if this is not the case, I can simply write them in plain text without gzipping if it is not worth the hassle.

+6
source share
1 answer

Note. On Unix systems, you should seriously consider using an external program written for this exact task:

  • logrotate (rotates, compresses and writes system logs)

You can set the speed so high that the first file will be deleted after 100 years or so.

In Python 2, logging.FileHandler accepts the encoding keyword argument, which can be set to bz2 or zlib .

This is because logging uses the codecs module, which in turn treats bz2 (or zlib ) as encoding:

 >>> import codecs >>> with codecs.open("on-the-fly-compressed.txt.bz2", "w", "bz2") as fh: ... fh.write("Hello World\n") $ bzcat on-the-fly-compressed.txt.bz2 Hello World

Python version 3 (although docs mention bz2 as an alias, you really need to use bz2_codec - at least from 3.2.3):

 >>> import codecs >>> with codecs.open("on-the-fly-compressed.txt.bz2", "w", "bz2_codec") as fh: ... fh.write(b"Hello World\n") $ bzcat on-the-fly-compressed.txt.bz2 Hello World
+8
source

More articles:


All Articles