Python: a suggestion on how to improve writing to a streaming text file in Python

Question

I am learning how to write to streaming strings as files in python .

I usually use an expression like

myfile = open("test.txt", w) for line in mydata: ... myfile.write(line + '\n') myfile.close() 

Python creates a text file in a directory and stores chunk-by-chunk values ​​at time intervals.

I have the following questions:

Can I install a buffer? (for example: saving data every 20 MB) is it possible to save line by line?

thanks for the suggestions, they always help to improve

Answer 1

File I / O in python is already buffered. The open() function allows you to determine to what extent the buffering buffer is:

The optional buffering argument specifies the file size required for the file: 0 means unbuffered, 1 means the line is buffered, any other positive value means using a buffer (approximately) of this size. Negative buffering means using the system default, which is usually buffered for tty devices and fully buffered for other files. If this parameter is omitted, the default system is used.

Personally, I use this file as a context manager using the with statement. As soon as all statements in the with package (at least one level of deepening) are completed or an exception occurs, the file object closes:

 with open("test.txt", 'w', buffering=20*(1024**2)) as myfile: for line in mydata: myfile.write(line + '\n') 

where I set the buffer to 20 MB in the example above.