Writing a BytesIO object to a file, "efficiently"

Question

Thus, a quick way to write a BytesIO object to a file would be to use:

with open('myfile.ext', 'wb') as f: f.write(myBytesIOObj.getvalue()) myBytesIOObj.close() 

However, if I wanted to iterate over myBytesIOObj and not write it in one piece, how would I do it? I'm on Python 2.7.1. Also, if BytesIO is huge, would it be a more efficient way to write by iteration?

thanks

Answer 1

shutil is a utility that efficiently writes a file. Copy in chunks, default is 16K. Any number that is a multiple of 4K should be a good cross-platform number. I chose 131072 rather randomly, because before I go to disk, the file is written to the OS cache in RAM, and the portion size is not that big.

 import shutil myBytesIOObj.seek(0) with open('myfile.ext', 'wb') as f: shutil.copyfileobj(myBytesIOObj, f, length=131072) 

By the way, at the end there was no need to close the file object. with defines a region, and a file object is defined inside that region. Therefore, the file descriptor automatically closes when you exit the with block.