Is there any way to find the buffer size of a file object

Question

I am trying to "display" a very large ascii file. Basically I read the lines until I find a specific tag, and then I want to find out the position of that tag so that I can find it again to pull out the related data.

from itertools import dropwhile with open(datafile) as fin: ifin = dropwhile(lambda x:not x.startswith('Foo'), fin) header = next(ifin) position = fin.tell() 

Now this tell not giving me the right position. This question was previously asked in various forms. The reason, apparently, is because python is buffering the file object. So python tells me where is the file pointer, not where is my file pointer. I do not want to disable this buffering ... Performance is important here. However, it would be nice to know if there is a way to determine how many bytes python chooses to buffer. In my actual application, while I close the lines starting with Foo , it does not matter. I can throw a few lines here and there. So, what I'm actually planning on doing is something like:

 position = fin.tell() - buffer_size(fin) 

Is there any way to find the size of the buffer?

Answer 1

It seems to me that the buffer size is hard-coded in Cpython - 8192. As far as I can tell, there is no way to get this number from the python interface, except to read one line when opening a file, do f.tell() to find out how much data is on actually read python and then continue searching until the beginning of the file.

 with open(datafile) as fin: next(fin) bufsize = fin.tell() fin.seek(0) ifin = dropwhile(lambda x:not x.startswith('Foo'), fin) header = next(ifin) position = fin.tell() 

Of course, this fails if the first line is longer than 8192 bytes, but this has no real effect on my application.