Python: read a huge number of lines from stdin

Question

I am trying to read a huge number of lines from standard input using python.

more hugefile.txt | python readstdin.py 

The problem is that the program freezes as soon as I read only one line.

 print sys.stdin.read(8) exit(1) 

This prints the first 8 bytes, but then I expect it to complete, but it will never happen. I think this is not just reading the first bytes, but an attempt to read the entire file in memory.

Same problem with sys.stdin.readline ()

What I really want to do is, of course, read all the lines, but with a buffer, so I do not have enough memory.

I am using python 2.6

Answer 1

This should work effectively in modern Python:

 import sys for line in sys.stdin: # do something... print line, 

Then you can run the script as follows:

 python readstdin.py < hugefile.txt 

Answer 2

That day, you needed to use xreadlines to get an effective huge IO IO - and docs will now be asked to use for line in file .

Of course, this will only help if you are actually working on strings one at a time. If you just read large binary drops to convey something else, then your other mechanism might be just as effective.