Difference in stdout buffering on Linux and Windows

Question

There seems to be a difference in how stdout buffered on Windows and Linux when written to the console . Consider this little python script:

 import time for i in xrange(10): time.sleep(1) print "Working" , 

When running this script on Windows, we see that Working appear one after another with a second continuation of the wait between them. On Linux, we need to wait 10 seconds, and then the entire line appears immediately.

If we change the last line to print "Working" , each line will appear separately on Linux.

So Linux, stdout seems to be line buffered, but not on Windows. We can disable buffering with -u -option (in this case, the script on Linux has the same behavior as on Windows). The documentation says:

-u Force stdin, stdout and stderr are not fully loaded.

In fact, he does not say that without -u -option, stdin and stdout buffered. And so my questions are:

• What is the reason for the different behavior on Linux / Windows?
• Is there any guarantee that if redirected to a file , stdout will be buffered, no matter which OS? At least it looks like Windows and Linux.

My main problem is not that (when some answers suggest) when information is flushed, but that if stdout not buffered, it can be a serious performance hit and should not be relied on.

Edit: It is worth noting that for Python3 the behavior is the same for Linux and Windows (but this is not surprising, because the behavior is explicitly specified by the print method parameters).

Answer 1

Assuming you're talking about CPython (probably), this is due to the behavior of the underlying C implementations.

The ISO C standard mentions ( C11 7.21.3 Files /3 ) three modes:

• unbuffered (characters appear as soon as possible);
• fully buffered (characters appear when the buffer is full); and
• line is buffered (characters appear on the output of a new line).

There are other triggers that cause characters to appear (for example, filling a buffer even if a new line is not output, a request for input in some circumstances, or closing a stream), but they are not important in the context of your question.

The important thing is that 7.21.3 Files /7 in the same standard:

As originally discovered, the standard error stream is not fully buffered; standard input and standard output streams are fully buffered if and only if a stream can be defined so as not to refer to an interactive device.

Pay attention to the place to maneuver. Standard output can be either buffered or unbuffered, unless the implementation knows that it is not an interactive device.

In this case (the console) this is an interactive device, so the implementation is not allowed to use unbuffered. However, you are allowed to choose either of the other two modes, so you see the difference.

Unbuffered output will see that messages appear as soon as you print them (a la your Windows). The line buffer will linger until a newline is output (your behavior on Linux).

If you really want your messages to be discarded regardless of mode, simply flush them yourself:

 import time, sys for i in xrange(10): time.sleep(1) print "Working", sys.stdout.flush() print 

In terms of ensuring that the output will be buffered when redirecting to a file, this will be examined in quotation marks from the standard that I have already shown. If a stream can be defined as using a non-interactive device, it will be fully buffered. This is not an absolute guarantee, since it does not indicate how it is defined, but I would be surprised if any implementation could not understand this.

In any case, you can test specific implementations by simply redirecting the output and checking the file to make sure that it is reset once per output or at the end.

Answer 2

The behavior is different in that buffering is usually not defined, which means that implementations can do whatever they want. And this means that implementations can change at any time or vary in undocumented ways, possibly even on the same platform.

For example, if you print the string “long enough” on Linux, without a new line ( \n ), it will most likely be written as if it had a new line (since it exceeds the buffer). You can also find a buffer size that varies between stdout, pipe and files.

It's very bad to rely on undefined behavior, so use flush() when you really need bytes to write.

And if you need to control buffering (for example, for performance reasons), then you need to implement your own buffering on top of write() and flush() . This is pretty simple to do, and it gives you full control over how and when bytes are actually written.

Answer 3

Windows and Linux have very different console output drivers. On Linux, the output is buffered until \ n happens in the case of your program.

If you want to force a buffer reset manually, use

 import sys sys.stdout.flush() 

Answer 4

This one already has the answers elsewhere, but I will describe below.

• The reason for the different behavior on Windows and Linux is due to the way the print command is executed (as noted in the eryksun comment). You can get more information about this here and here .

• In python, this can be fixed in many ways. Read more about it here .

Answer 5

The problem has been known since 2011, see this Python Error Problem # 11633

The print function does not perform buffering. The file object that he writes can. Even sys.stdout may or may not.

To take into account the differences in behavior, the solution found was to update the documentation to include the following sentence in bold:

The file argument must be an object with the write (string) method; if not or not, sys.stdout will be used. Output buffering is determined by the file.

It is worth noting that:

Guido said that this is what he should do and that "Applications that need to be flushed should call flush ()." Therefore, the code change is rejected.