Why is there a built-in read delay with popen ()?

Question

I execute a long-term (and often blocked) command via popen (): "ls -R /"

Problem: popen () reads into the buffer you supply, and it seems to be trying to fill the ENTIRE buffer before returning. This makes it block quite often (if your buffer is large).

The solution seems to make non-blocking fd invalid. When I do this, popen () is still blocked, usually about 1 second each time. Why is this happening?

Here is my code. Be sure to compile with -std = C ++ 11:

#include <cstdio> #include <iostream> #include <sys/time.h> #include <unistd.h> #include <fcntl.h> static constexpr size_t SIZE = 65536; struct Time { friend std::ostream &operator<<(std::ostream &os, Time const &t) { (void)t; timeval tv; gettimeofday(&tv, nullptr); os << tv.tv_sec << "." << std::fixed << tv.tv_usec << " "; return os; } }; int main() { FILE *file; file = popen("ls -R /", "r"); if(!file) { std::cerr << "Could not open app: " << errno; return -1; } // Make it non-blocking int fd = fileno(file); fcntl(fd, F_SETFL, O_NONBLOCK); char buffer[SIZE]; Time t; while(true) { int rc = fread(buffer, 1, SIZE, file); if(rc <= 0) { if(EAGAIN == errno) { usleep(10); continue; } std::cerr << t << "Error reading: " << errno << std::endl; break; } std::cerr << t << "Read " << rc << std::endl; } pclose(file); return 0; } 

Conclusion (note that they are approximately 1 second apart, although fd is not blocked, and I only have a 10 ms pause in the loop):

 1429625100.983786 Read 4096 1429625101.745369 Read 4096 1429625102.426967 Read 4096 1429625103.185273 Read 4096 1429625103.834241 Read 4096 1429625104.512131 Read 4096 1429625105.188010 Read 4096 1429625105.942257 Read 4096 1429625106.642877 Read 4096 

Answer 1

You must use read , not fread . The stdio functions have their own level of buffering outside the OS, so they can even block non-blocking file descriptors. Use read to avoid this.

Secondly, you need to stop ls from buffering its output. The default behavior for programs that reference glibc is to use line buffering when stdout connects to TTY, and full buffering when it connects to or redirects to a file. Full buffering means that the output signal is cleared only when the 4 KB buffer is full, and not cleared every time a new line is output.

You can use stdbuf to override this behavior. Note that it will only work for programs that use C streams and a link to glibc. These are most programs, but not all.

 popen("stdbuf -oL ls -R /", "r");