SGE script: print to file at runtime (not just at the end)?

Question

I have an SGE script to execute some python code sent to the queue using qsub. In a python script, I have several print statements (updating me according to program progress). When I run the python script from the command line, print statements are sent to stdout. For sge script, I use the -o option to redirect the output to a file. However, it seems that the script will only send them to the file after the python script completes. This is annoying because (a) I can no longer see real-time updates in the program, and (b) if my work does not finish correctly (for example, if my work begins with a queue), none of the updates are printed. How can I make sure that the script writes to the file every time I want to print something, as opposed to putting it all together at the end?

Answer 1

I think you have a problem with buffered output. Python uses the library to process its output, and the library knows that it is more efficient to write a block while it is not talking to tty.

There are several ways around this. You can run python with the "-u" parameter (see the python manual page for details), for example, with something like this line as the first line of your script:

#! /usr/bin/python -u

but this does not work if you use the trick "/ usr / bin / env" because you do not know where python is installed.

Another way is to open stdout again as follows:

import sys 
import os 

# reopen stdout file descriptor with write mode 
# and 0 as the buffer size (unbuffered) 
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) 

Note that the os.fdopen parameter's bufsize parameter is set to 0 so that it is not buffered. You can do something similar with sys.stderr.

Answer 2

, stdout, tty.

, stdout,

import sys
sys.stdout.flush()

.

Answer 3

SGE "unbuffer", , , IO. , - .

, , - sys.stdout , "write". , stdout, , IO , . , , , / IO .

:

import os, sys, time

class RedirIOStream:
  def __init__(self, stream, REDIRPATH):
    self.stream = stream
    self.path = REDIRPATH
  def write(self, data):
    # instead of actually writing, just append to file directly!
    myfile = open( self.path, 'a' )
    myfile.write(data)
    myfile.close()
  def __getattr__(self, attr):
    return getattr(self.stream, attr)


if not sys.stdout.isatty():
  # Detect redirected stdout and std error file locations!
  #  Warning: this will only work on LINUX machines
  STDOUTPATH = os.readlink('/proc/%d/fd/1' % os.getpid())
  STDERRPATH = os.readlink('/proc/%d/fd/2' % os.getpid())
  sys.stdout=RedirIOStream(sys.stdout, STDOUTPATH)
  sys.stderr=RedirIOStream(sys.stderr, STDERRPATH)


# Simple program to print msg every 3 seconds
def main():    
  tstart = time.time()
  for x in xrange( 10 ):  
    time.sleep( 3 )
    MSG = '  %d/%d after %.0f sec' % (x, args.nMsg,  time.time()-tstart )
    print MSG

if __name__ == '__main__':
  main()

Answer 4

SGE, , , python .

SGE, . , , .

Answer 5

This works for me:

class ForceIOStream:
    def __init__(self, stream):
        self.stream = stream

    def write(self, data):
        self.stream.write(data)
        self.stream.flush()
        if not self.stream.isatty():
            os.fsync(self.stream.fileno())

    def __getattr__(self, attr):
        return getattr(self.stream, attr)


sys.stdout = ForceIOStream(sys.stdout)
sys.stderr = ForceIOStream(sys.stderr)

and the problem is that NFS does not synchronize data with the master until the file is closed or fsync is called.

Answer 6

Why not print to a file instead of stdout?

outFileID = open('output.log','w')
print(outFileID,'INFO: still working!')
print(outFileID,'WARNING: blah blah!')

and use

tail -f output.log

Answer 7

Today I ran into this problem and decided to just write it to disk instead of printing:

with open('log-file.txt','w') as out:
  out.write(status_report)