PBS script -o file for multiple locations

Question

Sometimes, when I run jobs in a PBS cluster, I really want joblog (-o file) to be in two places. One in $PBS_O_WORKDIR to save all together and one ${HOME}/jobOuts/ for greping / awking / etc ...

Running a test from the command line works with tee :

 echo "hello" | qsub -o `tee $HOME/out1.o $HOME/out2.o $HOME/out3.o` 

But as soon as I try to put this in my PBS script, it does not work if I put it in a PBS script and qsub

 ####Parameterized PBS Script #### #PBS -S /bin/bash #PBS -l nodes=1 #PBS -l walltime=0:01:00 #PBS -j oe #PBS -o `tee TEE_TEST.o TEE_TEST.${PBS_JOBID}.o` #PBS -M [e-mail] #PBS -m abe #PBS -V cd $PBS_O_WORKDIR echo `date` 

Here is qsub and the error:

 qsub TEST.pbs qsub: directive error: -o `tee TEE_TEST.o TEE_TEST.${PBS_JOBID}.o` 

I tried a few more things below - nothing worked.

One line -o (comma, semi-colon, and space):

 #PBS -o ${PBS_JOBNAME}.${PBS_JOBID}.o,${HOME}/jobOuts/${PBS_JOBNAME}.${PBS_JOBID}.o #PBS -o ${PBS_JOBNAME}.${PBS_JOBID}.o,${HOME}/jobOuts/${PBS_JOBNAME}.${PBS_JOBID}.o #PBS -o ${PBS_JOBNAME}.${PBS_JOBID}.o ${HOME}/jobOuts/${PBS_JOBNAME}.${PBS_JOBID}.o 

and two lines:

 #PBS -o ${PBS_JOBNAME}.${PBS_JOBID}.o #PBS -o ${HOME}/jobOuts/${PBS_JOBNAME}.${PBS_JOBID}.o 

In two working liners, only the second option is -o, and one liner does not work.

Any suggestions? Is it possible?

Answer 1

I am very surprised that your command line example worked. Was your work really done? I guess the launch

 echo "hello" | qsub -o `tee $HOME/out1.o $HOME/out2.o $HOME/out3.o` 

shorts the work and actually just grabs the “hello” and sends it to the tee. I assume that someone who knows bash a little better can explain what is actually going on here.

The only way I can think of doing what you are requesting would be for your script to write things to tee. In order for this to work, you need the places to be in the network file system accessible from any possible node calculation.

Answer 2

I studied the qsub man page and I don't think there is a way to specify more than one output file (each) for standard output and standard error. Accepting signals from this page , this is how I achieved something similar to your goals. Your PBS environment may be slightly different. Also, I'm not a bash expert, so there may be more concise methods for achieving the same thing.

Assuming you use the default settings -o, at the end of your normal script job, enter the commands:

 # change to the directory from which this job was submitted cd $PBS_O_WORKDIR # the standard output by default will be in file JOBNAME.oJOBID # on my system, PBS_JOBID has ".machinename" at the end, which needs to be removed filename=${PBS_JOBNAME}.o${PBS_JOBID%%.*} echo "${PBS_O_WORKDIR}/copy.pbs $filename" | qsub 

This will start qsub using the stdin arguments telling it to run "copy.pbs" with the filename argument to be copied. The copy.pbs file that I used is:

 #!/bin/bash # change to the directory from which this job was submitted cd $PBS_O_WORKDIR newfile=${HOME}/jobOuts/$1 cp $1 $newfile 

This helped me copy the first standard PBS output to another directory. A side effect is that running copy.pbs with qsub creates two more output files, STDIN.e * and STDIN.o *. I decided that using qsub was again a good idea to make sure that the first work was done. To be more secure, you can use option-dependent with qsub, for example, "-W depend=afterok:$PBS_JOBID copy.pbs $filename" | qsub "-W depend=afterok:$PBS_JOBID copy.pbs $filename" | qsub . But I have not tested this method and, as I said, I am not an expert in this.