Creating a qsub block before shutting down?

Question

I currently have a driver program that runs several thousand instances of the "payload" program and does some post-processing of the output. The driver currently calls the payload program directly, using the shell() function, from multiple threads. The shell() function executes the command in the current working directory, blocks until the command is completed, and returns the data that was sent to the stdout command with the command. This works well on a single multi-core machine. I want to change the driver to send qsub to a large computing cluster instead of more parallelism.

Is there a way to make the qsub command output its results to stdout instead of a file and block until the task is completed? Basically, I want it to do as “normal” command execution as much as possible, so that I can parallelize the cluster with minimal modification to my driver program.

Edit: I thought that all mesh motors were pretty much standardized. If it is not, and it is important, I use Torque.

Answer 1

You did not mention which queuing system you are using, but SGE supports the “-sync y” option for qsub, which will force it to lock until the job completes or finishes.

Answer 2

In TORQUE, this is done using the -x and -I options. qsub -I indicates that it should be interactive, and -x says that only the specified command is run. For instance:

 qsub -I -x myscript.sh 

will not return until myscript.sh completes execution.

Answer 3

In PBS, you can use qsub -Wblock=true <command>