Redirect output to different directories for working with arrays of solar networks

Question

I have many jobs with the Sun Grid Engine. Since these are jobs (~ 100000), I would like to use array jobs that seem to be easier in the queue.

Another problem is that each job creates a stdout and stderr file that should track the error. If I define them in qsub -t 1-100000 -o outputdir -e errordir , I will end up with directories with 100,000 files, which is too much.

Is there a way for each task to write the output file to a directory (for example, a directory consisting of the first two characters of a job identifier, which is random hexadecimal letters or a modulu 1000 job number or something else).

thanks

Answer 1

I cannot think of a good way to do this with qsub, since the -o and -e options do not have any programming interfaces. However, there is a way to accomplish what you want.

Run qsub with -o and -e, pointing to / dev / null. Make the command you run is some kind of shell that redirects its own stdout and stderr to files in any way (i.e., with a breakdown of the directory structure) before it does the real work.