Geek Answers Handbook

Small-scale load balancing

I have a number of tasks to be completed; There is no dependence between tasks. I am looking for a tool to help me distribute these tasks on machines. The only limitation is that each machine must run only one task at a time. I am trying to maximize throughput because jobs are not very balanced. My current cracked shell scripts are less efficient because I pre-create a queue for each machine and cannot move jobs from the queue of a heavily loaded machine to the waiting one, having already finished everything.

Previous suggestions included SLURM, which seems redundant, and further overloads LoadLeveller.

GNU Parallel looks almost exactly what I want, but the remote machines do not say SSH; it uses a custom launcher (which does not have capabilities for queues). I would like to have Gnu Parallel, where the machine can simply be replaced in the shell script on the fly before submitting the job.

So in short:

  • List of jobs + List of machines that can accept: Maximize throughput. Preferably as close to the shell as possible.

In the worst case, something can be hacked along with the bash lockfile, but I feel that there must be a better solution somewhere.

+4
source share
2 answers

Assuming your jobs are in a jobs.tab text file similar to

 /path/to/job1 /path/to/job2 ...

Create dispatcher.sh something like

 mkfifo /tmp/jobs.fifo while true; do read JOB if test -z "$JOB"; then break fi echo -n "Dispatching job $JOB .." echo $JOB >> /tmp/jobs.fifo echo ".. taken!" done rm /tmp/jobs.fifo

and run one instance

 dispatcher.sh < jobs.tab

Now create launcher.sh as

 while true; do read JOB < /tmp/jobs.fifo if test -z "$JOB"; then break fi #launch job $JOB on machine $0 from your custom launcher done

and run one instance of launcher.sh for each target machine (giving the machine the first and only argument)

+2
source

GNU Parallel supports your own ssh command. Therefore, this should work:

 function my_submit { echo On host $1 run command $3; } export -f my_submit parallel -j1 -S "my_submit server1,my_submit server2" my_command ::: arg1 arg2
+1
source

More articles:


All Articles