Geek Answers Handbook

Remote task queue using bash & ssh for variable number of live workers

I want to distribute work from the main server to several production servers using packages.

Ideally, I would have a tasks.txt file with a list of tasks to perform

 cmd args 1 cmd args 2 cmd args 3 cmd args 4 cmd args 5 cmd args 6 cmd args 7 ... cmd args n

and each working server will connect using ssh, read the file and mark each line as done or done

 #cmd args 1 #worker1 - done #cmd args 2 #worker2 - in progress #cmd args 3 #worker3 - in progress #cmd args 4 #worker1 - in progress cmd args 5 cmd args 6 cmd args 7 ... cmd args n

I know how to make an ssh connection, read the file and execute it remotely, but I don’t know how to read and write the atomic operation, so as not to have cases when 2 servers start the same task, and how to update the line.

I would like each employee to go to the task list and block the next available task in the list, and not the server actively commanding the workers, since I will have a flexible number of work clones that I will start or close according to how quickly I need tasks.

UPDATE:

and my idea for a working script would be:

 #!/bin/bash taskCmd="" taskLine=0 masterSSH="ssh usr@masterhost " tasksFile="/path/to/tasks.txt" function getTask(){ while [[ $taskCmd == "" ]] do sleep 1; taskCmd_and_taskLine=$($masterSSH "#read_and_lock_next_available_line $tasksFile;") taskCmd=${taskCmd_and_taskLine[0]} taskLine=${taskCmd_and_taskLine[1]} done } function updateTask(){ message=$1 $masterSSH "#update_currentTask $tasksFile $taskLine $message;" } function doTask(){ return $taskCmd; } while [[ 1 -eq 1 ]] do getTask updateTask "in progress" doTask taskErrCode=$? if [[ $taskErrCode -eq 0 ]] then updateTask "done, finished successfully" else updateTask "done, error $taskErrCode" fi taskCmd=""; taskLine=0; done
+6
source share
3 answers

You can use flock to access the file at the same time:

 exec 200>>/some/any/file ## create a file descriptor flock -w 30 200 ## concurrently access /some/any/file, timeout of 30 sec.

You can specify the file descriptor in the task list or any other file, but, of course, the same file for flock works. The lock will be deleted as soon as the process that created it is completed or fails. You can also remove the lock yourself when you no longer need it:

 flock -u 200

Usage example:

 ssh user@x.x.x.x ' set -e exec 200>>f echo locking... flock -w 10 200 echo working... sleep 5 '

set -e crashes the script if any step crashes. Play with sleep time and run this script in parallel. Only one sleep will be executed at a time.

+2
source

try to implement something like

 while read line; do echo $line #check if the line contains the # char, if not execute the ssh, else nothing to do checkAlreadyDone=$(grep "^#" $line) if [ -z "${checkAlreadyDone}" ];then <insert here the command to execute ssh call> <here, if everything has been executed without issue, you should add a commad to update the file taskList.txt one option could be to insert a sed command but it should be tested> else echo "nothing to do for $line" fi done < taskList.txt

Claudio Relations

0
source

Check if you are inventing GNU Parallel:

 parallel -S worker1 -S worker2 command ::: arg1 arg2 arg3

GNU Parallel is a common parallelizer and makes it easy to run jobs in parallel on one computer or on multiple computers to which you have ssh access. It can often replace a for loop.

If you have 32 different tasks that you want to run on 4 processors, the direct way to parallelize is to run 8 tasks for each processor:

Simple scheduling

GNU Parallel instead launches a new process when it ends - saving active processors and, therefore, saving time:

GNU parallel scheduling

Installation

If GNU Parallel is not packaged for your distribution, you can perform a personal installation that does not require root access. This can be done in 10 seconds by following these steps:

 (wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options, see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

More details

Additional examples: http://www.gnu.org/software/parallel/man.html

Watch the videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Go through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Subscribe to your email list for support: https://lists.gnu.org/mailman/listinfo/parallel

0
source

More articles:


All Articles