How to perform several tasks in parallel in the structure

Question

I know that with the -P tag or @parallel tag I can run the task in parallel on multiple hosts.

I am trying to perform several long tasks in parallel on the same host:

 @task def task1(): # long running op @task def task2(): #long running op @task def task3(): #long running op @task def backup_all(): execute(task1) execute(task2) execute(task3) 

How can I run task1, task2 and task3 in parallel on the same host using a tag. I know that I can run several processes with different tasks, but I am looking for a solution that includes a structure.

Answer 1

You have several ways to solve this problem. You can use the bash / linux level controls and run tasks in the background using bg and then wait to complete them. Documentation explaining this mechanism control style .

If you still want to use Fabric / Python for this, you will most likely need to use job_queue , which already in lib and sorta writes your own queue to insert them, or read multiprocessing and just do some simple python-forking. Although this is essentially all that job_queue does.

Answer 2

fabric performs one task for each host. -P switch runs the task in parallel on different hosts . Lack of parallelization on the same host.

You can parallelize commands manually, for example using xargs -P or GNU parallel .

Invoke (beta) claims that you can run tasks in parallel. It extracts non-ssh parts from fabric . If it is installed on remote hosts, you can use the fabric to invoke the Invoke command, which will run tasks in parallel on the same host.