How to do the same work several times in parallel with Jenkins?

Question

I am testing Jenkins to see if it matches our build and testing structure. I found that Jenkins and its affordable plugins fit most of our needs. Except that I cannot find help on how to perform one specific type of task.

We are creating an application for embedded devices. We have 100 tests that need to be run on these devices. If we carry out all the tests on one device after assembly, it will take several hours to get the results. However, if we conduct tests on 100 parallel devices, we will get the results in a much shorter time.

All tests will have a very similar starting point. The test script is called with the IP address of the device to run the test on and username / pw. The script will run the necessary test on the device and report a skip / failure for each test item.

I think that a long / painful way to do this is to write 100 tasks in Jenkins, each of them will directly differ from the script test (with the above parameters) and run them in parallel using the available plugins. However, retaining all of these jobs will be very difficult in the long run.

Thus, the best way to do this would be to create a task (let it call its child_tester), which can take parameters such as: test script name, device IP address, username / pw, etc. Then use another task (call mother_tester) to call child_tester 100 times with different IP addresses and run them in parallel. I need some way to accumulate all the test results for each individual run of the child_tester jobs and send them back to mother_tester.

My question is: is there a plugin or any way to do this in Jenkins? I looked at the Build Flow, Parallel Test Executor, and Parameterized Trigger plugins. However, they do not seem to fit my needs.

Thank you for your help.

Answer 1

As far as I understand, you looked at the Build Flow plugin, but I'm not sure why you fired it. Perhaps you can point out holes in my sentence.

Assuming you have enough artists on your system to run tasks in parallel, I think the Build Flow plugin and Build Build Aggregator plugin can do what you want.

• The Build Flow plugin supports the simultaneous execution of tasks . I see no reason why Build Flow cannot plan your "child" task in parallel with different parameters.

• The assembly flow test aggregator captures the test results from the planned builds of the Build Flow job, so your “child” job will need to publish its own test results.

• You will need to configure your "child" task so that it can run in parallel, checking if necessary, "Perform parallel assemblies" in the task configuration.

• Regardless of which set of slaves allows you to connect to embedded devices, you will need a sufficient number of performers to run your tasks in parallel.

---

Update : with a simple build stream definition:

parallel ( { build("dbacher flow child", VALUE: 1) }, { build("dbacher flow child", VALUE: 2) }, { build("dbacher flow child", VALUE: 3) }, { build("dbacher flow child", VALUE: 4) } ) 

I get the output:

 parallel { Schedule job dbacher flow child Schedule job dbacher flow child Schedule job dbacher flow child Schedule job dbacher flow child Build dbacher flow child #5 started Build dbacher flow child #6 started Build dbacher flow child #7 started Build dbacher flow child #8 started dbacher flow child #6 completed dbacher flow child #7 completed dbacher flow child #5 completed dbacher flow child #8 completed } 

The task history shows that all four tasks are scheduled within a few seconds of each other. But the step of constructing the task contains an artificial delay (sleep mode), which does not allow you to quickly complete one separate assembly.

---

Update 2 . Here is an example of dynamically creating a list of parallel tasks from another data structure:

 // create a closure for the deploy job for each server def paramValues = (1..4) def testJobs = [] for (param in paramValues) { def jobParams = [VALUE: param] def testJob = { // call build build(jobParams, "dbacher flow child") } println jobParams testJobs.add(testJob) } parallel(testJobs) 

A list passed in parallel is a list of closures that invoke an assembly with unique parameters. I had to definitely define work parameters outside the closure function to ensure that tasks would be scheduled separately.

I cut the syntax from another answer and this thread on the Jenkins mailing list.

Answer 2

Please make sure that the number of artists in the Manage Jenkins → Manage Nodes settings is greater than the number of individual tasks in the MultiJob project. By default, I think it is 2. Therefore, we need to increase it.