Parallel launch with jenkins

Question

I have several (several hundred) files to run the test (each test takes several minutes).

The execution of the sequence is unacceptable and not all together. Therefore, I am looking for something like a consumer producer.

I tried to perform tasks on pipelines and execute a parallel command as follows:

def files = findFiles glob: 'test_files/*' def branches = [:] files.each{ def test_command = "./test ${it}" branches["${it}"] = { sh "${test_command} ${it}"} } stage name:'run', concurrency:2 parallel branches 

Problem:

All tasks run simultaneously (OOM and all the fun)

Answer 1

It does not have the same introspection as Jenkins parallel step, but since it does not support a fixed pool, you can use xargs to achieve the same result:

 def files = findFiles glob: 'test_files/*' def branches = [:] // there probably a more efficient way to generate the list of commands files.each{ sh "echo './test ${it}' >> tests.txt" } sh 'cat tests.txt | xargs -L 1 -I {} -P 2 bash -c "{}"' 

The -P argument is one that indicates that a fixed number of 2 (or N) processes should always be executed. Other tools, such as GNU Parallel, offer even greater customization of how many processes should be used.

You can also try using the lock step from the Lockable Resources plugin , a node step oriented to a fixed number of executors. However, for me this is too much overhead if your single tests already take tens of seconds each.