Problems with Jenkins Conveyor and Concurrent Work

Question

Background

I use Jenkins with the Build Pipeline plugin to create fairly complex projects that require several compilation steps:

• Create a source code RPM.
• Create a binary RPM (this is done twice, once for each platform).
• Expand to the YUM repository.

My strategy for solving assembly problems involves dividing the overall work into parameterized tasks that can be reused in projects and branches, with each task representing one stage in the pipeline. Each stage starts with parameters, and artifact collection is passed to the next task in the pipeline. However, I have some problems with this strategy, and I can really use some tips on how to solve this problem in the most elegant and flexible way.

To be more specific, there are two common libraries that are shared by other projects (but not in all projects). Libraries are built differently than dependent projects, but it is not necessary to indicate in Jenkins what the dependent projects are.

There are several branches, the main branch (rebuilt at night), the development branch (polled for changes), function branches (also polled) and release branches (polled, but created for release). Branches are built the same for several projects.

We create several repositories every month, and although we can expect a little customization for a new project, as a rule, we want it to be as simple and automatic as possible.

Problems

• I have many projects with several branches, and I do not want to build all branches or even all projects in the same way. Since most of the assembly steps are similar, I can turn these general steps into parameterized build jobs and get each job to run the next in the chain, pass parameters and build artifacts along the chain. However, this falls apart if one of the steps needs to be skipped, because I do not know how to conditionally skip the assembly step. This means that I will need to copy the build tasks so that I can configure them for each pipeline, resulting in a lot of build tasks. I could use a combination of plugins to create a job generator (e.g. dsl flow, dsl job, etc.) and hide as much as possible from users, but what is the most elegant Jenkins solution for this? Are there any plugins or examples that I could skip? What is your experience with this?

• Since step 2 can be divided into two tasks that can be run in parallel, this creates complexity, which causes me problems with my pipeline. My first attempt to run a parameterized assembly task twice with different parameters and then join the tasks after using the connection plugin, but it looked like it would be difficult to copy assemblies from two upstream tasks in the artifacts. This is important because I need build artifacts from both jobs for stage 3. What is the most elegant solution for combining parallel jobs and copying artifacts from them? Are there any examples that I could skip?

• I need to combine the test results generated from both tasks in the second stage, and copy them to the task that starts the assembly. What is the best way to handle this?

I am glad to read articles, presentations, technical articles, reference documentation, writing scripts and everything that is necessary to make this work beautiful, but I am not an expert of Jenkins. If anyone can give me some advice on these 3 issues then this will be helpful. In addition, I would appreciate any constructive advice on how to get the best results from Jenkins CI pipeline assembly, if necessary.

Answer 1

At the first point, the Generator Job Generator plugin was created, designed to solve this use case. You can find more information on the Job Generator wiki page . There is also the same type of plugin with a different approach (job generator as a build step), it is called Jobcopy Builder . The other approaches you talked about require some kind of DSL and may be a good choice.