I have the following case:
There are a couple of repositories (well, two-digit number). Each of them has some sort of (declarative) Jenkins pipeline - it would be one of three types:
- run tests -> build docker image -> deploy to test environment;
- run tests -> create package -> upload to package repository (i.e. PyPI);
Now, currently, each repository has a Jenkinsfile of one of the above types, and a Jenkinsfile which runs only the tests and does nothing more. These Jenkinsfiles are basically the same across the repositories - same code, same environment variables. The only thing that changes are the image/package names (and, of course, there are differences between building images and creating packages). If I have to make a change to any one Jenkinsfile, I have to make a change to all of them, go through Pull Requests etc., which is obviously too much unneeded work.
Not only this, but the project pipeline involves adding even more tasks performed by Jenkins - like running integration tests, deploying to higher environments, etc. Most of these integrations would require certain combinations of steps - i.e. today I want to build an image and deploy to environment A; tomorrow I want unit-test, deploy to environment B and integration-test; etc. As you see, these pipelines would have many steps in common. Following the current practice, this will lead to code duplication and maintenance costs.
If all this is implemented as separate Jenkinsfile for each pipeline, this will lead to explosion of Jenkinsfiles which are almost the same within a repository and across repositories.
What I want to do is to abstract away the common steps (like test a repository; build an image; create a package; run some integration tests; deploy to an environment) in a separate repository which will hold all such CI/CD code. Then I want to create parameterized (Jenkins) pipelines in each of the repositories, which just run these generic steps in any combination I wish with the configuration I have given for a concrete repository.
Three options I have researched:
- create a separate Jenkins job with a corresponding Jenkinsfile for each generic step; have a pipeline in each repository that just calls a combination of these generic steps providing parameters. This has the overhead of having to manually specify git-clone operations for the repository which needs to be built/tested in each generic step. This answer seems to point in this direction, but I am still concerned with the possible overhead.
- call a combination the generic Jenkinsfiles from within a pipeline providing the necessary parameters. This has the overhead of rather low maintainability (calling Jenkinsfile from Jenkinsfile doesn't seem that straightforward and includes more lines of code).
- call abstracted-away shell scripts. As I see it, this leads to even harder maintenance and even lower abstraction. It constrains the user to script everything in shell; loses all possible Jenkins plugins and other tools. While possible, it seems to me that it won't provide the level of abstraction I hope to achieve. Please correct me if I am wrong.
Now the question is: how can I achieve this using (declarative) Jenkins pipelines with the least amount of overhead?
Even more importantly - are there tools that are better-suited for this kind of work?
