What is the optimal size of a Jenkins Pipeline Stage?

Question

Should I create single checkout stage or multiple stages like checkout scripts, checkout code, checkout something else?

How should I decide on what piece of logic to put into separate stage? Or are stages just a pipeline visualization tool?

Answer 1

Single responsibility principle (SRP)

Martin defines a responsibility as a reason to change, and concludes that a class or module should have one, and only one, reason to be changed (i.e. rewritten). As an example, consider a module that compiles and prints a report. Imagine such a module can be changed for two reasons. First, the content of the report could change. Second, the format of the report could change. These two things change for very different causes; one substantive, and one cosmetic. The single responsibility principle says that these two aspects of the problem are really two separate responsibilities, and should therefore be in separate classes or modules. It would be a bad design to couple two things that change for different reasons at different times.

The reason it is important to keep a class focused on a single concern is that it makes the class more robust. Continuing with the foregoing example, if there is a change to the report compilation process, there is greater danger that the printing code will break if it is part of the same class.

5-15 lines

Functions should normally be short, between 5-15 lines is my personal "rule of thumb" when coding in Java or C#. This is a good size for several reasons:

• It fits easily on your screen without scrolling
• It's about the conceptual size that you can hold in your head
• It's meaningful enough to require a function in its own right (as a standalone, meaningful chunk of logic)
• A function smaller than 5 lines is a hint that you are perhaps breaking the code up too much (which makes it harder to read / understand if you need to navigate between functions). Either that or your're forgetting your special cases / error handling!

Personal view

I have the SRP in mind when creating Jenkins stages. This implies that I create a separate stage for checkout, build and publish for example. If a stage becomes too large, i.e. longer than 15 lines, then I create a script and run the script in this stage. One of the benefits is readability, possible to apply unit testing on the script and CI agnostic, i.e. migration to another CI would be possible as well.

Example

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                echo 'Building..'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying....'
            }
        }
    }
}

Answer 2

In addition to being a pipeline visualization tool, stages also identify portions of the pipeline that could be executed more or less independently from each other, potentially on different workers. Do not underestimate the importance of choosing the right pipeline structure.

Stages can be re-executed if/when needed, for example in case of failures caused by environment issues, without having to re-run the already completed upstream stages.

Stages provide ability for pipeline branching. For example a single build stage can produce artifacts that could be used by multiple downstream test stages than can be executed in parallel, for shorter overall pipeline execution time.

Each stage operation can be supported by pools of workers of different types and sizes.

Stages can be conditionally executed. For example an expensive integration test stage (lengthy and/or with a smaller resource pool than the build stage) could be executed only once in a while rather than for every commit, or only on a release branch and not on the primary development branch.

Since generally the intent is to have the pipeline triggered by each and every commit in the respective branch the size of the pipeline - i.e. the number of stages, their duration and the structure of the pipeline graph (it doesn't necessarily have to be linear) - has a potentially complex relationship with:

• the rate and profile of the commits triggering the pipelines
• the pool(s) of resources available to support the operation of the pipeline stage(s)
• the role of the branch, the associated balance between the quality level considered mandatory and its mapping onto the verification pipeline structure (i.e. which stage failures are considered blocking, requiring backout/rollback of the respective changes vs. those for which it's acceptable to just file issues to be fixed down the road)
• the amount of additional effort/resources necessary to identify the culprits for and fix the blocking failures detected by the pipeline executions, to complete the (continuous) integration/delivery/deployment process

In smaller scale projects things are usually simple, but in larger scale ones these aspects require serious thinking as they can have a huge impact on the overall development process.

I'll use your first question as an example:

• if you have a dozen of repositories which combined take less than a minute to checkout/pull - it's probably OK to keep things simple and just pull them all in one step.
• if you have several thousands repositories which may take 1h+ to pull you should probably just pull in each stage what you need for that stage. Or maybe pull them all in the first stage, cache them and re-use that cache in later stages if that helps avoid repeating lengthy pulls.

Whatever makes sense, in your particular project context.

Answer 3

Yes, in my perspective stages are mostly meant for a visualisation purpose, and we need to maintain the clear demarcation between stages. My suggestion to go with all the SCM related checkouts in a single stage itself .