What are best practices for creating and triggering various pipelines regards monorepos?

Question

According to this article a monorepo contains different projects. What are best practices for triggering different pipelines?

The current git repo contains the following projects:

╔═════════════╗
║ projects    ║
╠═════════════╣
║ A           ║
║ B           ║
║ C           ║
║ D           ║
║ E           ║
╚═════════════╝

There are three different pipelines, i.e. x, y and z

If a certain project inside the monorepo changes, different pipelines should be triggered, i.e.:

If a commit is done in folder a then pipeline x should be triggered (a --> x). If a change occurs in directory b or c then pipeline y should be run (b || c > y) and if either d or dir e changes then pipeline should start running (d || e > z).

How to achieve this? Most CI's are triggered by changes in a repository, e.g. one commits to a repo and a pipeline is triggered. There is a 1 on 1 relation. If this is translated to a monorepo that means that 1 change results in 3 pipeline triggers, while in some cases only 1 pipeline should be triggered.

Answer 1

If this is translated to a monorepo that means that 1 change results in 3 pipeline triggers, while in some cases only 1 pipeline should be triggered.

I would definitely take this point into further consideration. One of the most compelling reasons for using a monorepo is that it does not create a technical separation of projects. There is only a logical separation, with the only technical distinction possibly being your versioning (unless you have a universal version number). This means, that you have no 100% guarantee that any one project does not depend on another project. Because of this, your final pipeline at some point should include all of the projects within your repository. By doing so, you are guaranteeing that a change to one project does not negatively impact any other projects in your repository (at the cost of the project with the longest time to build, publish, etc.).

That being said, if there are some steps in your pipeline that you would like to run separately (lint, syntax, compile, etc.), most CI's should have a way to filter your repo for a specific folder/project. If it doesn't, and you still don't want to run against all projects, you may need to consider refactoring your repository into separate repos or changing CI tools to comply with your workflow.

Answer 2

You need to narrow down your scope and deal with tooling for monorepos. There are various awesome-monorepo lists on Github (this is a recent one) to actually find out, what kind of tooling fits your use case best (what programming languages are you using?).

If you have completely isolated projects in your repository you can cobble up some bash scripting utilizing git diff output or better use one of the many available monorepo-aware built utilities (see above awesome list for examples).