3

Are there established CI/CD strategies & best practices for working with a distributed team on a project that is utilizing monorepos?

I can provide two specific use-cases:

  1. Working on content distribution, e.g. a daily/weekly HTML email newsletter, and

  2. A multi-project directory for building LaTeX documents

Both of these scenarios will utilize the same CI/CD pipeline (build/send an email, compile and upload a LaTeX document) but with a growing & variable amount of content that makes independent repos per document undesirable.

Most of the projects using monorepos currently seem to be javascript projects with a lot of cross-dependencies on code. This is a step a way from that, but I'd still like to embrace the audit trail of version control with an automated development process for producing content.

Do these projects compile every piece of code on change? Is there a way to embrace a git diff or use some sort of 'flag' file in the repo root to use CI/CD tools with a monorepo?

1
+100

The fact that all the code is in a single repository doesn't mean that all the code is changing every time a commit is pushed. I would first make the "pathways" in the code explicit. E.g. perhaps you have a few subdirectories:

- App1/
  - Docs/
  - Code/
  - Tests/
  - Makefile
- App2/
  - Docs/
  - Code/
  - Tests/
  - Makefile

Assuming that App1 is independent from App2, you could trigger it's build by checking if anything in that directory changed (see https://stackoverflow.com/questions/424071/how-to-list-all-the-files-in-a-commit).

Most continuous integration tools have a git plugin or native feature that allows you to filter on a path. If you were using Jenkins, you could set up a few jobs, each with the included region to be be the pathway that should be triggered -- so, e.g. changes to App1/* would trigger jobs/app1/ which executes make -f App1/Makefile or similar.

The trick then to maintaining this project is really in the Makefiles, or other build tool of your choice. You would write the dependencies in there.

You could also have a single global Makefile and use the git diff-tree trick in the question mentioned above:

if grep -q App1 `git diff-tree --no-commit-id --name-only -r <commit>` ; then 
  # trigger pipeline App1
  cd App1
  make build
  make test
  make publish
fi
3
  • What if you change what's in the root directly like .gitlab-ci file? No build would be triggered because no file has been changed in App* – Kaymaz Aug 7 '19 at 20:25
  • of course the logic of the build trigger would have to accomodate your use cases. – Bruce Becker Aug 7 '19 at 20:27
  • This is really missing the fact of libraries and if you build library A, you must also build services C and D that depend on A. (I am trying to figure this out myself having worked at twitter which had it but they used pants build system to do it and pants kinda sucked...it was pretty slow) – Dean Hiller Apr 3 '20 at 23:53
1

I created some showcase for monorepo with Gradle as build tool and CircleCI or Bitbucket pipelines as CI tool.

It's based on the same ideas as in accepted answer and I used similar setup in two projects till now.

See: https://github.com/zladovan/monorepo

Gradle specific stuff should be replaceable quite easily by something else. Just change tools/ci/core/list-dependencies.sh.

1
  • Totally awesome!!!! Added one issue to help so master branch never breaks to the project. – Dean Hiller Apr 4 '20 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.