Jenkins: Re-use repository with reference option and do changes on it

Question

I would like to setup a Jenkins cluster where the slaves are building an Android ROM with some parameters like branch (e.g. lineage-16.0) and some optional patches. The whole repository will consume around 100GB.

Let's say i want to build branch lineage-16.0 with patch-A and patch-B and in parallel build branch lineage-16.0 with patch-A only and in parallel build branch lineage-17.1 with no patches and so on.

I just read about the --reference option in git (which can also be used in repo). Is it possible to use it this way or will changing the branch and applying patches modify the reference clone as well as it only uses pointers?

I need to save disk space and network traffic somehow. If this is not possible is there any other way without duplicating the 100GB for each jenkins job? I'm also open for other free tools than Jenkins.

Answer 1

If this is not possible is there any other way without duplicating the 100GB for each jenkins job?

To avoid frequent clones of the repository, you can put a single clone of the repository on-disk on your executor nodes outside of the normal workspace path. Then, in your Jenkins job, you can cd to that directory with the dir step, update the repo, and checkout the branch you want, e.g.:

dir('/path/to/repo') {
  sh('git fetch --all')
  sh('git checkout origin/lineage-16.0')
  // your build steps here
}

However, this has the major drawback that the one repo will be shared across all of your jobs, which means you cannot run those jobs in parallel without risking corruption. It sounds like running jobs in parallel was one of your requirements, so this may be a no-go.

You could probably get parallel builds to work by using a copy-on-write filesystem so that the repo appears to be in multiple independent directories but only actually has to be cloned once, but I've never tried something like this, so I can't say how well it would work.

Answer 2

Why is your "source repo" 100GB?

Admittedly, the article is several years old, but MS reported the windows code base repo was the largest known repo at the time and was 300GB.

Is your source really that big? Your Git repo should NOT contain derived artifacts (.class, .jar/war, .o, .dll, .exe, etc), nor 3rd-party libraries. There are other tools for that. You can use a .gitignore file to exclude these file types from being checked into your repo.

If you check in a binary file, Git will generally store a complete copy (generally non-compressible) of every copy you check in. I have seen developers check-in not just 3rd-party OSS jars in multiple locations in the same repo and same jars across multiple repos, resulting in massive duplication and wasted space, but then keep checking new versions. I've seen both parts of the Weblogic installer ( 2x 600 MB+) checked-in. And found one which was continuously checking in the Docker image he built which included 3party installers, making his repo almost 1GB larger every check-in!

You don't mention language, but the standard tools for Java libraries has been Nexus and Artifactory. They now handle as extensive range of language artifact types as well.

These tool also create local caches of the artifacts they retrieve so you no longer have to go to the remote host across the network if they have not changed.

Git-LFS is an excellent extension to handle large, non-derived binary files (audio samples, videos, datasets, and graphics).

Look at using BFG to remove unwanted artifacts from history and compress the repo.

As documented in the "largest repo ever" article, MS have contributed the "VFS for Git" to deal with really large repos if that really is the only solution available to you.

This is not an extensive how-to, just advice, "I think you're doing it wrong and that's why it's a PITA", plus some pointers to tools. You need to review your entire approach. That may include other tools like Ansible or Puppet to manage your infra configuration as well.

Summary: Manage your infrastructure as code and keep only source code in your source code repositories. Or just get a faster network.