0

I would like to setup a Jenkins cluster where the slaves are building an Android ROM with some parameters like branch (e.g. lineage-16.0) and some optional patches. The whole repository will consume around 100GB.

Let's say i want to build branch lineage-16.0 with patch-A and patch-B and in parallel build branch lineage-16.0 with patch-A only and in parallel build branch lineage-17.1 with no patches and so on.

I just read about the --reference option in git (which can also be used in repo). Is it possible to use it this way or will changing the branch and applying patches modify the reference clone as well as it only uses pointers?

I need to save disk space and network traffic somehow. If this is not possible is there any other way without duplicating the 100GB for each jenkins job? I'm also open for other free tools than Jenkins.

-1

Why is your "source repo" 100GB?

Admittedly, the article is several years old, but MS reported the windows code base repo was the largest known repo at the time and was 300GB.

Is your source really that big? Your Git repo should NOT contain derived artifacts (.class, .jar/war, .o, .dll, .exe, etc), nor 3rd-party libraries. There are other tools for that. You can use a .gitignore file to exclude these file types from being checked into your repo.

If you check in a binary file, Git will generally store a complete copy (generally non-compressible) of every copy you check in. I have seen developers check-in not just 3rd-party OSS jars in multiple locations in the same repo and same jars across multiple repos, resulting in massive duplication and wasted space, but then keep checking new versions. I've seen both parts of the Weblogic installer ( 2x 600 MB+) checked-in. And found one which was continuously checking in the Docker image he built which included 3party installers, making his repo almost 1GB larger every check-in!

You don't mention language, but the standard tools for Java libraries has been Nexus and Artifactory. They now handle as extensive range of language artifact types as well.

These tool also create local caches of the artifacts they retrieve so you no longer have to go to the remote host across the network if they have not changed.

Git-LFS is an excellent extension to handle large, non-derived binary files (audio samples, videos, datasets, and graphics).

Look at using BFG to remove unwanted artifacts from history and compress the repo.

As documented in the "largest repo ever" article, MS have contributed the "VFS for Git" to deal with really large repos if that really is the only solution available to you.

This is not an extensive how-to, just advice, "I think you're doing it wrong and that's why it's a PITA", plus some pointers to tools. You need to review your entire approach. That may include other tools like Ansible or Puppet to manage your infra configuration as well.

Summary: Manage your infrastructure as code and keep only source code in your source code repositories. Or just get a faster network.

  • It's not my repository. It's the one for AOSP and LineageOS and other Custom ROMs. I don't have any access to it and can do nothing about their size. I need to mirror and re-use them. – dtrunk Aug 31 '20 at 9:02
  • Then I'd suggest the best place for guidance would be their respective support sites. This ia not a Jenkins issue. I'm reasonably sure you can't be the only user struggling with such larhe repos. They must have guidance ams strategy. The AOSP site tallks about "you can specify --partial-clone when performing repo init which will make use of Git's partial clone capability, which only downloads Git objects when needed instead of downloading everything" – Ian W Aug 31 '20 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.