Jenkins: Dynamically git clone only the files required for a particular job from a monorepo

Question

We have a huge ~20GB monorepo. For a particular Jenkins job, we do not need to clone the entire repo but only need a few sub-directories. The solution for this is of course git sparse checkout. However this is quiet cumbersome to set up and maintain. You probably have to run the job a few times to make sure you did not miss any directory. Also, in the future if you added a dependency (inside another directory) that was not in the sparse checkout list, you have to remember to add it back.

What if we can improve this process?

1. We should be able to identify all the scripts that a particular Jenkins job needs by looking into all the files that the OS has opened inside that job's working directory.
2. We already have tools like lsof, sysdig, opensnoop which can show us all files that are open inside a directory.
3. On the first ever run of the Jenkins job, we clone the entire repo.
4. Then we start lsof +D /job/workspace and monitor which files got opened.
5. At the end of the job we create a data structure like {JOB_NAME: [list, of, files]} and save it somewhere.
6. Rerun the job ~5 times to ensure data correctness.
7. On all subsequent runs, the git clone command will parse above file and only sparse-checkout the required files.
8. If in the future, the job fails due to a FileNotFound error, we can infer that a new dependency was added, add that new dependency and re-run the job.

Wanted to run the viability of this idea with the DevOps Community.

Answer 1

Assuming I'm understanding you correctly, there's a much simpler way to do this. If you know the path to the file(s) that you need within the git repo, you can use git archive. This fetches a single file directly from a remote repository, so it always grabs the latest version. I've used this a few times from within a Jenkins Pipeline script. For instance:

def cmd = "git archive --remote='${repoUrl}' '${ref}' '${fileName}' | tar -xO"
sh(cmd)

In this example, repoUrl is the remote URL of the git repo, $ref is the ref (tag or branch) to fetch, and fileName is the file to fetch from the repository. If you need more than one file, you can list each one as a separate argument to git archive.

Note that if you're running this from a Pipeline script, you may need to wrap it in withCredentials in order to set up the proper SSH keys or username/password credentials for your git repo.