What do you need for a monorepo for a large codebase?

Question

From certain size of the codebase, would you still have Git or are there more specialized solutions?

(Also to checkout just a part of the codebase)

Answer 1

The answer is: a bit of both. To satisfy the constraints of "use git" and "manage a vast codebase" Microsoft developed a new filesystem (previously they were using a variant of Perforce called SourceDepot). It's open source but I have no personal experience of using it.

Why would you want a monorepo? The most obvious reason is that you can modify an API and all the callers of that API in an atomic commit. Also there are advantages to being able to do a git log search across the entire codebase...

Answer 2

Git works for monorepos, but it has a few problems:

1. You have to check out the entire repo.
2. You have to fetch the entire history (generally - shallow clones are an option, but usually not useful in actual development work).
3. Natively, everyone has read+write access to every directory if they have it at all.

Google, probably the most famous monorepo user, developed Piper to handle their needs. But you aren't Google, and so their solutions are probably not yours.

One of the key advantages to a monorepo is that you can make globally-atomic changes (i.e. you don't need to version many things because you can change the caller and the callee in the same commit). To power this, you really want to have a unified build system that tracks dependencies across the entire repo. Bazel is an open-source extraction of Google's build system, Blaze, and it attempts to do that (although it's young and immature and missing a lot of features that are necessary for non-Google use). Pants is a similar system out of Twitter.

If you're building tons of code when you make such an atomic change, then you probably also want a build farm that allows you to do that not on your local machine. Similarly, you'll need a powerful CI system to handle running tests across everything as you update.

Answer 3

Opinions differ on what a large code base is. If you are speaking about a company with 100 engineers, I would argue that Git should still be able to handle it. It has been developed for the needs of the Linux kernel, which is not a small project on its own.

Independent of the way how you store the repository, you may run into problems. For instance, if you are working on a large Java code base and are using tools like Eclipse or IntelliJ, they will use more memory and generally become slower.

On the other hand, having the option to operate on all code at once (e.g., when applying refactoring, or source code transformations) is one of the main advantages of monolithic repositories.

When you ask about whether you need specialized tools, then up a certain code size, the answer is yes. According to Google, which arguably has the biggest C++ codebase in the world, all available tools (open source or commercial) did not satisfy their requirements. They ended up developing an inhouse system called Piper:

• Why Google Stores Billions of Lines of Code in a Single Repository
• What version control system does Google use and why?

Answer 4

If I understand it correctly the "need" for a monorepo is simply the fundamental need of a single/coherent versioning scheme applied to a software project containing multiple loosely-related components/sub-projects which are/could be otherwise managed/versioned independently in separate repositories.

Similar, if you want, with the need of using a regular source repository to provide a single/coherent versioning scheme for a multitude of source files, each with their own, independent modification history.

Using an actual monorepo solution is definitely one, but IMHO not the only way of addressing this need.

Another possible approach is using an umbrella project repository containing one or more manifest files with the exact version of each of the individual project component repositories.

Even if the component repositories have their versions modified by independent, non-atomic commits, the project itself can be managed coherently simply by combining all related component repository version changes into a single commit to the manifest file(s) in the umbrella repository.

Such approach has several advantages over migrating to an actual monorepo solution:

• no need to change the existing component repositories
• can support mixes of components with different repository technologies
• each component repository can still be developed and managed independently
• adding/deleting project components is almost trivial
• integrating 3rd party (upstream) components is a whole lot easier
• the project history can be kept much cleaner, not polluted with all the details of each individual component repository change (which typically would be irrelevant for the other components)
• no need to worry about size/performance/scalability of a single repository, the solution itself is highly scalable.