Best practices for app and infrastructure code repositories

Question

We're looking for guidance on how to structure application code versus infrastructure code (IaC). More specifically, are there typical advantages/disadvantages to storing the two types of code in the same repository versus in different repositories?

A bit more background: We're on the journey to improve our ability to deliver systems more effectively. At the moment, we have application code in one repository and infrastructure code in a separate repository. We've newly begun having the app team change app-level infrastructure code. With that separation of repos, keeping the changes in sync introduces overhead. We think that bringing in the app-specific infrastructure code into the app repository would be advantageous.

Primarily, we're looking for info that helps to answer the question of which is the more effective option, one repo or two.

Answer 1

The correct answer, like almost everything in IT, is, "it depends." It depends on the way you work, the type of company you are working for, your requirements, your non-functional requirements, and possibly a lot of other factors.

tl;dr -- read the final section, it's the most important.

Layers

In this admittedly long answer, I'll talk about the 3 layers of an organisation's IT operations -- infrastructure, platform, and application. These 3 layers are blurred at the boundaries and there is a continual debate about their delineations, but if you're not familiar with this model, roughly:

• Infra: machines and VMs, security and resource groups, auto-scaling groups and elastic load-balancers -- stuff you typically provision with Terraform
• Platform is Kubernetes, OpenShift, IIS, etc. -- stuff you typically provision with Packer, kubectl, or helm (and, this is the first blurring, because EKS, AKS, GKE, are often provisioned with terraform too)
• Application is the business logic and software that runs your business, confusingly this is also often provisioned with kubectl or helm, and possibly packer if you're into the wonderful immutability model

Monorepos

I think the question you're asking relates to the concept of 'monorepo', where all the code needed to build and deploy your application is contained in a single repo and versioned together. In this answer, I'll mainly address the benefits, techniques, and drawbacks of monorepos, though you don't have to go deep into monorepo culture for this to apply. Simply maintaining a single version for all your applications, infra, and platform will have the same effect.

When making your decision you need to factor in whether you are working with an app per cluster, as is becoming more common in modern K8S systems, or a shared cluster. Another factor is which team owns the various layers of the architecture and how they will distribute their output.

Teams & Architecture

My experience comes mainly from large banks. In these organisations, we have hundreds (even thousands) of development teams and it is impractical and inefficient to expect each team to develop its own infrastructure and platform layers as well as their application. Distributing this effort to each team would be a governance and compliance nightmare and hiring that many people with the right skills would be impossible.

Instead, we have centralised teams for infra and platform, and they distribute their output to the development teams and we typically work with the cluster-per-app model. In modern orgs we work with a cross-functional team that has some skills in each of the areas we need, including DevOps, and each team member is also a member of a group that is focused on their speciality where they receive groupwide instruction.

We have 2 main models to provide access to the output of the centralised infra and platform teams:

1. The infra and platform teams can provision clusters on behalf of a dev team and provide the pipelines needed to deploy applications to it. This works well when a team has little or no DevOps capability, but it is far from being the best solution as it creates a dependency from the dev team to the infra and platform teams which usually causes impedance and added complexity.

2. The infra and platform teams can provide access to their git repos, and the dev teams can fork them into git submodules within their own mono-repos. This model is by far the most common and currently represents the best solution for highly regulated industries. One of the biggest benefits here is that a dev team can make the changes they need to make (to the infra and platform) and issue pull-requests for the application teams to manage, without having to coordinate the schedules of the teams and thus reducing impedance.

Both solutions have their problems and complexities, but both are manageable given good engineering leadership and tool support.

Solution 1 would require separate repos for infra, platform, and app, and this usually results in a need to synchronise the backlogs of the respective teams, which quickly becomes a logistical nightmare as the number of teams increases. Solution 2 would require the implementation of 'initial trust' base infra with something like Open Policy Agent (OPA) rules to ensure that dev teams are complying with organisational governance requirements (e.g. verifying that a dev team isn't running a bitcoin mining operation, or publishing pro-trump/fascist/racist propaganda to our public-facing websites). OPA is an amazing tool when decentralising responsibility for the lower layers of an app while maintaining central control and I can't recommend it highly enough.

The app per cluster model allows you to use a monorepo that can completely rebuild your app from the ground up with minimal human interaction. The alternative architecture, multi-tenanted, completely disallows the monorepo approach and you must have separate repos for each of the layers.

Versioning & Configuration Management

The benefits of a monorepo are manifold, but one of the major benefits is having a single version for all layers of your application. One common problem in IT is called Configuration Management -- I refer here to the ITIL meaning, where we talk about which version of what software/hardware work together, rather than the folk-meaning which relates to configuration files.

This is such a hard problem to solve that many books have been written on the subject and ITIL has an entire discipline dedicated to it. Many of the larger banks I have worked for have entire departments dedicated to this discipline and to do it right usually requires the Enterprise Architecture department to keep continually updated models of every component and relationship in an organisation's entire IT estate.

The configuration management problem is largely solved with a monorepo. You'll never hear anyone say "I tried to deploy your software but you've written a helm 3 chart and our cluster only has helm 2 installed," or, "we patched the OS with the latest security updates and your application crashed." With a monorepo, as long as you have sufficient testing in place, your helm version will always be in sync with your helm files and your software will have always been tested with the latest security patches.

Configuration files

I think the OP mentioned config files in one of the comments so I'll address that briefly here too.

If you are building software that a client will run, then configuration files per client or installation are often necessary. In all other cases, they are not needed and should be avoided. Service discovery should be the responsibility of a service-mesh (my preference is Consul), not a configuration file. Secrets should be provided via dynamically generated properties files housed in an in-memory filesystem only accessible to a single instance of your service. All other 'configuration' data should be managed by the platform, and it should be exactly the same for every environment. The days of placing the database name and port, and the database credentials, in your database.properties file were over a long time ago and you should move on now.

It isn't always possible to completely avoid configuration files. Where this is the case the files should preferably be 'owned' by the environment layer, not the application layer. These can be used to enable/disable platform services and control feature flags so as to customise a platform for your team's needs. These should mostly be interpreted by the deployment mechanism to ensure that only the things you actually use are deployed, rather than having things enabled/disabled at runtime, which is a big security no-no. You could use these configurations to down-scale a test environment so that its not as large as a prod environment, or to scale one up for load testing, or even to deploy to a less than six-sigma HA configuration for basic dev work.

Application-specific configuration is considered by many to be a code smell. It is often used to reference database names, ports, and the like. Your app should run in a container and should always run on port 80, so that doesn't need to be configured. Your database should always have a meaningful name and service discovery (DNS) should route you to the appropriate instance for your environment. Your memory and CPU constraints belong to the container, which is in the platform layer, not the app layer.

You may have some truly app-level configuration points but that is normally only true when you are deploying many instances of your apps for different clients (even if they are in your own organisation), in which case this is probably ok, but even then you should work on deriving them from the context in which your app runs, rather than hard-coding them in a config file unless absolutely necessary.

Environments

Another major benefit from the monorepo approach is that you gain the ability to 'print' new environments. An application that is designed to take advantage of this (autoscaling, auto-healing, service-mesh, etc.) will allow you to have a test environment that is 100% aligned with your production environment. Further, if time-to-market is a prominent enough factor of your business plan, you can spin up multiple test environments and run your tests in parallel rather than sequentially.

This also means that developers can have a production-like environment to experiment with whenever they need one. They simply have to run the provisioning scripts for the app and away they go.

Of course, there is a cost to 'printing' environments and not everyone has the unlimited resources of a major bank to fall back on, so this capability must be managed, but if you engineer your environments to be automatically torn down when they're not being actively used then you will go a long way towards reducing these costs. OPA rules can also be used to ensure that boundaries are not overstepped and costs are managed efficiently without human interaction.

Team Dependencies

Personally, I believe that the major factor in the productivity of individual teams is the relationships they necessarily have with their peers in different layers. If there are 300 dev teams, an infrastructure team will be continually inundated with requests for changes that must be understood, managed, prioritised, and scheduled -- this is a major effort in large orgs and can leave a team waiting months, or longer, for changes that support their needs, leading to project delays.

There are no winners here -- if the centralised team is working on your problem then they're not working on someone else's, so either you are delayed or the other 300 teams are. There are always 299 teams waiting to be serviced. It is a lose/lose situation and it simply does not work well at all. Monorepos with git's submodules (or similar structures) help fix this problem. Dev teams can make their own changes to their forked codebase and simply have it reviewed by an authorised representative of the centralised team and pull-requests can be used to manage this process.

Feature Flags

Of course, nothing is perfect. We don't yet have a silver bullet in software engineering and monorepos are no exception -- here be dragons. One fatal but unfortunately common misconception centres around the concept of feature-flags. These flags show what is finished, production-ready, and can be released, and what is a work in progress and should not yet be released to production. Many articles written by engineers suffering from the Dunning-Kruger effect suggest that these flags should be used to ensure that unfinished code should not be executable in a production environment. In reality, these flags should stop code from being released to production at all if it is not yet ready. Non-production code should not be in production -- it is a huge security hole if it is released and this must be avoided.

No choice really

As far as I understand it though, the structure of your repos and software architecture tends to follow the structure of the organisation you work for (see: Conway's Law). Ultimately, the deciding factor may not be related to software at all, but instead is related to your organisation's politics and maturity/sophistication/nature. You may experiment with counter-structures, but usually, these experiments fail to gain traction and are destined for the scrapheap.

If your organisation is split into separate teams with separate responsibilities then your repos will be split the same way. If you go for a cross-functional team approach then your repos will tend to be merged. If you just have one big team that's responsible for the success of the organisation, even if it has specialities then you will naturally end up with a monorepo and life will be good.

If your organisation is poorly structured, following the Harvard method espoused by various MBAs, of splitting the organisation into departments and assigning each of them a vice-president and then grouping them whenever there are more than 7 under a senior VP, then your path to working software will be fraught with problems and progress will be slow. I'm not trying to bash MBAs here, it's just that their models conflict with the primary activity of a lot of modern organisations, which is software development, and I feel that they need to modernise their approaches to take into account the opportunities provided by modern technology and big-data, rather than continuing a model that was developed by candlelight.

Wow, this has turned into a bit of a lecture. Sorry, I don't have time to make it shorter, but as I said up-front, this is a complex subject with a lot of variables. The last section is the most important though, so read that first -- it could save you a lot of pain.

Answer 2

Structuring application code, application configuration and infrastructure in Git repositories is sometimes not the easiest thing to do, and there are not a single correct solution, instead it sometimes depends on how you want to work and what tools you use e.g. monorepo with monorepo tools or one repo per app.

The classic book Continuous Delivery touches on this topic. From my own experience working full time with this problem for a few years on a Kubernetes platform I can confirm that it helps to follow the structure from that book to some extend, but again it also depends on how you want to work. If you use Kubernetes, the 2nd edition of Kubernetes Up & Running has a new chapter 18 about how to structure code and manifests in repositories.

Repositories

The Continuous Delivery book use a repository for application source code and a separate repository for application configuration. The repo for application configuration contains the environment specific configuration for the app, for each environment (e.g. test, staging and prod). When apps are deployed using some for of IaC e.g. Terraform or Kubernetes manifests, this lives in the repo for application configuration. If you are using Kubernetes, Kustomize with manifests and overlays for each environment is useful to use.

The reason to the separation of the source code and configuration here, as explained in the book, is that they have different lifecycles, e.g. they change at different points in a pipeline. An example, you typically don't want to run all your integration test if you only want to change a value in the configuration repo, e.g. a feature-toogle or when you do rollback to a previously working version.

App Build Pipeline

• A repository for application source code
• Every git-push to the repository initiates a Build Pipeline-run
• The build pipeline typically builds the app, test it, build docker images and perhaps run integration tests. If everything has succeeded, it also push the image to an image registry. The last task in the pipeline would be to git-clone the application configuration respository and updated the image reference to the newly built image.
• The result of this pipeline is that it either failed on a certain task, or that it succeeded. And if it succeeded on the mainline (not a feature-branch) you probably want to initiate a Deployment Pipeline, typically with a git-push to the application configuration repo.

App Deployment Pipeline

• A repository for application configuration (and IaC)
• The Build Pipeline typically writes to this repo, for successful builds on the mainline. But also Developers or Operations may want to write to this repo for e.g. configuration changes like feature-toggles or change of URLs, perhaps with a PullRequest reviewed by team members.
• The Deployment Pipeline typically apply the configuration with e.g. kubectl apply or terraform apply using a deployment strategy of choice e.g. Rolling Deployment, A/B-testing deployment or a Canary Deployment - some of these strategies includes two parallel deployments.
• In a more advanced Deployment Pipeline you might want to automatically monitor metrics after the deployment and perhaps rollback if it looks bad. (typically a part of Canary Deployment).

Infrastructure Pipeline

Above I have explained how custom developed applications is deployed using IaC. But there are also other kinds of infrastructure and IaC, like e.g. Kubernetes clusters, load balancers or Kafka clusters. These are typically pre-built software, where your responsibility mostly is about configuration. Pipelines for this kind of infrastructure are similar to App Deployment Pipeline but you may want to run tests and integration tests on Pull Requests to this repository. The book Infrastructure as Code: Dynamic Systems for the Cloud Age is good about this kind of systems, tests and pipelines.

• A repository for infrastructure configuration (possibly Terrafrom, Terragrunt or possibly Kubernetes manifests).
• Tests (e.g. linting) and possibly integration tests (provision the infrastructure in an isolated environment and validate properties)
• Apply configuration (to one ore more environments) e.g. using terragrunt apply or terraform apply or maybe kubectl apply

Terraform Module Pipeline

If you use Terraform, it is common that you want to compose your own modules. This may perhaps be similar for AWS Cloud Formation Stacks. Changes to these modules should be tested in a pipeline and result in a unique version of the module using a git tag.

• A repository for your Terraform module (or perhaps you want to have multiple modules in the same repository).
• A pipeline running Terratest code validating your Terraform module in an isolated environment (e.g. a seperate AWS account).
• Also validation and security scans with e.g. TfSec and TfLint.
• If your test is running on the mainline (e.g. after an approved PR and merge to mainline) you would typically end your pipeline with a task that creates an unique git tag - that you later use with e.g. Terragrunt when you want to apply this version of the module to your infrastructure in the Infrastructure Pipeline.

In the end, it all depends on how you want to work. Maybe you want to work with less process and PRs. Or maybe all your app soruce code is in a big monorepo, then you might want to work in a workflow and structure that fits with the tools you use.

Answer 3

It is definitely advantageous for CICD since you can ship dependent infra and app code changes together, instead of having to manually synchronize it. This means you can run tests that also cover the infrastructure and reduce the likeliness of shipping a breaking change.

But you have to be wary of possible downside, especially pipeline times since infra changes can lead to longer feedback time.

Certain technologies such as pulumi can make this a lot easier, so you might need to research if it makes sense to invest in them, or make do with the current ones in use for your projects.

Answer 4

Personally I would always seperate out the pure IaC code in a different repository. The only "Ops" related thing that should reside in the application repository should be Makefiles, Docker-Compose or pure Dockerfiles to bring up test environments for local or CI/CD tests.

Additionally you should follow the Principle of Least Privilege and therefore you may have developers in the team who should not be able to tear down the whole infrastructure because they can work and look into all the code, states and secrets in the same repository.

If you need to build up Infrastructure after you ship application features like: "We add this new app/microservice which then creates a Pub/Sub environment, a bucket for storage and some other stuff" I would consider looking into Terraform Modules (or as Michael Pereira already mentioned Pulumi). Maybe cdk8s could also fit, but this all depends on what "IaC" you are using and how.