The correct answer, like almost everything in IT, is, "it depends." It depends on the way you work, the type of company you are working for, your requirements, your non-functional requirements, and possibly a lot of other factors.
tl;dr -- read the final section, it's the most important.
In this admittedly long answer, I'll talk about the 3 layers of an organisation's IT operations -- infrastructure, platform, and application. These 3 layers are blurred at the boundaries and there is a continual debate about their delineations, but if you're not familiar with this model, roughly:
- Infra: machines and VMs, security and resource groups, auto-scaling groups and elastic load-balancers -- stuff you typically provision with Terraform
- Platform is Kubernetes, OpenShift, IIS, etc. -- stuff you typically provision with Packer, kubectl, or helm (and, this is the first blurring, because EKS, AKS, GKE, are often provisioned with terraform too)
- Application is the business logic and software that runs your business, confusingly this is also often provisioned with kubectl or helm, and possibly packer if you're into the wonderful immutability model
I think the question you're asking relates to the concept of 'monorepo', where all the code needed to build and deploy your application is contained in a single repo and versioned together. In this answer, I'll mainly address the benefits, techniques, and drawbacks of monorepos, though you don't have to go deep into monorepo culture for this to apply. Simply maintaining a single version for all your applications, infra, and platform will have the same effect.
When making your decision you need to factor in whether you are working with an app per cluster, as is becoming more common in modern K8S systems, or a shared cluster. Another factor is which team owns the various layers of the architecture and how they will distribute their output.
Teams & Architecture
My experience comes mainly from large banks. In these organisations, we have hundreds (even thousands) of development teams and it is impractical and inefficient to expect each team to develop its own infrastructure and platform layers as well as their application. Distributing this effort to each team would be a governance and compliance nightmare and hiring that many people with the right skills would be impossible.
Instead, we have centralised teams for infra and platform, and they distribute their output to the development teams and we typically work with the cluster-per-app model. In modern orgs we work with a cross-functional team that has some skills in each of the areas we need, including DevOps, and each team member is also a member of a group that is focused on their speciality where they receive groupwide instruction.
We have 2 main models to provide access to the output of the centralised infra and platform teams:
The infra and platform teams can provision clusters on behalf of a dev team and provide the pipelines needed to deploy applications to it. This works well when a team has little or no DevOps capability, but it is far from being the best solution as it creates a dependency from the dev team to the infra and platform teams which usually causes impedance and added complexity.
The infra and platform teams can provide access to their git repos, and the dev teams can fork them into git submodules within their own mono-repos. This model is by far the most common and currently represents the best solution for highly regulated industries. One of the biggest benefits here is that a dev team can make the changes they need to make (to the infra and platform) and issue pull-requests for the application teams to manage, without having to coordinate the schedules of the teams and thus reducing impedance.
Both solutions have their problems and complexities, but both are manageable given good engineering leadership and tool support.
Solution 1 would require separate repos for infra, platform, and app, and this usually results in a need to synchronise the backlogs of the respective teams, which quickly becomes a logistical nightmare as the number of teams increases. Solution 2 would require the implementation of 'initial trust' base infra with something like Open Policy Agent (OPA) rules to ensure that dev teams are complying with organisational governance requirements (e.g. verifying that a dev team isn't running a bitcoin mining operation, or publishing pro-trump/fascist/racist propaganda to our public-facing websites). OPA is an amazing tool when decentralising responsibility for the lower layers of an app while maintaining central control and I can't recommend it highly enough.
The app per cluster model allows you to use a monorepo that can completely rebuild your app from the ground up with minimal human interaction. The alternative architecture, multi-tenanted, completely disallows the monorepo approach and you must have separate repos for each of the layers.
Versioning & Configuration Management
The benefits of a monorepo are manifold, but one of the major benefits is having a single version for all layers of your application. One common problem in IT is called Configuration Management -- I refer here to the ITIL meaning, where we talk about which version of what software/hardware work together, rather than the folk-meaning which relates to configuration files.
This is such a hard problem to solve that many books have been written on the subject and ITIL has an entire discipline dedicated to it. Many of the larger banks I have worked for have entire departments dedicated to this discipline and to do it right usually requires the Enterprise Architecture department to keep continually updated models of every component and relationship in an organisation's entire IT estate.
The configuration management problem is largely solved with a monorepo. You'll never hear anyone say "I tried to deploy your software but you've written a helm 3 chart and our cluster only has helm 2 installed," or, "we patched the OS with the latest security updates and your application crashed." With a monorepo, as long as you have sufficient testing in place, your helm version will always be in sync with your helm files and your software will have always been tested with the latest security patches.
I think the OP mentioned config files in one of the comments so I'll address that briefly here too.
If you are building software that a client will run, then configuration files per client or installation are often necessary. In all other cases, they are not needed and should be avoided. Service discovery should be the responsibility of a service-mesh (my preference is Consul), not a configuration file. Secrets should be provided via dynamically generated properties files housed in an in-memory filesystem only accessible to a single instance of your service. All other 'configuration' data should be managed by the platform, and it should be exactly the same for every environment. The days of placing the database name and port, and the database credentials, in your
database.properties file were over a long time ago and you should move on now.
It isn't always possible to completely avoid configuration files. Where this is the case the files should preferably be 'owned' by the environment layer, not the application layer. These can be used to enable/disable platform services and control feature flags so as to customise a platform for your team's needs. These should mostly be interpreted by the deployment mechanism to ensure that only the things you actually use are deployed, rather than having things enabled/disabled at runtime, which is a big security no-no. You could use these configurations to down-scale a test environment so that its not as large as a prod environment, or to scale one up for load testing, or even to deploy to a less than six-sigma HA configuration for basic dev work.
Application-specific configuration is considered by many to be a code smell. It is often used to reference database names, ports, and the like. Your app should run in a container and should always run on port 80, so that doesn't need to be configured. Your database should always have a meaningful name and service discovery (DNS) should route you to the appropriate instance for your environment. Your memory and CPU constraints belong to the container, which is in the platform layer, not the app layer.
You may have some truly app-level configuration points but that is normally only true when you are deploying many instances of your apps for different clients (even if they are in your own organisation), in which case this is probably ok, but even then you should work on deriving them from the context in which your app runs, rather than hard-coding them in a config file unless absolutely necessary.
Another major benefit from the monorepo approach is that you gain the ability to 'print' new environments. An application that is designed to take advantage of this (autoscaling, auto-healing, service-mesh, etc.) will allow you to have a test environment that is 100% aligned with your production environment. Further, if time-to-market is a prominent enough factor of your business plan, you can spin up multiple test environments and run your tests in parallel rather than sequentially.
This also means that developers can have a production-like environment to experiment with whenever they need one. They simply have to run the provisioning scripts for the app and away they go.
Of course, there is a cost to 'printing' environments and not everyone has the unlimited resources of a major bank to fall back on, so this capability must be managed, but if you engineer your environments to be automatically torn down when they're not being actively used then you will go a long way towards reducing these costs. OPA rules can also be used to ensure that boundaries are not overstepped and costs are managed efficiently without human interaction.
Personally, I believe that the major factor in the productivity of individual teams is the relationships they necessarily have with their peers in different layers. If there are 300 dev teams, an infrastructure team will be continually inundated with requests for changes that must be understood, managed, prioritised, and scheduled -- this is a major effort in large orgs and can leave a team waiting months, or longer, for changes that support their needs, leading to project delays.
There are no winners here -- if the centralised team is working on your problem then they're not working on someone else's, so either you are delayed or the other 300 teams are. There are always 299 teams waiting to be serviced. It is a lose/lose situation and it simply does not work well at all. Monorepos with git's submodules (or similar structures) help fix this problem. Dev teams can make their own changes to their forked codebase and simply have it reviewed by an authorised representative of the centralised team and pull-requests can be used to manage this process.
Of course, nothing is perfect. We don't yet have a silver bullet in software engineering and monorepos are no exception -- here be dragons. One fatal but unfortunately common misconception centres around the concept of feature-flags. These flags show what is finished, production-ready, and can be released, and what is a work in progress and should not yet be released to production. Many articles written by engineers suffering from the Dunning-Kruger effect suggest that these flags should be used to ensure that unfinished code should not be executable in a production environment. In reality, these flags should stop code from being released to production at all if it is not yet ready. Non-production code should not be in production -- it is a huge security hole if it is released and this must be avoided.
No choice really
As far as I understand it though, the structure of your repos and software architecture tends to follow the structure of the organisation you work for (see: Conway's Law). Ultimately, the deciding factor may not be related to software at all, but instead is related to your organisation's politics and maturity/sophistication/nature. You may experiment with counter-structures, but usually, these experiments fail to gain traction and are destined for the scrapheap.
If your organisation is split into separate teams with separate responsibilities then your repos will be split the same way. If you go for a cross-functional team approach then your repos will tend to be merged. If you just have one big team that's responsible for the success of the organisation, even if it has specialities then you will naturally end up with a monorepo and life will be good.
If your organisation is poorly structured, following the Harvard method espoused by various MBAs, of splitting the organisation into departments and assigning each of them a vice-president and then grouping them whenever there are more than 7 under a senior VP, then your path to working software will be fraught with problems and progress will be slow. I'm not trying to bash MBAs here, it's just that their models conflict with the primary activity of a lot of modern organisations, which is software development, and I feel that they need to modernise their approaches to take into account the opportunities provided by modern technology and big-data, rather than continuing a model that was developed by candlelight.
Wow, this has turned into a bit of a lecture. Sorry, I don't have time to make it shorter, but as I said up-front, this is a complex subject with a lot of variables. The last section is the most important though, so read that first -- it could save you a lot of pain.