What are Pros and Cons of various VCS and CI/CD tools to store, change and deploy configuration snippets of more than 1000 customers in a VCS?

Question

Follow-up to this question.

We are in the middle of a transition to Configuration as Code (CaC). At the moment a lot of sites are configured manually. A side effect of this is that so now and then configuration gets broken and nobody knows what happened. By applying CaC the Version Control System (VCS) could be consulted when an environment is broken and then it is easier to figure out the cause of the issue. Another advantage is that people cannot change the configuration directly on the production systems.

One of the comments to this Q&A was whether the configuration is shared between customers. At the moment this is not the case and this is something we would like to prevent as sharing too much could also be a disadvantage as this could cause tangled configuration, i.e. spaghetti.

In summary, there are multiple configuration files per customer site.

Process

When the configuration is changed and merged into master or trunk then the configuration has to be deployed.

There are more than 1000 customer environments that have to be migrated. One approach is to create 1000 VCS repositories. An advantage of this is that a configuration change will result in a deploy to a specific environment, e.g. code pushed to VCS customer0001 will deploy the configuration to customer0001.

Discussion

As there are more than 1000 customers, I wonder whether 1000 repositories will be a good idea in terms of costs, maintenance and so on. What about creating a separate VCS for the configuration and create a folder per customer, i.e. a monorepo. E.g.:

VCS repository

• customer0001
  • configFileA
  • configFileB
  • configFileC
• customer0002
  • configFileA
  • configFileB
  • configFileC
• customer0004
  • configFileA
  • configFileB
  • configFileC
• customer0005
  • configFileA
  • configFileB
  • configFileC
• customer0998
  • configFileA
  • configFileB
  • configFileC
• customer0999
  • configFileA
  • configFileB
  • configFileC
• customer1000
  • configFileA
  • configFileB
  • configFileC

An advantage is that it is clear that all configuration resides in one repository, but how to trigger a certain build if a certain customer folder change and deploy it to the environment of the customer?

It is possible to create a custom groovy script in Jenkins that does some filtering or use branches in certain VCS repositories like bitbucket, but I wonder whether there are more efficient solutions.

Preferred answer

A table that gives an overview of the pros and cons:

| VCS | Monorepo | CICD     | Pros               | Cons                  | 
|-----|----------|----------|====================|=======================|
|     |          | Circleci |                    |                       |
|-----|----------|----------|--------------------|-----------------------|
|     | true     | Jenkins  |                    | groovy script required|
                                                 | to filter customer    |
                                                 | folders               |
                            |--------------------|-----------------------|
|     | false    | Jenkins  | code change results|more than 1000 jenkins |
                            | in deploy to       |jobs required          |
                            | associated customer|                       | 
                            | environment        |                       |
|                           |--------------------|-----------------------|
| git |          |          |                    |                       |

Answer 1

Personally I'd keep them in separate repositories, for clear customer isolation:

• minimal/no risk of unwanted interference between customers
• different access control for different customers is possible
• different CI/CD pipelines and/or configs for different customers is possible
• simpler/standard CI/CD configurations
• clean per-customer repository history
• easier calculation of per customer resource usage
• lower maintenance risks - better to have just a few customers down than all of them at once

I see some cons as well, but IMHO not enough to tip the balance:

• more resources needed (one CI/CD pipeline per customer), but these costs can be reflected in the service pricing. And they should get a lot smaller due to automation.
• sharing of configs/procedures across customers becomes difficult

If a unified VCS vision/presentation is desired I'd still keep separate per-customer repositories (thus keeping some/all of the pros above), collectively managed under a single umbrella repository:

• the umbrella repository would contain just a manifest file listing the per-customer repositories, their versions and maybe some other relevant info
• changes in one or more of the individual repositories need to be accompanied by corresponding manifest changes in the umbrella repository to advance the overall project
• a small set of wrapper tools would be used to operate at the umbrella repository level but controlling the individual repositories