2

I work at a municipal government with many departments. Teams from these departments request new servers for specific uses/applications. Our environment is currently using VMware VSphere (~1500 VMs, 2 datacenters, 50 hosts/blades) and we'll be migrating gradually to Azure over the next few years.

I'm just starting with Terraform and can build VMs with it, one at a time. Should I create one Terraform project per server? Or is it worth it to group them somehow? Most of the requests are for one or two servers at a time; sometimes up to four at a time. Each request tends to be unrelated to any previous requests.

Is there a source that covers the theory of organizing something on this scale? Or should I just accept that we'll have to maintain 1500 sets of TF project files?

2 Answers 2

0

I assume this is what Terragrunt would help to solve. It adds a new layer where you can easily manage complex infrastructure or keep your work DRY.

1
  • Alas, Terragrunt is not yet available for Azure. The Terragrunt material shows examples for just a handful of servers and environments. I'm not sure where to find discussions of using Terragrunt/Terraform for >1000 mostly unrelated servers. I mean, I can organize them by city department, but that's still a lot of server project files to manage. It's the scaling from a few to hundreds or more that I'm unclear about. Commented Sep 28, 2022 at 19:53
0

I'll give this a shot because I think it's a very common situation which could use a bit of analysis.

I imagine that due to the large number of internal customers, you don't want to treat all of them as special cases, but instead provide a kind of front-office which they can interact with and order the components they need to build out the service that they want.

Let's start at the end:

Is there a source that covers the theory of organizing something on this scale?

I think what you need here is to apply the concept of a Service Portfolio. Instead of offering "VM's", try to understand what components your internal customers will need, and focus on building an inventory of that instead. Doing this will allow you to focus on the quality of the components, rather than the quality of the finished product. This hearkens back to the Deming principle of "build quality in".

In a well-designed scenario, you will end up having to maintain a handful of components, which will be mapped to a handful of well-architected patterns, instead of a few hundred custom solutions.

Our environment is currently using VMware VSphere (~1500 VMs, 2 datacenters, 50 hosts/blades) and we'll be migrating gradually to Azure over the next few years.

When it comes time to evolve the environment, from VMWare to a cloud, you have to build new modules for the new environment, which do the same thing as their cousins in the old environment. This will allow you to build common interfaces to underlying components so that the users of these components don't have to care about the difference. So, you could have a Terraform module for "A VM with a private network and firewall" which is implemented one way for VMWare and another for Azure, but provides the same functionality to the user. Hashicorp's page on "module composition" gives a good overview of the considerations when combining modules and how to design them for re-usability.

The next concept you could use is that of self-service.

With a set of architectures and components, application owners can pick them the shelf and deploy them with a high-degree of certainty that they will work, and you will be able to provide support for them, because they will fit into well-known categories (or be declared out of scope for support -- similar to voiding the warranty).

With that out of the way, let me take a stab at answering the actual questions:

Should I create one Terraform project per server?

You should create a Terraform module for a pattern, and allow your customers to Terraform your datacentre with them. If you do this on behalf of your customers, then the Terraform definition can reside alongside the code for the actual application in the same repo, or in a sister repo containing only the Terraform definition. This will allow you to create pipelines for specific applications and destroy them cleanly when it comes time to decommission.

Or is it worth it to group them somehow? Most of the requests are for one or two servers at a time; sometimes up to four at a time. Each request tends to be unrelated to any previous requests.

The "grouping" here I interpret as "instantiations of a similar pattern" and it is the reason for making Terraform modules, or components, as I referred to them above. So yes, group them into abstract components, and provide as specific an interface as necessary, via the variables that the module exposes, for them to be useful in well-defined cases, which you then test rigorously.

Or should I just accept that we'll have to maintain 1500 sets of TF project files?

This depends on their lifecycle. You may end up having to write many Terraform definitions, which all consume the same modules -- but do they change at a similar rate? Are they all created and destroyed at the same time? These are considerations for how to create deployment pipelines, if that's what you're doing.

If you are Terraforming by hand, and all of the environments have a similar lifecycle (e.g. every tuesday you have a maintenance window and pull in / apply changes), then you could start considering using some form of wrapper.

Personally, I have avoided Terragrunt because it is an extra layer over something which can already do the job, but your situation might be different. The DRYness in my experience is best introduced via paralellism in CI/CD pipelines, and with appropriate automation at the repo level. Again -this depends very much on your operating environment, and the existing tools you use, so be intentional about the approach.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.