Sorry for adding a direct answer, I cannot add comments still.
So from what I have worked with and especially in companies that already had legacy infrastructure with legacy Terraform, that was not well written, what I would first suggest:
- Read the book Terraform: Up & Running, 2nd Edition Link, it helped me a lot!
- It will give you an idea of how to structure your Terraform and other amazing things. There is also a git repo for this book:GitHub
- Create modules, as much as you can.
- You can create a module for Cloudfront for example, that can later be reused in other environments.
- Don't use one state file. Split when needed the state file, for example, if you have EC2 instances that use a ec2.tfstate and you have an EKS cluster, don't use them in the same state, create an eks.tfstate file that will be only for the EKS. Another thing is, don't use local tfstate files. If you are in AWS for example create an S3 that is split by live/ /staging/ legacy/ or whatever fits the needs structure and in there put your ec2,eks,cloudwatch or other state files.
- You can split your terraform in multiple repos if needed, but being in one repo, split into folders and environments works in big infrastructure as well. But you have to have some kind of rules, patterns and structure on how to write terraform, so it follows one way across all your engineers.
- Example decide on naming conventions.
- Decide that xx amount of terraform will be built in modules, rather than just random .tf files.
- Decide how you will execute it.
- Manual execution: This approach is not really a good idea as that means an engineer somewhere is doing a terraform apply from his laptop. But its still something you can do if needed.
- Pipeline: If possible as you asked building some kind of CI/CD pipeline that does a terraform init,plan, before terraform apply --auto-approve, so you know what the changes are before being approved, that will help you go back and see on a specific deployment what the terraform plan outputted and have some monitoring over it(also that will allow a few reviews and approvals from other engineers). Also, that means things can be scheduled as well. Terraform works great with Azure DevOps, GitLab and GitHub actions, from what I have worked with.
- To talk about legacy infrastructure. You have to decide is it worth managing a specific infrastructure that was created manually, with Terraform. What I mean is that if after a few months you will migrate that legacy infra and working on importing the legacy infra to terraform will take you a lot of time, by all means, it's not worth it. Before writing terraform, you need to sit down and plan everything: The good, the bad and the not worth it in the long run.
- Make code reviews for Terraform, the same way you would for code.
Going back to point number 1, read the book. Its 600 pages with 70% of it are examples. Go through the repo that I shared and see some example structure, even though that's small, believe me if you split it by env and modules, for example, those 4 points you gave at the top: Cloudfront, API, EC2, Lambdas, if you put them in separate modules folders all under modules, that would be much clearer. It's a small start, but it has to start from somewhere. You cannot import 5 years of infra at once. Take it piece by piece. Also, that's how I would deploy it as well. Deploy a VPC, then deploy on top an EC2, and so on, if you get my point.