TL;DR; Terraform does allow you to get a lot of what you want done.
Also, you might benefit from a more managed solution like AWS Fargate (which you also declare with Terraform).
Whenever I use search keywords such as "orchestration" or "server clusters" or "iaas" I'll always end up in hundreds of results that are going to talk about containers only.
That is because containers and serverless are the way of the future and most of us are moving our legacy workloads to those technologies. You should try and avoid starting new projects with an EC2 deployment pattern. In particular if you don't have the skillset to manage it all or the money to waste.
My case is to be able to deploy, scale and monitor VM instances such as DO's droplets or AWS' EC2s.
Amazon provides auto scaling groups that allow to scale out EC2 instances.
For monitoring, you either use basic metrics (CPU, Network, etc) or you install an agent to monitor system internals (RAM and other application specific data).
The installation of the agent (either a vendor like Datadog or AWS CloudWatch) is usually done through the instance's "User Data".
Terraform allows you to pass "User Data" scripts to your instances and has a nice templating syntax.
Currently I'm using a CD tool that will use Ansible to spawn (using just REST API) and provision newly created machines. Although neither of these two tools will have capability to monitor created instances, check up on their resources usage, scale them on demand or map them with respective HTTP endpoints (assume 1 VM hosts n webservices).
Correct, as mentioned before Ansible is not the right tool for the job. I would say it is terrible in the space. The main aspect is the lack of state tracking and the fact that it is procedural instead of declarative (this is how I want something done vs this is what I want my end state to be go figure it out).
So to assist CD+Ansible in their jobs I need custom tools that would: keep track of number of instances, their status, DNS server, integration with ingress loadbalancer (haproxy) to keep track of which servers are up and when, what are the IP addresses and/or networks of the newly created droplets and more.
If you are using an AutoScaling group, which seems to be what you are suggesting, you don't track individual instances. You define your minimum, maximum and your scaling criteria. Terraform can definitely declare and deploy the autoscaling group.
The autoscaling group allows you to define health checks to kill instances.
If you are in the autoscaling space you don't want to have individual DNS records for your instances (though for service discovery with HAProxy you might need to do SRV records or integrate with Cloud Map).
Also, if you change load balancer technologies AWS ALB allows you to attach your instances to a target group without you having to do anything.
Both an AWS ALB or HAProxy would also make use of health checks so your orchestration tool doesn't need to know about the health of the instances.
I don't quite understand if terraform is even helpful in this case. From what I understood it'll merely help me with spawning a new droplet which I can as well do with single HTTP request. If I want to have more I need to go with paid cloud?
We have thousands of instances and containers, hundreds of ALBs and a handful of lambdas all managed with Terraform. We let Fargate and other tools manage the blue/green aspects of the deployment process but Terraform defines the infra we want.
What would be a helpful stack to aid my case?
Move to containers, use AWS Fargate so you don't have to manage the orchestration aspects of your autoscaling, health checks, rollbacks, etc.
Drop Ansible, it is not the right tool for this job. Definitely give Terraform a try.
