How is 'self-healing' to be reconciled with Infrastructure as Code?

Question

As a relative newcomer to the developments happening in Operations - DevOps, SRE, etc. - I'm struggling with a big-picture problem. The following is a specific example, using a network load-balancer.

From an IaC perspective, I take it that we should hold the configuration (ie. the pools for the different virtual servers) outside of the load-balancer itself. Changes to the load-balancer should be made in the first place to this external configuration and applied to the actual application via a pipeline.

From a 'self-healing' perspective - taking automated remedial actions to problems - we might want code to change the network load-balancer configuration in response to some event. If some sensor indicates a problem with a system, for example, we might want to have it removed from a pool.

Now, the obvious way of reconciling these two requirements is for the automated remedial actions to take place via a change to the load-balancer's external configuration. However, some of the reasons for wanting the IaC approach, and some possible implementations of IaC, can make this problematic. For instance, suppose that the external configuration of the load-balancer is held in yaml files in a Git repository, and there are controls over the check-ins (someone senior has to review them). In this case, the remedial action could get stuck in someone's to-do list.

Obviously the problem gets worse the more frequently and more rapidly we want such automated responses.

I was wondering what people's responses to this might be. Is the main problem using something like Git to hold the configuration? Is it a mistake to apply the same controls to all check-ins? Is it a mistake to try to treat the load-balancer set-up as 'configuration' in this way (note that it has some run-time state which isn't captured anywhere, like internal responses to health-checks that it runs). Would appreciate any thoughts.

Answer 1

You do not specify any technology, but let me get an inspiration from one particular approach to this problem and maybe it can be then extended to other use cases. In AWS you have a load balancer (ELB) and an autoscaling group (ASG).

The load balancer has its own health check and the ASG may have its own as well. If an instance stops responding, it is removed from the ELB. If an ASG detects an unhealthy instance, it will just destroy it and it will rebuild a new one, which is then automatically added to the ELB again. The ASG knows its minimal number of instances and its maximum and it maintains the number of actual instances based on some metric (CPU load would be a simple example).

So the only use case when you need to change the configuration is when the system behavior gets out of the expected boundaries. We have some ASGs where we may have the minimum 3 and maximum 30 instances. Usually we have around 4, so if we need more than 30 - the Internet is probably falling down anyway.

Regarding the config change via git - that can work ok together with a full CI/CD pipeline with maybe some blue/green deployment. But maybe it should be considered to have the parameters stored somewhere outside of git unless you want to avoid all reusability and you want to always make copies of the definitions of your infrastructure. You can have an s3 versioned yaml/json file or SSM parameters (speaking still about AWS) or in general a persistent versioned parameter storage (no-sql database may work as well).

As mentioned in the comments - there is no silver bullet here, but it may be useful to stop thinking in the framework of a static universe. If your aim would be to make at least 2 production deployments every day, your "senior reviewer" will need to be probably replaced by something better.