I want to install a highly available kubernetes cluster (multiple master nodes) on a public cloud (AWS, Azure, GCP and maybe others).
I'm looking for a non-vendor specific option to do it.
I'm aware of several options like
I was wondering if anyone has a good automated example of doing this with such tools like Terraform, Ansible or others.
The HA is a must as this if for production use!
At work we are currenly using a multi AZ Kubernetes cluster on AWS and we're using kops along with Terraform (kops generates the Terraform configuration files) to provision the cluster. What is not clear to me is if your intention is to run a single multi-cloud cluster or if you want to run multiple clusters in multiple clouds.
Anyway, our current setup is a multi-master highly-available multi-AZ Kubernetes cluster. I'll try to explain step by step.
The first thing in order to create the cluster is to generate the Terraform configuration with kops (you could directly apply the changes in AWS by directly using kops, but in our case we think it's best to keep the Terraform files versioned in git, to be able to change details about this configuration and inspect them, also). For instance, this could be the command we used:
kops create cluster \
--name=my_new_cluster \
--ssh-public-key=k8s.pub \
--dns-zone=<your-name>.<your-domain> \
--cloud=aws \
--master-size=t2.medium \
--node-size=t2.medium \
--vpc=<vpc-id> \
--zones=us-east-1a,us-east-1b,us-east-1c \
--master-zones=us-east-1a,us-east-1b,us-east-1c \
--out=my_new_cluster \
--target=terraform
This creates the Terraform configurations (in the directory specified by the --out flag) for a Kubernetes cluster which nodes and masters are distributed along 3 Availability Zones (us-east-1a, us-east-1b and us-east-1c). This implies creating:
kops uses kubectl internally to create the cluster and all necessary configurations in Kubernetes.
After that, you could terraform apply the configurations and watch it all go up in AWS. It's quite simple, really, but you have to learn how Terraform works. I've found it very intuitive.
Later on, you may need to edit you cluster. There's a command: kops edit cluster <cluster-name> that let you do that. It's all configurations in yaml format. I won't get into specifics but after some changes you may need to kops rolling-update <cluster-name> to apply them.
Hope I've been clear.
Regards.
maxPrice and that's it. If this matter interests you, I can get into specifics when implementing this.
(Kublr CTO) Take a look at Kublr - https://kublr.com/
It is designed exactly for a use-case like this: centralized management and operations of multiple Kubernetes clusters in multiple and/or heterogeneous target environments. Currently it supports AWS, Azure, Google Cloud, vSphere, vCloud Director, and on-prem/BYOI (Bring Your Own Infrastructure mode.
In this response - https://devops.stackexchange.com/a/9281/17283 - I described in more details which requirements we considered important when designing Kublr.
kopsinteracts directly with AWS API if you run it as is. With the--target=terraformflag, it generates the configuration for you to apply, as I detailed in my answer.