At work we are currenly using a multi AZ Kubernetes cluster on AWS and we're using
kops along with Terraform (
kops generates the Terraform configuration files) to provision the cluster. What is not clear to me is if your intention is to run a single multi-cloud cluster or if you want to run multiple clusters in multiple clouds.
Anyway, our current setup is a multi-master highly-available multi-AZ Kubernetes cluster. I'll try to explain step by step.
The first thing in order to create the cluster is to generate the Terraform configuration with
kops (you could directly apply the changes in AWS by directly using
kops, but in our case we think it's best to keep the Terraform files versioned in git, to be able to change details about this configuration and inspect them, also). For instance, this could be the command we used:
kops create cluster \
This creates the Terraform configurations (in the directory specified by the
--out flag) for a Kubernetes cluster which nodes and masters are distributed along 3 Availability Zones (us-east-1a, us-east-1b and us-east-1c). This implies creating:
- all necessary subnets, security groups and route tables
- IAM roles
- AWS Auto Scaling Groups to manage the cluster nodes and masters (it's the magic of all this, they're the ones that maintain the high availability of the cluster by launching extra nodes as traffic increases, and replaces faulty ones)
- all EC2 instances and EBS volumes
- Route53 DNS records to refer to the cluster API and internal DNS for k8s to communicate with pods.
kubectl internally to create the cluster and all necessary configurations in Kubernetes.
After that, you could
terraform apply the configurations and watch it all go up in AWS. It's quite simple, really, but you have to learn how Terraform works. I've found it very intuitive.
Later on, you may need to edit you cluster. There's a command:
kops edit cluster <cluster-name> that let you do that. It's all configurations in
yaml format. I won't get into specifics but after some changes you may need to
kops rolling-update <cluster-name> to apply them.
Hope I've been clear.
kopsinteracts directly with AWS API if you run it as is. With the
--target=terraformflag, it generates the configuration for you to apply, as I detailed in my answer.