5

I want to install a highly available kubernetes cluster (multiple master nodes) on a public cloud (AWS, Azure, GCP and maybe others).
I'm looking for a non-vendor specific option to do it.

I'm aware of several options like

  • Kops
  • Juju
  • Other Github projects I found...

I was wondering if anyone has a good automated example of doing this with such tools like Terraform, Ansible or others.

The HA is a must as this if for production use!

  • Kops is already using Terraform and ansible under the hood IIRC – Tensibai Sep 8 '17 at 22:41
  • 2
    @Tensibai Actually, no. kops interacts directly with AWS API if you run it as is. With the --target=terraform flag, it generates the configuration for you to apply, as I detailed in my answer. – klaus Sep 10 '17 at 18:22
  • @klaus blame my faulty memory :) – Tensibai Sep 10 '17 at 20:09
6

At work we are currenly using a multi AZ Kubernetes cluster on AWS and we're using kops along with Terraform (kops generates the Terraform configuration files) to provision the cluster. What is not clear to me is if your intention is to run a single multi-cloud cluster or if you want to run multiple clusters in multiple clouds.

Anyway, our current setup is a multi-master highly-available multi-AZ Kubernetes cluster. I'll try to explain step by step.

The first thing in order to create the cluster is to generate the Terraform configuration with kops (you could directly apply the changes in AWS by directly using kops, but in our case we think it's best to keep the Terraform files versioned in git, to be able to change details about this configuration and inspect them, also). For instance, this could be the command we used:

kops create cluster \ --name=my_new_cluster \ --ssh-public-key=k8s.pub \ --dns-zone=<your-name>.<your-domain> \ --cloud=aws \ --master-size=t2.medium \ --node-size=t2.medium \ --vpc=<vpc-id> \ --zones=us-east-1a,us-east-1b,us-east-1c \ --master-zones=us-east-1a,us-east-1b,us-east-1c \ --out=my_new_cluster \ --target=terraform

This creates the Terraform configurations (in the directory specified by the --out flag) for a Kubernetes cluster which nodes and masters are distributed along 3 Availability Zones (us-east-1a, us-east-1b and us-east-1c). This implies creating:

  • all necessary subnets, security groups and route tables
  • IAM roles
  • AWS Auto Scaling Groups to manage the cluster nodes and masters (it's the magic of all this, they're the ones that maintain the high availability of the cluster by launching extra nodes as traffic increases, and replaces faulty ones)
  • all EC2 instances and EBS volumes
  • Route53 DNS records to refer to the cluster API and internal DNS for k8s to communicate with pods.

kops uses kubectl internally to create the cluster and all necessary configurations in Kubernetes.

After that, you could terraform apply the configurations and watch it all go up in AWS. It's quite simple, really, but you have to learn how Terraform works. I've found it very intuitive.


Later on, you may need to edit you cluster. There's a command: kops edit cluster <cluster-name> that let you do that. It's all configurations in yaml format. I won't get into specifics but after some changes you may need to kops rolling-update <cluster-name> to apply them.

Hope I've been clear.

Regards.

  • Great, detailed answer. Long live Kubernetes :-) and thank you Terraform! – Eldad AK Sep 10 '17 at 5:33
  • @EldadAK I'm glad I've helped you. Extra: You can also build your cluster using spot instances ;) as per github.com/kubernetes/kops/blob/master/docs/… You need only set the maxPrice and that's it. If this matter interests you, I can get into specifics when implementing this. – klaus Sep 10 '17 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.