What are the cost effective ways to deploy services using Kubernetes on multiple deployment environments?

Question

We are converting a traditional docker-compose based service stack to a Kubernetes based services

Before the conversion we had three deployment environments

1. Local : Basically this is the developer's working machine.
2. Dev : Where the everyday releases happens and developers who belongs to other projects like web apps and mobile app uses endpoints from this environment to get their daily work done.
3. Staging : Project owners/clients see this space
4. Production : This is where the world facing actual services are living.

So when its come to Kubernetes, we need to specially invest on Kubernetes supported clusters and those are heavily expensive. Since we need three deployment environments , do we need to purchase three Kubernetes cluster groupes? Or how other traditional teams managing this requirement?

Answer 1

Based on my experience I'd suggest following scenario:

First you should make your apps deployment process similar for all environments, so same tool and set of files will be responsible for this part. And only difference should goes to parameters.

And second it's always a good practice to have kubernetes cluster per stage. Very common situation when your changes goes not only to kubernetes apps manifests, but to surrounding infrastructure also (LoadBalancer, Object Buckets, Disks Storage, etc).

You cat try to cut costs on kubernetes part by using same cluster for dev/stage but still situations with differences in infra resources could happen. And that could lead to more efforts on support and release deployment level.

In case you will use AWS, you can put kuberentes load to EC2 Spot Instances and that could heavily reduce your monthly bill.

As for toolset for k8s apps manifests, I've tried multiple tools and ways, and from my experience terraform+terragrunt works better here. Single tool for all infra resources, parameterization works easily, common code could be separated to reusable modules.

Other option is kustomize or helm, but those tools could help with kubernetes manifests only, and can't manage external resources.