I'm running into some stability problems with Kubernetes and etcd and I was hoping someone could give me some guidance. This isn't a new problem, but an ongoing one that was observed with current and previous versions of etcd and k8s. We're currently using K8s 1.22.2 along with etcd 3.5.

A typical cluster is configured with 3 kubernetes nodes running on 3 VMs hosted on VMWare. The disk devices used are provided by vCenter and backed by an SSD based SAN. Etcd is running on the same VMs as the Kubernetes nodes. The VMs are configured pretty tightly for memory/CPU. The cluster has 6 ESXI hosts and perhaps 300 running VMs. CPU/Memory utilization shown in vCenter is roughly 60%.

In general, this works well, but periodically, we'll get an etcd related error. For example, a scripted kubectl command will fail with an etcd timeout. Checking logs will show a lot of election and duration related messages. On another vcenter with a spindle based SAN, deployment of pods fails on mounting configmap or secret volumes. Reviewing the logs in those cases shows etcd/kubernetes API errors.

I've played with some of the tuning recommendations for etcd (ionice, snapshot size, dedicated etcd disk with dedicated SCSI controller) without result.

I've read through the etcd and kubernetes docs, and it looks to me like the recommendation is that etcd should be run on dedicated hardware (or VMs), with SSD backed storage. As described above, we're putting etcd on the same VMs as the K8s workload. The complete solution is installed on-premises in customer sites, so dedicated VMs or having requirements like non-shared SSD storage isn't possible.

It seems like k8s is so sensitive to etcd performance that even small transient issues cause problems. I think that even with careful tuning, the shared infrastructure (SAN, network, 300 VMs on 6 hosts) is sometimes going to have etcd issues. I'm sure another factor is sharing the etcd VM with the k8s workload as well.

I guess my question is, can this be made to work better? Does anyone have any recommendations on tuning.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.