How to defragment etcd database in Kubernetes cluster

Question

I am trying to resolve a recurring Prometheus alert I'm getting that states that my Kubernetes cluster etcd database is fragmented and should be defragmented. I have never used the etcdctl command before, which is apparently installed automatically in the container in which etcd is running.

I have found a post that gives a very brief description of what to do, but it does not work for me. I have also been looking at documentation on the etcd.io site, but everything there seems to assume the user is in the pod, running the etcdctl command directly.

The command I have tried to use is:

kubectl exec $(kubectl get pods --selector=component=etcd -A -o name | head -n 1) -n kube-system -- etcdctl defrag --cluster --cacert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/apiserver.key --cert /etc/kubernetes/pki/etcd/apiserver.crt

I am not 100% sure if I'm specifying the correct file paths for the --key and --cert options. The first link just uses server.key and server.crt, which it complains don't exist for my set-up. I did find the files apiserver.key and apiserver.crt, but maybe those are not the files to use. If they are not, I don't know what should be used.

I'm running this from Git Bash. It errors out with a message that seems to make no sense. It's almost as though it's trying to use the local filesystem (which is a PC running Windows 11).

The error message:

Error: open C:/Program Files/Git/etc/kubernetes/pki/etcd/apiserver.crt: no such file or directory
command terminated with exit code 128

Are these directions correct? Does anyone know why I might be getting the strange error?

UPADTE:

As suggested by @Botje, I ran the following (note the extra slash at the start of each file path):

$ kubectl exec $(kubectl get pods --selector=component=etcd -A -o name | head -n 1) -n kube-system -- etcdctl defrag --cluster --cacert //etc/kubernetes/pki/etcd/ca.crt --key //etc/kubernetes/pki/etcd/apiserver.key --cert //etc/kubernetes/pki/etcd/apiserver.crt

Unfortunately this did not work either:

Error: open //etc/kubernetes/pki/etcd/apiserver.crt: no such file or directory
command terminated with exit code 128

Answer 1

You have to login to that host where etcd pods is running (usually Mater node) and execute :

etcdctl defrag

OR

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379  --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key defrag

prerequisite it to install etcdctl (by installing etcd to that host)

Answer 2

You need to run the command on one of the Kubernetes nodes where the etcd pods are running.

I'm running this from Git Bash. It errors out with a message that seems to make no sense. It's almost as though it's trying to use the local filesystem (which is a PC running Windows 11).

That's because this is exactly what you are doing, and your Windows 11 install doesn't have the certificate files.