0

Following directions from here. End goal is upgrading cluster to 1.24 and beyond.

At the end when I run kubectl get nodes -o wide seems like I would see 'containerd' as the runtime (link) but instead I see docker://23.0.1. Granted - the other nodes are showing docker://1.13.1 so its an improvement?

On the node I see many instances of /usr/bin/containerd-shim-runc-v2 whereas on other nodes I see /usr/bin/docker-containerd-shim-current so its definitely different.

Anyway - what step(s) did I miss?

1 Answer 1

2

First check what kubelet says

journalctl -u kubelet

What worked for me was that in the plugins section in /etc/containerd/config.toml that systemdcgroup had to be true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            SystemdCgroup = true

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

systemctl restart containerd after that of course

On another cluster the logs said something like "server http response to https client" so containerd did not end up pulling images.

The solution for that was to turn on TLS skip verify

 [plugins."io.containerd.grpc.v1.cri".registry.configs."<MASTERIP>:5000".tls]
          insecure_skip_verify = true

and most important setting grpc.mirrors endpoint as http.

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."<MASTERIP>:5000"]
  endpoint = ["http://<MASTERIP>:5000"]

So one cluster on k8s V1.26 has the systemdcgroup as true and https enabled and the other on k8s V1.23 has systemdcgroup as true and http enabled. Only after those kubectl get nodes showed a CRI of containerd on both

4
  • I updated this setting but unf it didn't fix the problem. Somewhere along the way Ive taken a wrong turn.
    – ethrbunny
    Commented Mar 10, 2023 at 14:49
  • 1
    updated with another thing I remembered over the weekend that might solve the problem Commented Mar 13, 2023 at 6:19
  • 1
    This mainly solves the problem. Note that (at least in my case) /var/lib/kubectl/kubeadm-flags.env doesn't have a trailing " by default. This caused me no end of headaches.
    – ethrbunny
    Commented Mar 20, 2023 at 12:41
  • those things can take a long time indeed Commented Mar 21, 2023 at 5:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.