Why is the containerd still pulling from the network even though the server contains an image

Question

Today I found the kubernetes pod shows error like this:

Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image "registry.k8s.io/pause:3.6": failed to pull image "registry.k8s.io/pause:3.6": failed to pull and unpack image "registry.k8s.io/pause:3.6": failed to resolve reference "registry.k8s.io/pause:3.6": failed to do request: Head "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.6": dial tcp 64.233.187.82:443: i/o timeout

Because of the network policy, I can not access the registry.k8s.io. So I load the pause image into the server like this:

ctr -n=k8s.io image import /root/pause.tar

this is the image list:

[root@k8sslave01 ~]# ctr -n=k8s.io image list|grep "pause"
registry.k8s.io/pause:3.6                                                                                                         application/vnd.docker.distribution.manifest.v2+json      sha256:79b611631c0d19e9a975fb0a8511e5153789b4c26610d1842e9f735c57cc8b13 669.8 KiB linux/amd64                                                                   io.cri-containerd.image=managed
[root@k8sslave01 ~]#

it worked for weeks but today it seems like it did not pull from local first. is it possible to fix this issue?

Answer 1

There 2 are places that defines pause image tag, if you run kubernetes using containerd.

1. On containerd configuration file.

   You can check running configuration by running

    containerd config dump | grep sandbox
   

You should modify this part on /etc/containerd/config.toml, so it similar as below:

    [plugins]
       [plugins."io.containerd.grpc.v1.cri"]
         sandbox_image = "yourregistryserver/pause:3.6"

This part defines default pause image.

2. kubelet configuration on /var/lib/kubelet/kubeadm-flags.env.

   This part defines pause image used by current kubernetes version. You will find configuration like this:

    KUBELET_KUBEADM_ARGS=".... --pod-infra-container-image=yourregistryserver/pause:3.8"
   

As far as I recall, containerd configuration relevant only during initial initialisation. You can modify the value in both places if you want to be sure.