k3s: Eviction manager: attempting to reclaim" resourceName="ephemeral-storage

Question

When I run kubectl get pods -w --namespace kubeapps I see all my pods are pending

NAME                                                         READY   STATUS    RESTARTS   AGE
kubeapps-internal-dashboard-584c686cc6-qwjnf                 0/1     Pending   0          19h
kubeapps-585bc584db-j994c                                    0/1     Pending   0          19h
kubeapps-internal-apprepository-controller-5d7685c4f-zsxnk   0/1     Pending   0          19h
kubeapps-internal-kubeops-7686579468-ddp8z                   0/1     Pending   0          19h
kubeapps-internal-kubeappsapis-6f65bccf89-mzbtn              0/1     Pending   0          19h
kubeapps-internal-dashboard-584c686cc6-wxnfd                 0/1     Pending   0          19h
kubeapps-postgresql-0                                        0/1     Pending   0          19h
kubeapps-585bc584db-rgvfq                                    0/1     Pending   0          19h
kubeapps-internal-kubeops-7686579468-tzjwf                   0/1     Pending   0          19h
kubeapps-internal-kubeappsapis-6f65bccf89-rbdrm              0/1     Pending   0          19h

When I check journalctl -u k3s, I see

E0604 11:37:07.861130    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik: object "kube-system"/"chart-content>
E0604 11:37:07.861184    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217>
E0604 11:37:08.062023    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik-crd: object "kube-system"/"chart-con>
E0604 11:37:08.062042    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik: object "kube-system"/"chart-values-t>
E0604 11:37:08.062060    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik-crd: object "kube-system"/"chart-valu>
E0604 11:37:08.062089    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217>
E0604 11:37:08.062099    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e>
E0604 11:37:08.062105    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e>
I0604 11:37:08.079010    1301 eviction_manager.go:338] "Eviction manager: attempting to reclaim" resourceName="ephemeral-storage"
I0604 11:37:08.079027    1301 container_gc.go:85] "Attempting to delete unused containers"
I0604 11:37:08.079496    1301 image_gc_manager.go:327] "Attempting to delete unused images"
I0604 11:37:08.082310    1301 eviction_manager.go:349] "Eviction manager: must evict pod(s) to reclaim" resourceName="ephemeral-storage"
I0604 11:37:08.082338    1301 eviction_manager.go:367] "Eviction manager: pods ranked for eviction" pods=[kube-system/coredns-d76bd69b-2svkn >
E0604 11:37:08.082353    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/coredns-d76bd69b-2svk>
E0604 11:37:08.082359    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/metrics-server-7cd5fc>
E0604 11:37:08.082363    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/local-path-provisione>
I0604 11:37:08.082368    1301 eviction_manager.go:390] "Eviction manager: unable to evict any pods from the node"

I think this is a result of the eviction manager, because my system is low on disk space? Perhaps it's just looking at percent free, the below is the device from df -h

I'm also seeing this in the error log,

k3s[1301]: I0604 11:52:13.741674    1301 image_gc_manager.go:310] "Disk usage on image filesystem is over the high threshold, trying to free bytes down to the low threshold" usage=98 highThreshold=85 amountToFree=337223212236 lowThreshold=80
k3s[1301]: E0604 11:52:13.742880    1301 kubelet.go:1347] "Image garbage collection failed multiple times in a row" err="failed to garbage collect required amount of images. Wanted to free 337223212236 bytes, but freed 0 bytes"

Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/x1c7--vg-root  1.8T  1.7T   50G  98% /

Where are these settings on k3s?

Answer 1

This is because of the default settings for the eviction manager (currently 5% hard, and 10% minimum for image reclaim) safeguard a new pod exhausting the space on your hard disk, and the metrics are a bit overly conservative. You can change the eviction manager settings by providing a --kubelet-arg that overrides the k3s defaults.. This can be persisted by providing kubelet-arg to your k3s KUBECONFIG at /etc/rancher/k3s/config.yaml.

Like this,

kubelet-arg:
  - "eviction-minimum-reclaim=imagefs.available=2%,nodefs.available=2%"
  - "eviction-hard=memory.available<500Mi,nodefs.available<10Gi"

---

Note, while currently required, passing options by commandline to k3s is deprecated in upstream Kubernetes and is subject to removal in k3s when/if it gets finally removed.