When I run kubectl get pods -w --namespace kubeapps I see all my pods are pending

NAME                                                         READY   STATUS    RESTARTS   AGE
kubeapps-internal-dashboard-584c686cc6-qwjnf                 0/1     Pending   0          19h
kubeapps-585bc584db-j994c                                    0/1     Pending   0          19h
kubeapps-internal-apprepository-controller-5d7685c4f-zsxnk   0/1     Pending   0          19h
kubeapps-internal-kubeops-7686579468-ddp8z                   0/1     Pending   0          19h
kubeapps-internal-kubeappsapis-6f65bccf89-mzbtn              0/1     Pending   0          19h
kubeapps-internal-dashboard-584c686cc6-wxnfd                 0/1     Pending   0          19h
kubeapps-postgresql-0                                        0/1     Pending   0          19h
kubeapps-585bc584db-rgvfq                                    0/1     Pending   0          19h
kubeapps-internal-kubeops-7686579468-tzjwf                   0/1     Pending   0          19h
kubeapps-internal-kubeappsapis-6f65bccf89-rbdrm              0/1     Pending   0          19h

When I check journalctl -u k3s, I see

E0604 11:37:07.861130    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik: object "kube-system"/"chart-content>
E0604 11:37:07.861184    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217>
E0604 11:37:08.062023    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik-crd: object "kube-system"/"chart-con>
E0604 11:37:08.062042    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik: object "kube-system"/"chart-values-t>
E0604 11:37:08.062060    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik-crd: object "kube-system"/"chart-valu>
E0604 11:37:08.062089    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217>
E0604 11:37:08.062099    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e>
E0604 11:37:08.062105    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e>
I0604 11:37:08.079010    1301 eviction_manager.go:338] "Eviction manager: attempting to reclaim" resourceName="ephemeral-storage"
I0604 11:37:08.079027    1301 container_gc.go:85] "Attempting to delete unused containers"
I0604 11:37:08.079496    1301 image_gc_manager.go:327] "Attempting to delete unused images"
I0604 11:37:08.082310    1301 eviction_manager.go:349] "Eviction manager: must evict pod(s) to reclaim" resourceName="ephemeral-storage"
I0604 11:37:08.082338    1301 eviction_manager.go:367] "Eviction manager: pods ranked for eviction" pods=[kube-system/coredns-d76bd69b-2svkn >
E0604 11:37:08.082353    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/coredns-d76bd69b-2svk>
E0604 11:37:08.082359    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/metrics-server-7cd5fc>
E0604 11:37:08.082363    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/local-path-provisione>
I0604 11:37:08.082368    1301 eviction_manager.go:390] "Eviction manager: unable to evict any pods from the node"

I think this is a result of the eviction manager, because my system is low on disk space? Perhaps it's just looking at percent free, the below is the device from df -h

I'm also seeing this in the error log,

k3s[1301]: I0604 11:52:13.741674    1301 image_gc_manager.go:310] "Disk usage on image filesystem is over the high threshold, trying to free bytes down to the low threshold" usage=98 highThreshold=85 amountToFree=337223212236 lowThreshold=80
k3s[1301]: E0604 11:52:13.742880    1301 kubelet.go:1347] "Image garbage collection failed multiple times in a row" err="failed to garbage collect required amount of images. Wanted to free 337223212236 bytes, but freed 0 bytes"
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/x1c7--vg-root  1.8T  1.7T   50G  98% /

Where are these settings on k3s?

1 Answer 1


This is because of the default settings for the eviction manager (currently 5% hard, and 10% minimum for image reclaim) safeguard a new pod exhausting the space on your hard disk, and the metrics are a bit overly conservative. You can change the eviction manager settings by providing a --kubelet-arg that overrides the k3s defaults.. This can be persisted by providing kubelet-arg to your k3s KUBECONFIG at /etc/rancher/k3s/config.yaml.

Like this,

  - "eviction-minimum-reclaim=imagefs.available=2%,nodefs.available=2%"
  - "eviction-hard=memory.available<500Mi,nodefs.available<10Gi"

Note, while currently required, passing options by commandline to k3s is deprecated in upstream Kubernetes and is subject to removal in k3s when/if it gets finally removed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.