0

I have a home lab running k3s.

NAME         STATUS   ROLES                         AGE    VERSION
k8s-node-001 Ready    control-plane,master,worker   302d   v1.27.6+k3s1
k8s-node-002 Ready    worker                        302d   v1.27.6+k3s1

I recently removed a node forcefully, since the disk stopped working. (k8s-node-003 was the failure - now deleted from the cluster) Unfortunately that node was running the pvc for my portainer setup.

I looked for a way to "move" the pvc to another node without much luck. So I deleted it kubectl delete pvc portainer -n portainer

I scaled down the portainer deployment with kubectl scale deployment portainer -n portainer --replicas=0 and tried to scale it back up kubectl scale deployment portainer -n portainer --replicas=1

Inspecting the pod under deployment kubectl describe pod portainer-858644df85-sjdbk -n portainer I see that

...

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  95s   default-scheduler  0/2 nodes are available: persistentvolumeclaim "portainer" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
  Warning  FailedScheduling  13s   default-scheduler  0/2 nodes are available: persistentvolumeclaim "portainer" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..

I verified that the claim is gone

kubectl get pv -n portainer portainer

Error from server (NotFound): persistentvolumes "portainer" not found

kubectl get pvc -n portainer portainer

Error from server (NotFound): persistentvolumeclaims "portainer" not found

I guess I should have saved the pvc and then reinstated that using a command like kubectl get pvc --namespace portainer -o json > portainer-pvc.yaml and then reinstated it with something like kubectl apply -f portainer-pvc.yaml -n portainer but 20/20 hindsight is not useful now...

Is it sufficient to find the portainer pvc definition and applying that?

UPDATE

I found the command i used to update portainer and ran that

helm upgrade --install --create-namespace -n portainer portainer portainer/portainer \
     --set enterpriseEdition.enabled=true \
     --set tls.force=true \
     --set tls.existingSecret=portainer-tls-secret

which appears to have reinitialised the pvc

kubectl get pvc -n portainer portainer

NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
portainer   Pending                                      local-path     4m12s

...but it seems that there is some other reference to the defunct node:

Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  4m24s  default-scheduler  0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..

Is there a way to alter the configuration besides wiping the whole portainer configuration and reaapplying it ?

1 Answer 1

0

My saviour was

kubectl edit deploy portainer -n portainer -o yaml

changed the node name, saved the file and watched everything work

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.