I had a node get automatically removed from our kubernetes cluster by the autoscaler. However, it appears the scheduler didn't get the memo, and it's still trying to assign pods to the dead node:

I0508 18:42:38.615641       7 scheduler.go:166] Failed to schedule pod: default/auth--web-1859671213-t95st
E0508 18:42:38.615718       7 factory.go:659] Error scheduling default auth--web-1859671213-t95st: node 'ip-10-100-70-212.ec2.internal' not found; retrying
I0508 18:42:38.615778       7 factory.go:734] Updating pod condition for default/auth--web-1859671213-t95st to (PodScheduled==False)
I0508 18:42:38.615915       7 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"auth--web-1859671213-t95st", UID:"8f42739e-71b3-11e9-b6cb-06d5afd487fe", APIVersion:"v1", ResourceVersion:"183075022", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' node 'ip-10-100-70-212.ec2.internal' not found

A kubectl get nodes confirms that the node is no longer a member of the cluster. I can see the event in the AWS auto scaling group logs where the node was removed. It's definitely dead and gone.

I found a series of bug reports where this happened while leaving the scheduler in bad state, culminating in a PR to fix the problem:

However, the version of kubernetes we're running is old (1.7), so we're currently missing out on the patch. Clearly, I should upgrade the cluster to a modern version.

But I'd still like to get the Pending pods back up and running first. Is there a way I can manually force the scheduler to update its cache, or to otherwise force the pods to be rescheduled onto a node that actually exists?

Things I've already tried:

  • Adding more nodes to the cluster, to verify that there are enough resources to run the pods.
  • Restarting the scheduler, it comes back online with the same error.
  • Deleting the Pending pod, the replacement just tries to schedule back onto the same dead node.
  • 1
    We've abandoned this cluster for a new one running a modern version of kubernetes. The old cluster is shut down, it's unlikely we'll ever be able to debug this issue further again. – Brad Koch May 20 '19 at 20:24
  • Thanks for the update. I assume that the reported issue does not appear in the new cluster? – 030 Dec 25 '19 at 9:41
  • 1
    Yeah, issue has never shown up again, likely because the new cluster is running code with the mentioned patch. No point trying to repro this, any reasonably recent build should be unaffected. – Brad Koch Dec 25 '19 at 21:05

Browse other questions tagged or ask your own question.