How to exit "Pending" status pods in K8s?

Question

I have scenario where PODs are ending up into “Pending” state.

The reason is that when I launch a POD, I launch it with node affinity because the requirement is that I want to run the POD on specific node.

Now in case of auto-scanning up/down. Sometime the node gets deleted but the POD created to be scheduled on the Node end-up into “Pending” state.

Is there a way that “Pending” pods gets existed/deleted by themself if the Node is not available itself?

Answer 1

The "pending" pods won't get deleted automatically. Pods will get deleted only when the pod object gets deleted. If you want to delete the pods when there are no nodes present, you may need to write a custom solution (something like a cron job which keeps checking the pending pods and deleting them). Also, if the pods are being created by other objects such as statefulsets/deployments etc, you need to delete the object which is responsible for creating the pods. (You can also just scale down the size to 0 in case of statefulsets/deployments)

PS: The pods are in pending state is the default behaviour and whenever a new node is available, the workload will be getting scheduled. This is one of the features of kubernetes which helps in auto healing capability.

Answer 2

By default, if a pod is unable to be scheduled because of a node affinity constraint and the specified node is not available, the pod will remain in a "Pending" state until the node becomes available or the node affinity constraint is removed.

One option to address this issue is to use a pod disruption budget (PDB) to specify the minimum number of available nodes that must be present in the cluster at all times. This can help ensure that there are always enough available nodes to satisfy the node affinity constraints of your pods.

For example, you can use a PDB to specify that at least one node must always be available in the cluster by setting the minAvailable field to 1:

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: my-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: my-app

This PDB would prevent any nodes from being deleted if there are pods with node affinity constraints that are scheduled on those nodes.

Alternatively, you can use a deployment or replica set to manage the lifecycle of your pods and specify a node affinity constraint for the pods in the deployment or replica set. If a node becomes unavailable, the deployment or replica set will automatically reschedule the pods on another available node. This can help ensure that your pods are always running and avoid the "Pending" state.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - node1
      containers:
      - name: my-container
        image: my-image

I hope this helps!

Answer 3

You may delete all pending pods after a while This lists all pending pods

kubectl get pods --all-namespaces --field-selector=status.phase=Pending

This will delete all pending pods that are older than 1 minute

kubectl get pods --all-namespaces --field-selector=status.phase=Pending -o json | jq '.items[] | select((now - (.metadata.creationTimestamp | fromdateiso8601)) > 60) | .metadata.name' | xargs -I{} kubectl delete pod {} --force --grace-period=0