0

I have large Kubernates jobs which take the whole resources of one node / droplet in Digitalocean (it's not running on AWS).

Using resources in a deploy.yaml, the cluster autoscaler works to scale up, but sometimes the nodes continue to run instead of killing / scaling down when the job is done.

How do I make sure the nodes get killed when the job is done? They are expensive!

1

When a Job completes, no more Pods are created, but the Pods are not deleted either.

https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-termination-and-cleanup

  1. clear it by another cronjob.
  2. use operator to wrapper it and add remove mechanism in the operator.
2
  • Thanks - I don't mind if the pods stay around, but I want the actual nodes to scale back down automatically. Which they do sometimes but not always. May 15 '21 at 11:35
  • The hanging pod could be the reason that clusterscaler does not scale down the node. Try to manually delete them to verify if it works. How do you ensure that the job takes the whole resource of the node? If via the required resources, you might try the node selector + podAntiAfinity and only requires fewer resources.
    – Jerry Chen
    May 16 '21 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.