0

Let say there's a daemonset, each pod of this daemonset requests 1 CPU and 1 Gi memory. For some reason pod terminated, during this time kubernetes allocate cpu/memory resource to other pods which causes daemonset pod re-create failed(Insufficient cpu/memory resource).

Is there any way to reserve 1 CPU and 1 Gi memory for this pod during it terminated ?

3
  • Do you have some logs showing this error, and what else was on the same node at the time (and the total capacity of the node)?
    – benjimin
    Dec 28, 2022 at 16:19
  • @benjimin No, this is a assumption. Sorry for the misunderstanding, I just want to know how to preserve some node resource for a future create pod. Dec 29, 2022 at 4:53
  • if you have cluster autoscaling, the new pod will come up fine. Also if you have CPU and memory limits on the pods k8s cannot just allocate more cpu/memory to other pods. either you can remove the cpu/memory limits from your resources (there is a huge debate about this) or enable cluster autoscaling
    – Riv P
    Dec 29, 2022 at 17:18

1 Answer 1

0

You could raise the scheduling priority of your pod so that it will preempt (evict) the other pods if necessary.

You could also check that there isn’t a restart policy causing your pod to end (rather than restart its containers) when the processes terminate.

Normally though, those aren’t supposed to be issues for daemonsets anyway. You shouldn’t need to try to pre-reserve capacity (other than by ensuring the resource requests are ample for the contained processes). You may need to configure your daemonset to skip particular node types that have too little capacity for your collection of daemonsets.

(To reiterate, the most likely cause of the error described by the question is if one of your nodes only has 3GB total of memory, and there are two different daemonsets that request 2GB each of memory, which is unsatisfiable regardless of other pods.)


Daemonsets are controlled quite differently from other pods. For a bare pod (or even a pod from a deployment) that you intend to assign to a particular node (say by setting nodeName or nodeSelector in the PodSpec, or merely by lack of a cluster autoscaler) then definitely do raise the scheduling priority (so that once created it will most likely be able to evict other pods if necessary to make space, notwithstanding several exceptions..). Also keep the default configuration of restarting any containers if they terminate (rather than relinquishing the requested memory back to the scheduler, then recreating a replacement pod..).

It simply is not a kubernetes pattern to reserve resources for a potential future pod. Or rather, you may think of a "pod" API object as already a pre-allocation for the node resources that could be consumed in future by processes in the pod's containers (but allocated before those containers even get created, and usually long before the processes will ever consume that amount of memory). Your pod could use an init container to wait arbitrarily after reserving node resources and before starting the main container process (and more complex orchestration might be achievable with a sidecar container configured to share process namespace..).

An exception is that placeholder pods are sometimes used to over-provision an autoscaled cluster. A pod with very low priority can still trigger the slow process of adding another node to the cluster, so that the new node has a chance to become ready before the scheduler needs it for a normal pod.

9
  • Is there a way to tell kubernetes reserve some resources for a future create pod ? Dec 28, 2022 at 15:45
  • @chenxinlong Why? (How is it that you already know what resources your pod will need, and why don't you wish to create the pod yet, and what is your objection to letting less-important pods try to use the resources in the meantime?)
    – benjimin
    Dec 29, 2022 at 8:55
  • I'm using kubevirt to run vm in pod(vm per pod), after vm shutdown kubevirt will delete this pod. During this vm is shutdown, other new create vm will occupied the shutdown vm's resources(cpu,memory,gpu...), which may causes the shutdown vm can not start successfully(because there's no more allocatable resources for the pod of this shutdown vm). So I was wondering how to reserve resources for these shutdown vms. Dec 29, 2022 at 10:19
  • @chenxinlong Why do you want the pod on that specific node? What do you want to happen to the other pod? Is there some obstacle to adding more nodes?
    – benjimin
    Dec 29, 2022 at 13:03
  • adding more nodes should not even be a thought if the cluster autoscaling is configured properly. if cost is a concern there are so many other ways you can resolve this, you can use SPOT instances, move from intel machines to AMD which are 10 percent cheaper to run on AWS
    – Riv P
    Dec 29, 2022 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.