DevOps Stack Exchange is a question and answer site for software engineers working on automated testing, continuous delivery, service integration and monitoring, and building SDLC infrastructure. It only takes a minute to sign up.
Sign up to join this community
Let say there's a daemonset, each pod of this daemonset requests 1 CPU and 1 Gi memory. For some reason pod terminated, during this time kubernetes allocate cpu/memory resource to other pods which causes daemonset pod re-create failed(Insufficient cpu/memory resource).
Is there any way to reserve 1 CPU and 1 Gi memory for this pod during it terminated ?
You could raise the scheduling priority of your pod so that it will preempt (evict) the other pods if necessary.
You could also check that there isn’t a restart policy causing your pod to end (rather than restart its containers) when the processes terminate.
Normally though, those aren’t supposed to be issues for daemonsets anyway. You shouldn’t need to try to pre-reserve capacity (other than by ensuring the resource requests are ample for the contained processes). You may need to configure your daemonset to skip particular node types that have too little capacity for your collection of daemonsets.
(To reiterate, the most likely cause of the error described by the question is if one of your nodes only has 3GB total of memory, and there are two different daemonsets that request 2GB each of memory, which is unsatisfiable regardless of other pods.)
Daemonsets are controlled quite differently from other pods. For a bare pod (or even a pod from a deployment) that you intend to assign to a particular node (say by setting nodeName or nodeSelector in the PodSpec, or merely by lack of a cluster autoscaler) then definitely do raise the scheduling priority (so that once created it will most likely be able to evict other pods if necessary to make space, notwithstanding several exceptions..). Also keep the default configuration of restarting any containers if they terminate (rather than relinquishing the requested memory back to the scheduler, then recreating a replacement pod..).
It simply is not a kubernetes pattern to reserve resources for a potential future pod. Or rather, you may think of a "pod" API object as already a pre-allocation for the node resources that could be consumed in future by processes in the pod's containers (but allocated before those containers even get created, and usually long before the processes will ever consume that amount of memory). Your pod could use an init container to wait arbitrarily after reserving node resources and before starting the main container process (and more complex orchestration might be achievable with a sidecar container configured to share process namespace..).
An exception is that placeholder pods are sometimes used to over-provision an autoscaled cluster. A pod with very low priority can still trigger the slow process of adding another node to the cluster, so that the new node has a chance to become ready before the scheduler needs it for a normal pod.
