i need to deploy a k8s deployment to be stretched across two kinds of nodes, dedicated and spot nodes.
The end-goal is to have deployment which has some pods on dedicated nodes, but when scaled uses spot nodes.
A deployment has 2 pods (min) and is scalable up to 5 pods (max).The deployment should behave such that the first 2 pods are always scheduled on dedicated nodes and the rest 3 pods (5-2) are always deployed on spot nodes.
Use case:- To improve cost/reliability of my platform.
- Cost:- A prefixed set of pods running at idle vs the max pods running at maximum load gives me an idea of how much my pods are scaling. It also ensures that the end of the day when load subsides,the node counts return back to a fixed predictable count.
- Reliability:- I want to reduce the possibility of outages when the spot nodes are taken away.Ensuring that atleast 2 pods are running on a dedicated nodes, give me confidence that my application does not face an outages should all the spot nodes get deleted.(It does happen!)
What I have tried so far:-
- Labels and Node Affinity:- Node affinity ascertains where the pods will be scheduled. But this approach forces me to select either dedicated or spot nodes. (not a combination of both)
- Pod Anti Affinity:- It works somewhat,it deploy one pod/node till the number of pods = number of dedicated nodes, the starts deployment on spot. But this forces me to limit the size of dedicated nodepool. Also one pod/node is not the use case I am looking for.
Understanding of the problem:-
It seems like a common enough use case where atleast some portion of pods are always running but I have not seen a solution to this anywhere.
It may go against the k8s philosophy where the app should be unaware/not tightly coupled with the infrastructure below but I am not sure if this is the case here.
Running a free for all nodes (no affinity),can cause unbalanced deployments, where scaled pods of one deployment, may prevent any pods of another deployment from getting dedicated node resources.
Let me know if this is achievable and if not , how i can reduce the risk of app outage by running on all spot nodes.