0

i need to deploy a k8s deployment to be stretched across two kinds of nodes, dedicated and spot nodes.

The end-goal is to have deployment which has some pods on dedicated nodes, but when scaled uses spot nodes.

Example:-

A deployment has 2 pods (min) and is scalable up to 5 pods (max).The deployment should behave such that the first 2 pods are always scheduled on dedicated nodes and the rest 3 pods (5-2) are always deployed on spot nodes.

Use case:- To improve cost/reliability of my platform.

  • Cost:- A prefixed set of pods running at idle vs the max pods running at maximum load gives me an idea of how much my pods are scaling. It also ensures that the end of the day when load subsides,the node counts return back to a fixed predictable count.
  • Reliability:- I want to reduce the possibility of outages when the spot nodes are taken away.Ensuring that atleast 2 pods are running on a dedicated nodes, give me confidence that my application does not face an outages should all the spot nodes get deleted.(It does happen!)

What I have tried so far:-

  • Labels and Node Affinity:- Node affinity ascertains where the pods will be scheduled. But this approach forces me to select either dedicated or spot nodes. (not a combination of both)
  • Pod Anti Affinity:- It works somewhat,it deploy one pod/node till the number of pods = number of dedicated nodes, the starts deployment on spot. But this forces me to limit the size of dedicated nodepool. Also one pod/node is not the use case I am looking for.

Understanding of the problem:-

  • It seems like a common enough use case where atleast some portion of pods are always running but I have not seen a solution to this anywhere.

  • It may go against the k8s philosophy where the app should be unaware/not tightly coupled with the infrastructure below but I am not sure if this is the case here.

  • Running a free for all nodes (no affinity),can cause unbalanced deployments, where scaled pods of one deployment, may prevent any pods of another deployment from getting dedicated node resources.

Let me know if this is achievable and if not , how i can reduce the risk of app outage by running on all spot nodes.

Improve this question

1 Answer 1

Reset to default
1

You can follow below approach:

  1. Use Cluster Autoscaler which should scale-in or scale-out nodes as and when required.
  2. Create nodegroups with multiple instance types so that you always get a spot node.
  3. Use labels and anti-affinity rules to schedule applications on any of these available spot nodes.
Improve this answer
1
  • Hi Samit, I have already done the same as mentioned in my question. This does not answer my use case.
    – rohatgisanat
    Commented Jun 7, 2021 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.