K8s cluster not deploying deployments across all the nodes

Question

I am a beginner in kubernetes. I have a cluster running with 4 nodes. All the nodes are currently running fine. They are not tainted, or drained. Every time I deploy a deployment it will only deploy it across 2 nodes instead of 4 nodes. I am trying to look for kube-scheduler logs, but because I am using Oracle Cloud OKE service (kubernetes service for Oracle cloud). They don't make kube-scheduler logs available to customers. What tests or what verifications do I need to do?

Answer 1

If you want to place a pod on each node you can use DaemonSet. DaemonSets are configured to distribute the application to each node. For detailed information;

https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

You can find out if DaemonSet is what you need by looking specifically at this title;

https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#deployments

Answer 2

The scheduler placing your pods on one or two nodes, until they're out of resources, could be a "normal" behavior depending on your cluster scheduler policy.

You could try to use an antiAffinity rule, or a topologySpreadConstraints, placing your pods on all 4 nodes.

The following sample, from Kubernetes docs, would use topology key 'kubernetes.io/hostname', which should spread your pods on different nodes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
spec:
  selector:
    matchLabels:
      app: web-store
  replicas: 3
  template:
    metadata:
      labels:
        app: web-store
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - web-store
            topologyKey: kubernetes.io/hostname

The requiredDuringSchedulingIgnoredDuringExecution would prevent the Kubernetes scheduler from ever starting a replicas on a node that already has a Pod that belongs to your deployment.

We could do something similar with a topologySpreadConstraints:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
spec:
  selector:
    matchLabels:
      app: web-store
  replicas: 3
  template:
    metadata:
      labels:
        app: web-store
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: web-store