2

Given a working Kubernetes cluster consisting of a master and some workers.

We need to add a node to run a very specific pod and be part of the cluster for networking reasons. Then being largely ignored for later pod creation by the master.

Adding selectors to every deployment to avoid this node is out of question.

Thanks!

Improve this question

2 Answers 2

Reset to default
2

Possibly you are looking for this feature which is closed with no action since there are ways to work around this problem.

kubeadm already allows to set taints for the joining node using the configuration file at join time (check my other answer for details on how to do that)

One Other way to achieve this is by running join as usual but taking few extra steps as below after join

1) After successful join , Force Drain the new node to make sure it removes unwanted pod which might have scaled to it post join operation kubectl drain <node name> --force

2) Drain command will auto cordon off the node. (Note : also consider special handling to remove any daemonset which might have scaled to this node after join)

3) Now Taint the new node as needed and then uncordon it

4) Add nodeselector plus tolerance to pods you want to be scheduled on this new node.

Logs of above steps for reference are listed below.

On below example i have joined a new node-02 and assume that some pod were scaled in an existing deployment so new node got used immediately on join, so before adding we need drain action and then taint the node so going forward no scheduling can be done unless pod has tolerance to the taint added

Nodes Before Drain or Taint

$ kubectl get all -o wide
NAME                         READY   STATUS    RESTARTS   AGE   IP               NODE                NOMINATED NODE   READINESS GATES
pod/nginx-86c57db685-8cdkx   1/1     Running   0          27s   192.168.58.120   k8s-node02-calico   <none>           <none>
pod/nginx-86c57db685-xnh6q   1/1     Running   0          65s   192.168.182.78   k8s-node01-calico   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d    <none>

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES   SELECTOR
deployment.apps/nginx   2/2     2            2           65s   nginx        nginx    app=nginx

NAME                               DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES   SELECTOR
replicaset.apps/nginx-86c57db685   2         2         2       65s   nginx        nginx    app=nginx,pod-template-hash=86c57db685

Drain The node

$ kubectl drain k8s-node02-calico
node/k8s-node02-calico cordoned
evicting pod "nginx-86c57db685-8cdkx"
pod/nginx-86c57db685-8cdkx evicted
node/k8s-node02-calico evicted


$ kubectl get nodes
NAME                STATUS                     ROLES    AGE   VERSION
k8s-master-calico   Ready                      master   2d    v1.17.2
k8s-node01-calico   Ready                      <none>   2d    v1.17.2
k8s-node02-calico   Ready,SchedulingDisabled   <none>   2d    v1.17.2

Taint the new node

$ kubectl taint node k8s-node02-calico newnode=nobodyallowed:NoSchedule
node/k8s-node02-calico tainted

Now we are good to uncordon the node, no more pods should be able to use this noe unless they add toleration to the new taint.

$ kubectl uncordon k8s-node02-calico
node/k8s-node02-calico uncordoned


$ kubectl get nodes
NAME                STATUS   ROLES    AGE   VERSION
k8s-master-calico   Ready    master   2d    v1.17.2
k8s-node01-calico   Ready    <none>   2d    v1.17.2
k8s-node02-calico   Ready    <none>   2d    v1.17.2

We test by scaling the deployment and nothign should be added on node02

$ kubectl scale deployment --replicas=3 nginx
deployment.apps/nginx scaled

$ kubectl get all -o wide
NAME                         READY   STATUS    RESTARTS   AGE     IP               NODE                NOMINATED NODE   READINESS GATES
pod/nginx-86c57db685-vm8ql   1/1     Running   0          3m30s   192.168.182.75   k8s-node01-calico   <none>           <none>
pod/nginx-86c57db685-xnh6q   1/1     Running   0          10m     192.168.182.78   k8s-node01-calico   <none>           <none>
pod/nginx-86c57db685-zh2sj   1/1     Running   0          8m19s   192.168.182.76   k8s-node01-calico   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d    <none>

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES   SELECTOR
deployment.apps/nginx   3/3     3            3           10m   nginx        nginx    app=nginx

NAME                               DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES   SELECTOR
replicaset.apps/nginx-86c57db685   3         3         3       10m   nginx        nginx    app=nginx,pod-template-hash=86c57db685
Improve this answer
2
  • I am speechless... this is the perfect answer, @DT. Thank you very much!
    – RooSoft
    Commented Feb 5, 2020 at 15:18
  • in my view my second answer below is better approach because it will taint the node directly at join time so that not even a deamonset can come into your new space.
    – DT.
    Commented Feb 5, 2020 at 15:22
1

kubeadm already allows to set taints for the joining node using the configuration file at join time.

Steps on how to do this at node join time are as below

Construct a Config.yaml file with taint details and bootstrap details as recived from master at kubeadm init time.

Example config.yaml you should update values as per your cluster info on below

apiVersion: kubeadm.k8s.io/v1beta1
kind: JoinConfiguration
discovery:
  bootstrapToken:
    apiServerEndpoint: "<control_plain_ip>:6443"
    token: "your_token"
    caCertHashes:
    - sha256:$CERT_HASH
nodeRegistration:
  taints:
  - effect: NoSchedule
    key: specialnode1.kubernetes.io/SpecialNode1

Pass this config from node you want to join it will join with a taint set at startup.

root@k8s-node01:# kubeadm join --config=config.yaml

[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Node should join the cluster and will have a taint at startup time so pods not having tolerance will not be scheduled on this new node.

ubuntu@k8s-master:~$ kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
k8s-master   Ready    master   18m     v1.17.2
k8s-node01   Ready    <none>   8m11s   v1.17.2

Verify the taint as below

$ kubectl describe node k8s-node01 | grep -i Taint

Taints:             specialnode1.kubernetes.io/SpecialNode1:NoSchedule

To further test this we will create a daemon set which will run on node01 and add another node02 with new taint , daemon set should not autoscale to new node with new taint.

ubuntu@k8s-master:~$ kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   24m   v1.17.2
k8s-node01   Ready    <none>   14m   v1.17.2


ubuntu@k8s-master:~$ kubectl get all -o wide
NAME              READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
pod/nginx-npmz5   1/1     Running   0          50s   192.168.85.193   k8s-node01   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   24m   <none>

NAME                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE   CONTAINERS   IMAGES   SELECTOR
daemonset.apps/nginx   1         1         1       1            1           <none>          50s   nginx        nginx    app=nginx

Add new Node with new config.yaml and verify the node is ready and taint is set on new node added to cluster

apiVersion: kubeadm.k8s.io/v1beta1
    kind: JoinConfiguration
    discovery:
      bootstrapToken:
        apiServerEndpoint: "<control_plain_ip>:6443"
        token: "your_token"
        caCertHashes:
        - sha256:$CERT_HASH
    nodeRegistration:
      taints:
      - effect: NoSchedule
        key: specialnode2.kubernetes.io/SpecialNode2

$ kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   25m   v1.17.2
k8s-node01   Ready    <none>   15m   v1.17.2
k8s-node02   Ready    <none>   22s   v1.17.2

Note the new taint on node02

$ kubectl describe node k8s-node02 | grep -i Taint
Taints:             specialnode2.kubernetes.io/SpecialNode2:NoSchedule

Now check if daemon set has scaled to new node or not, it will not unless we remove the taint or add tolerance to daemonset

$ kubectl get all -o wide
NAME              READY   STATUS    RESTARTS   AGE     IP               NODE         NOMINATED NODE   READINESS GATES
pod/nginx-npmz5   1/1     Running   0          3m46s   192.168.85.193   k8s-node01   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   27m   <none>

NAME                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE     CONTAINERS   IMAGES   SELECTOR
daemonset.apps/nginx   1         1         1       1            1           <none>          3m46s   nginx        nginx    app=nginx

So we can use above steps to add a new node to a running cluster and choose what is scheduled on it using taints.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.