How do I configure a Readiness Probe for Selected Services?

Question

I have 2 pods and my application is based on a cluster i.e. application synchronizes with another pod to bring it up. Let us say in my example I am using appod1 and appod2 and the synchronization port is 8080.

I want the service for DNS to be resolved for these pod hostnames but I want to block the traffic from outside the apppod1 and appod2.

I can use a readiness probe but then the service doesn't have endpoints and I can't resolve the IP of the 2nd pod. If I can't resolve the IP of the 2nd pod from pod1 then I can't complete the configuration of these pods.

E.g.

app1_sts.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    cluster: appcluster
  name: app1
  namespace: app
spec:
  selector:
    matchLabels:
      cluster: appcluster
  serviceName: app1cluster
  template:
    metadata:
      labels:
        cluster: appcluster
    spec:
     containers:
       - name: app1-0
         image: localhost/linux:8
         imagePullPolicy: Always
         securityContext:
          privileged: false
         command: [/usr/sbin/init]
         ports:
         - containerPort: 8080
           name: appport
         readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 120
            periodSeconds: 30
            failureThreshold: 20
         env:
         - name: container
           value: "true"
         - name: applist
           value: "app2-0"

app2_sts.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    cluster: appcluster
  name: app2
  namespace: app
spec:
  selector:
    matchLabels:
      cluster: appcluster
  serviceName: app2cluster
  template:
    metadata:
      labels:
        cluster: appcluster
    spec:
     containers:
       - name: app2-0
         image: localhost/linux:8
         imagePullPolicy: Always
         securityContext:
          privileged: false
         command: [/usr/sbin/init]
         ports:
         - containerPort: 8080
           name: appport
         readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 120
            periodSeconds: 30
            failureThreshold: 20
         env:
         - name: container
           value: "true"
         - name: applist
           value: "app1-0"

Check the Statefulset

[root@oper01 onprem]# kubectl get all -n app
NAME             READY   STATUS    RESTARTS        AGE
pod/app1-0       0/1     Running   0               8s
pod/app2-0       0/1     Running   0               22s


NAME                        READY   AGE
statefulset.apps/app1       0/1     49s
statefulset.apps/app2       0/1     22s



kubectl exec -i -t app1-0 /bin/bash -n app


[root@app1-0 ~]# nslookup app2-0
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find app2-0: NXDOMAIN

[root@app1-0 ~]# nslookup app1-0
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find app1-0: NXDOMAIN

[root@app1-0 ~]#

I understand the behavior of the readiness probe and I am using it as it helps me to make sure service should not resolve to app pods if port 8080 is down. However, I am unable to make out how can I complete the configuration as app pods need to resolve each other and they need their hostname and IPs to configure. DNS resolution can only happen once the service has end points. Is there a better way to handle this situation?

Answer 1

You want to use two services.

One of your services should set some spec.publishNotReadyAddresses: true. That one you would use for internal communications, initializing your cluster while Pods could still be not Ready.

And the other service exposing your cluster to its clients, observing the defaults readiness.

Eg, for redis, I would use something like:

apiVersion: v1
kind: Service
metadata:
  name: redis-kube
spec:
  clusterIP: None
  ports:
  - name: tcp-6379
    port: 6379
    protocol: TCP
    targetPort: 6379
  - name: tcp-26379
    port: 26379
    protocol: TCP
    targetPort: 26379
  publishNotReadyAddresses: true
  selector:
    name: redis-kube
  sessionAffinity: None
  type: ClusterIP

Also: when trying to resolve name for the other containers in your cluster, instead of nslookup app-0, try nslookup <statefulset-pod-name>.<statefulset-spec.serviceName>.<statefulset.namespace>

to resolve app1-0, you would use nslookup app1-0.app1cluster.app or nslookup app1-0.app1cluster.app.svc.cluster.local