1

I try to run ha redis with kubernetes and I have a problem with deleted pods. When the node which redis master is working on dies, pod that should die keeps working invisibly and also creating new pod so I can't access redis master with a service. I use redis sentinel for failover and when the node which redis master is working on dies, a slave replaces master and the slave becomes the new master. Hidden pods prevent this because the sentinel thinks the master is not dead and adds the newly created redis instance to the cluster as a slave.

My yaml file is like this:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  labels:
    app: redis
spec:
  serviceName: redis
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      terminationGracePeriodSeconds: 0
      tolerations:
      - key: node.kubernetes.io/not-ready
        effect: NoExecute
        tolerationSeconds: 0
      - key: node.kubernetes.io/unreachable
        effect: NoExecute
        tolerationSeconds: 0
      serviceAccountName: redis-sa
      containers:
      - name: redis
        image: exampleregistry.com/redis
        imagePullPolicy: Always
        args: ["/data/redis.conf"]
        ports:
        - containerPort: 6379
          name: redis-port
        readinessProbe:
          tcpSocket:
            port: 6379
        envFrom:
        - secretRef:
            name: redis
        resources:
          requests:
            memory: 64Mi
          limits:
            memory: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-sentinel
  labels:
    app: redis-sentinel
spec:
  replicas: 3
  selector:
    matchLabels:
      app: redis-sentinel
  template:
    metadata:
      labels:
        app: redis-sentinel
    spec:
      terminationGracePeriodSeconds: 0
      tolerations:
      - key: node.kubernetes.io/not-ready
        effect: NoExecute
        tolerationSeconds: 0
      - key: node.kubernetes.io/unreachable
        effect: NoExecute
        tolerationSeconds: 0
      containers:
      - name: sentinel
        image: exampleregistry.com/redis
        imagePullPolicy: IfNotPresent
        args: ["/data/sentinel.conf", "--sentinel"]
        ports:
        - containerPort: 26379
          name: redis-port
        readinessProbe:
          tcpSocket:
            port: redis-port
        envFrom:
        - secretRef:
            name: redis
1
  • Can you please clarify what you mean by node? If you mean a worker node, what do you mean by die? I can't think of how a pod could keep running on a worker node that is decommissioned. The master pod could have just been rescheduled to a new worker node. I'd recommend adding a liveness probe as well, to make sure that each redis pod is in good shape (e.g. redis-cli ping). Also what are the symptoms from the application perspective? Redis cluster doesn't have master? Can you please provide output of k get pods | grep redis after this happens? Mar 29, 2022 at 12:39

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.