I try to run ha redis with kubernetes and I have a problem with deleted pods. When the node which redis master is working on dies, pod that should die keeps working invisibly and also creating new pod so I can't access redis master with a service. I use redis sentinel for failover and when the node which redis master is working on dies, a slave replaces master and the slave becomes the new master. Hidden pods prevent this because the sentinel thinks the master is not dead and adds the newly created redis instance to the cluster as a slave.
My yaml file is like this:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
labels:
app: redis
spec:
serviceName: redis
replicas: 3
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
terminationGracePeriodSeconds: 0
tolerations:
- key: node.kubernetes.io/not-ready
effect: NoExecute
tolerationSeconds: 0
- key: node.kubernetes.io/unreachable
effect: NoExecute
tolerationSeconds: 0
serviceAccountName: redis-sa
containers:
- name: redis
image: exampleregistry.com/redis
imagePullPolicy: Always
args: ["/data/redis.conf"]
ports:
- containerPort: 6379
name: redis-port
readinessProbe:
tcpSocket:
port: 6379
envFrom:
- secretRef:
name: redis
resources:
requests:
memory: 64Mi
limits:
memory: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-sentinel
labels:
app: redis-sentinel
spec:
replicas: 3
selector:
matchLabels:
app: redis-sentinel
template:
metadata:
labels:
app: redis-sentinel
spec:
terminationGracePeriodSeconds: 0
tolerations:
- key: node.kubernetes.io/not-ready
effect: NoExecute
tolerationSeconds: 0
- key: node.kubernetes.io/unreachable
effect: NoExecute
tolerationSeconds: 0
containers:
- name: sentinel
image: exampleregistry.com/redis
imagePullPolicy: IfNotPresent
args: ["/data/sentinel.conf", "--sentinel"]
ports:
- containerPort: 26379
name: redis-port
readinessProbe:
tcpSocket:
port: redis-port
envFrom:
- secretRef:
name: redis
redis-cli ping
). Also what are the symptoms from the application perspective? Redis cluster doesn't have master? Can you please provide output ofk get pods | grep redis
after this happens?