I have a situation where I have a Postgres and a Node server which connects to it. I am using Kubernetes to deploy them.
If my Postgres container fails and stops working then its livenessProbe will restart him which will take some time, meanwhile, the server container is useless so the Server's readinessProbe should fail and Ingress stops routing traffic to it. If this happens what should be the behavior of the Server?
Some solutions I thought but was not sure about them:
- Server's livenessProbe sees database down and restarts the container (doesn't make sense to me because it could generate a cascade effect, Postgres failure makes the Server's container restart)
- Server keep trying to reconnect forever and becomes a useless running container while Postgres doesn't come back
- Server tries to reconnect a limited number of times with exponential interval between them, after that it restarts