Should my zombie backend container be restarted when database is down?

Question

I have a situation where I have a Postgres and a Node server which connects to it. I am using Kubernetes to deploy them.

If my Postgres container fails and stops working then its livenessProbe will restart him which will take some time, meanwhile, the server container is useless so the Server's readinessProbe should fail and Ingress stops routing traffic to it. If this happens what should be the behavior of the Server?

Some solutions I thought but was not sure about them:

1. Server's livenessProbe sees database down and restarts the container (doesn't make sense to me because it could generate a cascade effect, Postgres failure makes the Server's container restart)
2. Server keep trying to reconnect forever and becomes a useless running container while Postgres doesn't come back
3. Server tries to reconnect a limited number of times with exponential interval between them, after that it restarts

Answer 1

The standard option is #2, keep retrying the connection, possibly with slower intervals, but I'd have an upper limit to the longest interval. The liveness probe should only fail if that specific container is broken, not any of it's dependencies. If there are cases where your application is failing and won't self recover when the DB is restored, then the liveness probe should fail in that scenario, and I'd consider an init container to check for the availability of the DB before starting the full application.

There should be some logging and metrics enabled to make the DB failure visible so that you debug the correct thing. If there is a concern of too many retries impacting the network, then a service mesh with circuit breaker functionality may be a good option.