How to Shutdown Docker Stack on Container Failure

Question

Say I have several docker services using swarm defined as such:

version : '3'
services:
    server:
        image : my_server:latest
        deploy: 
            replicas: 5
            restart_policy: any
    database:
        image : my_db:latest
        deploy: 
            replicas: 5
            restart_policy: any
    logs:
        image : my_logger:latest
        deploy: 
            replicas: 1
            restart_policy: any

I then deploy this as a docker-stack as such. The services are server, database and logs. I make great use of the network abstractions provided by docker swarm, but my failure-case is a bit undockerish.

*In this case when all containers in database fail, then I want the entire stack to be shutdown or otherwise made unavailable*. Is there any built-in way to accomplish this task? Does the concept of a stack exist in a way for the individual services to access?

This question can be alternatively answered with a generalization : "How, with built-in docker features, can I be made aware of container failure"? Because I'm assuming there is no way to do exactly what I am asking.

Answer 1

If you don't want the containers to restart, you can set the condition: to none. See https://docs.docker.com/compose/compose-file/#restart_policy

On the swarm manager, you could use docker stack ps to check the status of the containers on a regular basis and shut down the stack if the database containers aren't running (using cron or whatever). For example:

docker stack ps --filter "desired-state=Running" your-stack |grep database || docker stack rm your-stack.

If containers are not able to restart, they shouldn't show up in the list once they fail, and once none are found, you can remove the stack.

Answer 2

Using Docker's healthcheck feature (a side process for an additional script) you could register and sync your container health status with an external service e.g. Consul. The API of this service would then allow further services to take some action, for example as you said redeploy the whole stack or raise an issue.