How to update Docker Swarm services all at once?

Question

I have a Docker Swarm with three services: proxy, backend, frontend.

Currently it seems that if I do docker stack deploy and one service fails, it might just rollback to the previous version while the rest will be on the new version.

I want to ensure two things:

1. A request coming to proxy will always be served by the same combination of service versions. Doesn't matter if old or new, but I don't want proxy:v2 to route a request to backend:v1.

2. No downtime during an upgrade. proxy:v1 should continue serving requests until the full set of new services is ready, and only then the failover should happen.

Is this possible?

Answer 1

To ensure a stack is in a consistent state after an upgrade, see the documentation on update_config. failure_action defaults to pause so you need to set it to continue to ensure docker updates the services regardless.

services:
  proxy:
    deploy:
      update_config:
        on_failure: continue
  backend:
    deploy:
      update_config:
        on_failure: continue

That said - docker does not support green/blue deployments - but it is possible to use network aliases to ensure proxy:2 does not route to backend:1. Something like this for e.g.:

services:
  proxy:
    image: proxy:${VER}
    environment:
      BACKEND_HOST: backend-${VER}

  backend:
    image: backend:${VER}
    networks:
      default:
        aliases: ["backend-${VER}"]

To achieve your final requirement, you need to set the update_config parallelism to 1, and ensure that the order is set to start-first.

A naive deployment would have a race condition if "proxy:2" is available before "backend:2" is available, but as long as there is a healthcheck, "proxy:2" can prevent itself receiving traffic until it has a successful connection test to "backend-2"

services:
  proxy:
    image: proxy:${VER}
    environment:
      BACKEND_HOST: backend-${VER}
    healthcheck: ["CMD", "/bin/sh","-c","test_backend.sh"]

Of course, you can run into problems if "proxy:2" finishes deploying, but "backend:2" errors out - you could eventually have a situation where you have no proxy:1 running, only proxy:2, but only backend:1 running and no backend:2. So you want to ensure the update_config delays updating the next task until the current one is health - or pause if that never happens.

PS. It seems potentially much less hassle to just design and code around the principal that backwards compatibility of +-1 version is expected.