Docker Swarm stop working with redeploy service, require system reboot

Question

I got a Docker Swarm with 3 nodes, and contain around 40-50 stack on it, now its random stop working to redeploy my new changes its just say in new mode, and really don't know what going one here.

ID             NAME                            IMAGE                                                               NODE                                DESIRED STATE   CURRENT STATE                ERROR
                        PORTS
1wyc57lkgebc   translation-manager_web.1       {images}                                       Running         New 6 minutes ago

6xthicsqmkst    \_ translation-manager_web.1   {images}   docker-swarm-02   Shutdown        Shutdown 6 minutes ago

Normal its taking like 20-30sec to redeploy new changes to my Docker Swarm setup.

If I'm running the service update command its just stay like this and never ending

translation-manager_web
overall progress: 0 out of 1 tasks
1/1: new       [=====>                                             ]

Its look like its random witch service / container there hit this error.

I got 50% free disk space, lots of memory and free CPU that's the first I think its wrong.

Swarm node info:

ID                            HOSTNAME                                    STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
6n19d0ryoon2zy4s295s928pp     staging-docker-swarm-01           Ready     Active                          20.10.8
8se5pzi4znzde6qtlmu0yt0og     staging-docker-swarm-02           Ready     Active                          20.10.8
p7srx7ojo50rmdxtk6ppriomd     staging-docker-swarm-03           Ready     Active                          20.10.8
yf8ubbk650rrkmrwkyk3no8zb     staging-docker-swarm-04           Ready     Active                          20.10.8
f3acixkd61015h0kqgp61ilav *   staging-docker-swarm-manager-01   Ready     Active         Leader           20.10.8

-- Update: 18/08/2021

I have added one more worker to my swarm, and 3 new manager to it, and its its still the same issue.

I have 62 Stacks with 109 services running in my swarm right now if the numbers helping.

Answer 1

I found out of what happening here. I hit the limit of the IP amount in the ingress network. Default ingress network for Docker Swarm when you create it is 254 IP's its landing in the IP range /24

So what I do its change this IP range to /16 and now get 65k IP's

Every published port you are using multiply with replicates take a single IP so in my case when I got 109 services with avg. replicates its taking 218 IP's out of the default IP range.

So after this change everything are working again in my Docker Swarm setup.

Here is the command to change your ingress network from /24 to /16

docker network rm ingress
docker network create \
  --driver overlay \
  --ingress \
  --subnet=10.11.0.0/16 \
  --gateway=10.11.0.2 \
  --opt com.docker.network.driver.mtu=1200 \
  ingress