Docker container stuck in "new" state - Docker swarm - service stack

Question

I am deploying a service stack to a Docker swarm like so:

$ dt stack deploy --with-registry-auth -c tmp.stack.docker-compose.yml tmp
Creating network tmp_my-network
Creating service tmp_my-service

$ dt stack ps tmp
ID                  NAME                IMAGE               NODE    DESIRED STATE       CURRENT STATE       ERROR               PORTS
yyjue6fdqu5u        tmp_my-service.1    nginx:latest                            Running             New 7 minutes ago

As you can see it's just an nginx, but the state of the service never leaves "new".

I can not stack deploy any other service either. No matter what I write into my compose file, they are all stuck in state "NEW".

I am not sure how to produce the following output from commandline, but in a running portainer I was able to get the following information:

Services > tmp_my-service > qlvqjofehvqys8kje3b32y50v > Logs
"Error grabbing logs: rpc error: code = Unknown desc = warning: incomplete log stream. some logs could not be retrieved for the following reasons: task qlvqjofehvqys8kje3b32y50v has not been scheduled"

This makes sense, as it looks like the service's tasks are never being scheduled, nothing is actually ever started. So I am assuming something (at orchestrator level) stops Docker swarm from carrying out the request to get any service started.

Where might I find more information about what could be wrong?

I have tried:

$ docker events
network create bcqcoakf5worjmfpelb199xrp (name=my-network)
node update rddzqxvlu2v2ylbcbywpa8kze (name=node-name)
service create x549akxlbonn9zvtxgh8oq2mm (name=my-service)

https://success.docker.com/article/why-do-my-services-stay-pending-when-trying-to-schedule-them-with-placement-contraints

https://success.docker.com/article/swarm-troubleshooting-methodology

Details of the task:

$ dt stack services tmp
dID                  NAME                MODE                REPLICAS            IMAGE               PORTS
x549akxlbonn        my-service      replicated          0/1                 nginx:latest

$ dt service ps my-service
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE       ERROR               PORTS
m4d929n5su7r        my-service.1    nginx:latest                            Running             New 4 minutes ago

$ dt inspect --type task m4d929n5su7r
[
    {
        "ID": "m4d929n5su7re5igit3w2tnvg",
        "Version": {
            "Index": 2397643
        },
        "CreatedAt": "2019-10-10T15:58:33.416210977Z",
        "UpdatedAt": "2019-10-10T15:58:33.416210977Z",
        "Labels": {},
        "Spec": {
            "ContainerSpec": {
                "Image": "nginx:latest@sha256:aeded0f2a861747f43a01cf1018cf9efe2bdd02afd57d2b11fcc7fcadc16ccd1",
                "Labels": {
                    "com.docker.stack.namespace": "tmp"
                },
                "Command": [
                    "tail",
                    "-f",
                    "/dev/null"
                ],
                "Privileges": {
                    "CredentialSpec": null,
                    "SELinuxContext": null
                },
                "Isolation": "default"
            },
            "Resources": {},
            "RestartPolicy": {
                "Condition": "on-failure",
                "Delay": 5000000000,
                "MaxAttempts": 3,
                "Window": 120000000000
            },
            "Placement": {
                "Constraints": [
                    "node.role == manager"
                ],
                "Platforms": [
                    {
                        "Architecture": "amd64",
                        "OS": "linux"
                    },
                    {
                        "OS": "linux"
                    },
                    {
                        "Architecture": "arm64",
                        "OS": "linux"
                    },
                    {
                        "Architecture": "386",
                        "OS": "linux"
                    },
                    {
                        "Architecture": "ppc64le",
                        "OS": "linux"
                    },
                    {
                        "Architecture": "s390x",
                        "OS": "linux"
                    }
                ]
            },
            "Networks": [
                {
                    "Target": "bcqcoakf5worjmfpelb199xrp",
                    "Aliases": [
                        "my-service"
                    ]
                }
            ],
            "ForceUpdate": 0
        },
        "ServiceID": "x549akxlbonn9zvtxgh8oq2mm",
        "Slot": 1,
        "Status": {
            "Timestamp": "2019-10-10T15:58:33.41619488Z",
            "State": "new",
            "Message": "created",
            "PortStatus": {}
        },
        "DesiredState": "running"
    }
]

It all gives me the same info: Desired is "running", actual state is "new" but it won't give me more info on why.

edit: Under "placement > Constraints" it says "node.role == manager" in the ouput. This is just temporary, as I have been trying to get it to run on individual nodes by constraining it. It does not work with this constraint, nor any other constraint, nor with no constraint at all.

Answer 1

After some more time I found my answer.

The service I wanted to deploy was a slightly different version of a service we already have. Stack file for the "original" service:

version: '3.7'

services:
my-service:
    image: my-repo:port/company/my-service
    ports:
    - 81:81
    networks:
    - my-network
    deploy:
    replicas: 1
    restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
networks:
my-network:
    driver: overlay
    internal: false
    attachable: true
    ipam:
    driver: default
    config:
        - subnet: 10.1.99.0/24

We took the same file but on a different git branch to deploy the slightly altered version (as makes sense from a "code" perspective).

From a Docker perspective this meant however that the stack with the original service with the exact same network definition was still running and we deployed another stack with the exact same network definition again, which lead to the service never being scheduled as, apparently, you must not re-declare the same network twice.

Of course it will get a different network name, but the overlay/subnet config will be identical.

It would be nice of Docker to inform you of this when you're trying to do that, but, as it so often is: Crap in, crap out. Our fault!

For future readers who do not have the exact same problem but their containers are getting stuck in "new" state: It might be safe to assume that it has something to do with your stack-file, your network definition or even something like correct YAML indentation. It is most certainly something "meta" and not something related to your Docker host and specifically not your image or your container (as when the task is stuck in "new" state, there is not even a container in existence as it never got scheduled).