Docker stack deploy doesn't pull images

Question

I'm launching through Ansible (just SSH basically) a docker stack deploy, the issue is also true when launching directly from a manager:

docker stack deploy --prune --with-registry-auth -c /docker/docker-compose.yml "test-stack"

I have my credentials to access this registry are defined in .docker/config.json.

The problem is first seen here with a docker stack ps test-stack:

 ... Shutdown            Rejected 11 minutes ago      "No such image: myregistry.com:12345/myimage:mytag"

The workers aren't uploading the images I want. On their logs, I get the following:

Jan 31 11:22:24 myhost dockerd[1149]: time="2019-01-31T11:22:24.574302458+01:00" level=warning msg="Error getting v2 registry: Get https://myregistry.com:12345/v2/: http: server gave HTTP response to HTTPS client"
Jan 31 11:22:24 myhost dockerd[1149]: time="2019-01-31T11:22:24.574342640+01:00" level=info msg="Attempting next endpoint for pull after error: Get https://myregistry.com:12345/v2/: http: server gave HTTP response to HTTPS client"
Jan 31 11:22:24 myhost dockerd[1149]: time="2019-01-31T11:22:24.618063948+01:00" level=info msg="Attempting next endpoint for pull after error: unauthorized: authentication required"
Jan 31 11:22:24 myhost dockerd[1149]: time="2019-01-31T11:22:24.618121224+01:00" level=error msg="pulling image failed" error="unauthorized: authentication required" module=node/agent/taskmanager node.id=rarbt9afb16y7j77bfwiacby2 service.id=ej4y7wbwp539u7mmmhfashz93 task.id=c1uy2gnwdcmibp6l2p9rfxcsh
Jan 31 11:22:24 myhost dockerd[1149]: time="2019-01-31T11:22:24.618840324+01:00" level=error msg="fatal task error" error="No such image: myregistry.com:12345/myimage:mytag" module=node/agent/taskmanager node.id=rarbt9afb16y7j77bfwiacby2 service.id=ej4y7wbwp539u7mmmhfashz93 task.id=c1uy2gnwdcmibp6l2p9rfxcsh
Jan 31 11:22:29 myhost dockerd[1149]: time="2019-01-31T11:22:29.548642584+01:00" level=warning msg="failed to deactivate service binding for container test-stack_myservice.1.c1uy2gnwdcmibp6l2p9rfxcsh" error="No such container: test-stack_myservice.1.c1uy2gnwdcmibp6l2p9rfxcsh" module=node/agent node.id=rarbt9afb16y7j77bfwiacby2

The problem is that, when I do a docker pull myregistry.com:12345/myimage:mytag on the workers, it pulls the image successfully and without any problem, but not through an docker stack deploy command. Despite the fact that i have the option --with-registry-auth.

Answer 1

So it turned out that some of the workers didn't have the correct credentials to the docker registry. After I applied the correct ones, the command worked normally.

That's weird, since I could do it locally... Anyway, case solved.

Answer 2

@jayooin that's weird, if you use --with-registry-auth, you don't normally have to set up the credentials on each worker node as the credentials are passed to the wokers by the manager.

In my case the issue was that I was using a self-signed certificate for accessing the private docker registry. I had configured it correctly on the master node, but not on the workers, so connecting to the registry raised a "x509: certificate signed by unknown authority" error on the workers. Fixing the certificate error on the workers solved the issue, without setting up the registry credentials on them.