Gitlab CI/CD fails to build docker image

Question

I have built a custom base image from alpine and use it as the base for ruby services that we develop.

FROM ruby:2.4.2-alpine
...

I use Gitlab and Gitlab's docker registry along with it. Each project has its own docker image storage.

I created a separate project for the alpine-ruby-service dockerfile. Then build and pushed the image to Gitlab.

In the ruby service project docker images, I reference it as follows

FROM gitlab.mycompany.dk/alpine-ruby-service-base
...

I have a CI/CD pipeline (.gitlab-ci.yaml) which builds these ruby service images when there is a commit to one of the branches.

The problem is that when the CI/CD pipeline is run it fails with the following message

 $ docker build --pull -t $CONTAINER_TEST_IMAGE .
   Sending build context to Docker daemon  709.6kB

 Step 1/13 : FROM gitlab.mycompany.dk:xxxx/alpine-ruby-service-base
 pull access denied for gitlab.mycompany.dk:xxxx/alpine-ruby-service-base,
 repository does not exist or may require 'docker login': denied: 
 requested access to the resource is denied

I have an instruction to login to the Gitlab docker registry before the build command above using

 docker login --username=gitlab-ci-token --password=$CI_JOB_TOKEN gitlab.mycompany.dk:xxxx

the login succeeds but the following docker build instruction fails!!! can someone point me what is going wrong?

Answer 1

It looks like that the docker login and the docker build were done by different users. If a docker login is performed, then a ~/.docker directory will be created in the namespace of user X, if the build is done by user Y then the latter user does not have a .docker folder and is not authorized to pull the image. One could verify this by echo $USER in the pipeline, in this case gitlab-ci.yaml to check this.

Answer 2

This could also be happening if you perform a docker login before the docker pull in a gitlab runnner as instructed if:

Your shell runner is configured to run multiple builds and all the builds share the same home directory /home/gitlab-runner-username. Per default docker login saves the credentials in ~/.docker/config.json. When there is multiple build jobs running the second build job may overwrite the credentials of the first one. When the second one is triggered by a different user or finishes before the first one, the first one may get the denied: requested access to the resource is denied message even tough you logged in properly.

On our servers i solved this by using a custom credentials script that does not store the credentials in a shared file but uses the credentials present in the gitlab ci job environment