GitLab Runner is using its Docker host's hostname as Git server FQDN/hostname

Question

When using a GitLab Runner with a "docker" executor, the following build error occurs:

Running with gitlab-runner 10.7.0 (7c273476)
  on main 80500676
Using Docker executor with image centos:latest ...
Pulling docker image centos:latest ...
Using docker image sha256:e934aafc22064b7322c0250f1e32e5ce93b2d19b356f4537f5864bd102e8531f for centos:latest ...
Running on runner-80500676-project-1-concurrent-0 via hostname-of-gitlab-runner-and-gitlab-server...
Cloning repository...
Cloning into '/builds/some-group-name/some-repository-name'...
fatal: unable to access 'http://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@hostname-of-gitlab-runner-and-gitlab-server/some-repository-name/some-repository-name.git/':
Could not resolve host: hostname-of-gitlab-runner-and-gitlab-server
ERROR: Job failed: exit code 1

It looks like the GitLab Runner is using the hostname of its Docker host as the FQDN or hostname of the Git server to pull the code from. This doesn't make sense, as it's not in any of the configuration files:

This is the GitLab configuration:

$ sudo cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0

[[runners]]
  name = "main"
  url = "http://127.0.0.1:1337/"
  token = "80500676b5518df3f3f3b1f772e20b"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "ubuntu:latest"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    #extra_hosts = ["hostname-of-gitlab-runner-and-gitlab-server:actual-fqdn-of-gitlab-server-where-git-repo-is"]
  [runners.cache]

This is the relevant configuration for external_url in the GitLab server configuration:

$ sudo egrep '\bexternal_url\b' /etc/gitlab/gitlab.rb
##! For more details on configuring external_url see:
external_url = 'https://server:actual-fqdn-of-gitlab-server-where-git-repo-is'
# gitlab_pages['artifacts_server_url'] = nil # Defaults to external_url + '/api/v4'

This is the relevant configuration for host: in the GitLab server's generated configuration:

$ sudo egrep -C 2 -n '^    host: ' /var/opt/gitlab/gitlab-rails/etc/gitlab.yml | head -n 5
11-  gitlab:
12-    ## Web server settings (note: host is the FQDN, do not include http://)
13:    host: hostname-of-gitlab-runner-and-gitlab-server
14-    port: 80
15-    https: false

When I change hostname-of-gitlab-runner-and-gitlab-server to actual-fqdn-of-gitlab-server-where-git-repo-is, and run sudo gitlab-ctl reconfigure, it gets reset.

If I edit gitlab.host in /var/opt/gitlab/gitlab-rails/etc/gitlab.yml directory, and run sudo gitlab-ctl restart without running sudo gitlab-ctl reconfigure, it will stay intact.

Some unanswered questions are:

1. How can I permanently set the gitlab.host configuration value that gets generated in /var/opt/gitlab/gitlab-rails/etc/gitlab.yml when sudo gitlab-ctl reconfigure is run?

2. Why does gitlab.host in gitlab.yml keep getting set to the hostname of the GitLab server?

Answer 1

Neil, it's because the = sign in your /etc/gitlab/gitlab.rb file.

When you'll look on the full comment before the external_url line, you'll see a link to the documentation:

## GitLab URL
##! URL on which GitLab will be reachable.
##! For more details on configuring external_url see:
##! https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab
external_url 'http://gitlab.example.com'

In the default gitlab.rb file (just after installing the Omnibus GitLab package) and also in the linked documentation you can see, that the external_url value is assigned without the = sign. It's done like that because the external_url is a method, not a variable. So here you're not assigning the value, but calling the method with this value. Ruby allows to call a method without brackets around the arguments, and this is a commonly used pattern when designing a DSLs based on Ruby.

When you try to set the value with:

external_url = 'https://server:actual-fqdn-of-gitlab-server-where-git-repo-is'

then the external_url method is not receiving any argument and in that case Omnibus GitLab falls back to a default value which is based on host's hostname.

In most cases the hostname of the server where GitLab is installed will be the same as the hostname that the server is accessible with. In some cases - including yours - the external_url needs to be set explicitly.

So the solution seems to be:

1. Change external_url = 'URL_HERE' to external_url 'URL_HERE'
2. Execute sudo gitlab-ctl reconfigure

This should set the proper values in gitlab.yml file and resolve your problems.