2

I am running GitLab Runner using a Helm chart as described on the GitLab Runner Helm Chart page.

Everything seems to be working, but after about two months of running I see on my GitLab group that controls the GitLab runner on the "CI / CD Settings|Runners" page the message

"Available group Runners: 54".

Here is the first couple on that page:

enter image description here

Rather than listing all 54 here is the list run through uniq -c: 

      3 gitlab-runner-gitlab-runner-9fcb644b9-4f5ss
      2 gitlab-runner-gitlab-runner-9fcb644b9-4gqqq
      1 gitlab-runner-gitlab-runner-9fcb644b9-889rc
      2 gitlab-runner-gitlab-runner-9fcb644b9-grb5j
      1 gitlab-runner-gitlab-runner-9fcb644b9-hhv8f
      1 gitlab-runner-gitlab-runner-9fcb644b9-kzcdv
     13 gitlab-runner-gitlab-runner-9fcb644b9-nvrms
      7 gitlab-runner-gitlab-runner-9fcb644b9-pl2k4
      2 gitlab-runner-gitlab-runner-9fcb644b9-sxlmp
      9 gitlab-runner-gitlab-runner-9fcb644b9-vdmsj
      3 gitlab-runner-gitlab-runner-9fcb644b9-vgdnv
      8 gitlab-runner-gitlab-runner-9fcb644b9-wst8q
      2 gitlab-runner-gitlab-runner-9fcb644b9-xnzhx

Looking at my GitLab Runner Kubernetes container there seems to be only one runner:

$ /usr/bin/gitlab-runner list
Runtime platform                                    arch=amd64 os=linux pid=4953 revision=1b659122 version=12.8.0
Listing configured runners                          ConfigFile=/home/gitlab-runner/.gitlab-runner/config.toml
gitlab-runner-gitlab-runner-9fcb644b9-grb5j         Executor=kubernetes Token=XXXXXXXXXXXX-XXXXX URL=https://myserver.example.com/

Another potentially important piece of information: the Kubernetes node that hosts the GitLab Runner container is preemptible meaning it gets killed and restarted about every 24 hours.

How do I convince GitLab that these old runners are not still around? And how do I avoid this problem in the future?

Improve this question

1 Answer 1

Reset to default
3

Just unregister offline runners:

As per GitLab Runner commands manual:

To delete the old and removed from GitLab runners, execute the following command.

    gitlab-runner verify --delete

You are not alone

P.S. The problem is common: there are many issues with questions like yours.

Moreover, there are several custom recipes to unregister "offline" runners:

API for "zombie" runners

As rlandster noted, sometimes you'll need an API to unregister "zombie" runners.

curl -S --header "PRIVATE-TOKEN:<token>" "https://gitlab.example.com/api/v4/runners/all" | jq '.[] | select(.status == "offline") | .id' | xargs -I runner_id curl -S --request DELETE --header "PRIVATE-TOKEN:<token>" "https://gitlab.example.com/api/v4/runners/runner_id"`

This will list all runners, instead of the personal ones: https://docs.gitlab.com/ee/api/runners.html#list-all-runners

P.S. note about runners

Pls note that Gitlab itself doesn't manage runners. So, e.g. to restart or shutdown the runner, use appropriate commands on the runner's hosts.

Improve this answer
1
  • Running gitlab-runner verify --delete does not unregister the "zombie" jobs; the API interface was necessary to do that.
    – rlandster
    Commented Apr 29, 2020 at 10:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.