1

I try to improve the performance of my pipeline stages in gitlab cicd. The Gitlab is self hosted and the Runner runs as docker with docker executor. I recognized, that for both jobs in my stages (test, build) they begin with pulling the images from docker hub:

Running with gitlab-runner 14.8.0 (565b6c0b)
  on Shared gitlab-runner on ARWS. 3yyScM1M
Preparing the "docker" executor 00:03
Using Docker executor with image python:3.9 ...
Pulling docker image python:3.9 ...
Using docker image sha256:4819be0df94257e1e31cd64dda12d46ff8b2180f9576ad9eaf98dcac9d70d9f9 for python:3.9 with digest python@sha256:57274f3fe86ad3d010025e9019bc41dcf9ee5833cfe4ba7b6773ef49672d855f ...
Preparing environment
.
.
.

This part seems to take most of the pipeline time. At the moment not much is happening in both jobs. I already started to implement caching in the pipelinbe:

image: python:3.9

stages:
  - test
  - build

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  key:
    files:
      - poetry.lock
      - .gitlab-ci.yml
    prefix: ${CI_JOB_NAME}
  paths:
    - .venv
    - .cache/pip

before_script:
  - python --version  # For debugging
  - pip install poetry
  - poetry config virtualenvs.in-project true
  - poetry install
  - source `poetry env info --path`/bin/activate

testing:
  tags: 
    - docker
    - linux
  stage: test
  script:
    - echo "Launching Job 'testing' in stage 'test'"
    - pytest -v --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: /builds/$CI_PROJECT_PATH/report.xml

building:
  tags:
    - docker
    - linux
  stage: build
  script:
    - echo "Launching Job 'building' in stage 'build'"

And there is a huge possibility I do some stupid stuff. But the real question bothers me: Is it possible to cache the environment instead of pulling and preparing it at the start of every job?

1 Answer 1

0

GitLab has some docs on how to speed up builds, and one of the suggestions does involve caching runner images.

https://docs.gitlab.com/runner/configuration/speed_up_job_execution.html#use-a-proxy-for-containers

Use a proxy for containers

You can speed up the time it takes to download Docker images by using:

  • The GitLab Dependency Proxy or
  • A mirror of the DockerHub Registry
  • Other open source solutions

There are other non-container related suggestions too, as well as information on how to use these up at that link.

You can also host your own runners, in which case you can run them on docker or just centos, etc. I assume, in that case, that you could already have the images available on the runner node to avoid new pulls. We run ours in kubernetes and are looking at harbor as an option to speed these things up too, but we haven't tested it yet. I imagine self-hosted k8s and harbor would be similar to GitLab using its own dependency proxy solution.

4
  • My Runner(s) are actually self hosted. They run on a Server as Docker-Runner. If I understand correctly instead of linking a proxy somewhere I could save the images (In my example python:3.9) locally inside the gitlab runner node and reference them?
    – MaKaNu
    Feb 26, 2022 at 22:29
  • If you're using a single-server docker setup, I wouldn't expect it to re-pull images it already had. The underlying docker daemon should recognize it already has an image/relevant layers and should just reuse it without going to docker-hub/etc. That's how docker works. Feb 27, 2022 at 2:28
  • So it seems I am doing something wrong in the pipeline, because I just use one Dockerized-Runner on a single server, but both jobs are pulling the python:3.9 image.
    – MaKaNu
    Feb 27, 2022 at 15:00
  • The docs you link to involve caching in a registry (even if local) as opposed to the local docker cache, which is what I'm looking for, and what I assume the question is asking for as well. Jun 5 at 10:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.