I try to improve the performance of my pipeline stages in gitlab cicd. The Gitlab is self hosted and the Runner runs as docker with docker executor. I recognized, that for both jobs in my stages (test, build) they begin with pulling the images from docker hub:
Running with gitlab-runner 14.8.0 (565b6c0b)
on Shared gitlab-runner on ARWS. 3yyScM1M
Preparing the "docker" executor 00:03
Using Docker executor with image python:3.9 ...
Pulling docker image python:3.9 ...
Using docker image sha256:4819be0df94257e1e31cd64dda12d46ff8b2180f9576ad9eaf98dcac9d70d9f9 for python:3.9 with digest python@sha256:57274f3fe86ad3d010025e9019bc41dcf9ee5833cfe4ba7b6773ef49672d855f ...
Preparing environment
.
.
.
This part seems to take most of the pipeline time. At the moment not much is happening in both jobs. I already started to implement caching in the pipelinbe:
image: python:3.9
stages:
- test
- build
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
cache:
key:
files:
- poetry.lock
- .gitlab-ci.yml
prefix: ${CI_JOB_NAME}
paths:
- .venv
- .cache/pip
before_script:
- python --version # For debugging
- pip install poetry
- poetry config virtualenvs.in-project true
- poetry install
- source `poetry env info --path`/bin/activate
testing:
tags:
- docker
- linux
stage: test
script:
- echo "Launching Job 'testing' in stage 'test'"
- pytest -v --junitxml=report.xml
artifacts:
when: always
reports:
junit: /builds/$CI_PROJECT_PATH/report.xml
building:
tags:
- docker
- linux
stage: build
script:
- echo "Launching Job 'building' in stage 'build'"
And there is a huge possibility I do some stupid stuff. But the real question bothers me: Is it possible to cache the environment instead of pulling and preparing it at the start of every job?