What would be the best way to improve this Gitlab CI/CD (Gitlab.com) pipeline's efficiency and consistency?

Question

This project has a pipeline that runs integration tests against a spun up instance of Gitlab. Each Job in the integration test stage has its own instance testing different build distributions (es, cjs, original source). Unfortunately, the shared runners have limited resources and inconsistently cause these jobs to fail because the spun up GitLab instances have insufficient resources. This occurs more often when I have more than one instance spin up at a time since the jobs run in parallel.

Here is a test repo showcasing the problem and it's pipeline

Gitlab Support has suggested using a dedicated runner OR refactoring my pipeline. Since I'm trying to maintain a $0 cost for this open-source project, the only feasible suggestion (that I'm aware of) is the latter one.

I've tested doing just this, specifically splitting these integration tests into separate stages to stop creating multiple GitLab instances at the same time. This, however, has not improved the results. The only other idea I can think of is running all the integration tests in the same script, instead of separate stages or jobs, which would only require spinning up one instance of GitLab. In this solution, I would have to add some sort of id's to the data being created in the tests to avoid having clashes (i.e., creating a project must use a unique name)

What I want to figure out is if I'm overlooking something? Are there any better ways to improve the pipeline to avoid the resource bottlenecks causing the crashing?

Answer 1

Can you explain the reason for why the "step_script" installs the dependencies and not use a prebuild images that has all the dependencies installed?

Solution

Based on the information you provided, Your test infrastucture rebuild even the test container from scratch.

You tests take a long time to be done as those lines indicate also a fresh composing of the docker container:

(1/11) Installing ca-certificates (20191127-r2)
(2/11) Installing nghttp2-libs (1.40.0-r1)
(3/11) Installing libcurl (7.67.0-r3)
(4/11) Installing expat (2.2.9-r1)
(5/11) Installing pcre2 (10.34-r1)
(6/11) Installing git (2.24.3-r0)
(7/11) Installing c-ares (1.15.0-r0)
(8/11) Installing libgcc (9.3.0-r0)
(9/11) Installing libstdc++ (9.3.0-r0)
(10/11) Installing nodejs (12.20.1-r0)
(11/11) Installing yarn (1.19.2-r0)

If you build a docker image with those thing preinstalled, your Tests will run much faster.

This Process creates clean environment where the only change to it will be the code that has changed!!

The new Docker image then need to be only pulled ones and can be reused.

Build/test infra

docker run -v "$(pwd)"/target:/app:ro -name BUILDID jdalrymple/docker-compose-with-node yarn test

run that for each of your tests.

Answer 2

My solution to this problem was two fold.

First: As mentioned in the question, I was able to reduce the amount of concurrent Gitlab Instance runners to one, to avoid the usage of too many resource which was causing the shared runner freeze. To do this i moved all my integration tests into a single job. This didn't increase the time of the tests significantly because the largest time cost was the Gitlab instance setup.

Second: I utilized the retry functionality of the docker image, to respin the CI job if there was an error. I set the retry property to 2, that way it would catch erroneous instance issues, but still fail if a consistent error occurred.

This approach seems to be working well so far! Hopefully it holds out.