0

This project has a pipeline that runs integration tests against a spun up instance of Gitlab. Each Job in the integration test stage has its own instance testing different build distributions (es, cjs, original source). Unfortunately, the shared runners have limited resources and inconsistently cause these jobs to fail because the spun up GitLab instances have insufficient resources. This occurs more often when I have more than one instance spin up at a time since the jobs run in parallel.

Here is a test repo showcasing the problem and it's pipeline

Gitlab Support has suggested using a dedicated runner OR refactoring my pipeline. Since I'm trying to maintain a $0 cost for this open-source project, the only feasible suggestion (that I'm aware of) is the latter one.

I've tested doing just this, specifically splitting these integration tests into separate stages to stop creating multiple GitLab instances at the same time. This, however, has not improved the results. The only other idea I can think of is running all the integration tests in the same script, instead of separate stages or jobs, which would only require spinning up one instance of GitLab. In this solution, I would have to add some sort of id's to the data being created in the tests to avoid having clashes (i.e., creating a project must use a unique name)

What I want to figure out is if I'm overlooking something? Are there any better ways to improve the pipeline to avoid the resource bottlenecks causing the crashing?

3
  • Wouldn't test:integration:browser: also need to include the &integration template? Jan 21, 2021 at 15:48
  • Also the test project gitlab-bug you created looks like the pipelines work great, so what's the problem here? Jan 21, 2021 at 15:53
  • @Squirrelkiller The browser test currently doesn't test against a running instance of Gitlab. As for the gitlab-bug repo, the results are inconsistent. If you look at the pipelines list, they fail randomly due to the Gitlab instances not having enough resources when they are spun up. gitlab.com/jdalrymple/gitlab-bug/-/pipelines So im looking for ideas to better manage the pipeline resources Jan 21, 2021 at 17:41

2 Answers 2

1

Can you explain the reason for why the "step_script" installs the dependencies and not use a prebuild images that has all the dependencies installed?

Solution

Based on the information you provided, Your test infrastucture rebuild even the test container from scratch.

You tests take a long time to be done as those lines indicate also a fresh composing of the docker container:

(1/11) Installing ca-certificates (20191127-r2)
(2/11) Installing nghttp2-libs (1.40.0-r1)
(3/11) Installing libcurl (7.67.0-r3)
(4/11) Installing expat (2.2.9-r1)
(5/11) Installing pcre2 (10.34-r1)
(6/11) Installing git (2.24.3-r0)
(7/11) Installing c-ares (1.15.0-r0)
(8/11) Installing libgcc (9.3.0-r0)
(9/11) Installing libstdc++ (9.3.0-r0)
(10/11) Installing nodejs (12.20.1-r0)
(11/11) Installing yarn (1.19.2-r0)

If you build a docker image with those thing preinstalled, your Tests will run much faster.

This Process creates clean environment where the only change to it will be the code that has changed!!

The new Docker image then need to be only pulled ones and can be reused.

Build/test infra

docker run -v "$(pwd)"/target:/app:ro -name BUILDID jdalrymple/docker-compose-with-node yarn test

run that for each of your tests.

10
  • Oh nice catch! Ill try that. I'm not sure if it will solve the inconsistent start up of the GitLab container, but who knows. Ill test it out and follow up. Jan 21, 2021 at 18:36
  • check out gitlab.com/jdalrymple/gitlab-bug/-/jobs/970772969 and collaps all sections. on the right side it says how long it took to execute this step. this case 10:15, which translates to 10 mins and 14 secs
    – Max N
    Jan 21, 2021 at 18:43
  • It will fix your slow test problem as about 90% is spend on unnessary setup!!!!!
    – Max N
    Jan 21, 2021 at 18:48
  • But wouldnt that not solve the issue of the resources required by the running gitlab instance? OH! true, how should i modify the DinD image?? Right now, im modifying the image for the stage: docker/compose:latest, with an image that already includes yarn node etc. What should I also do to the DinD image? Jan 21, 2021 at 18:56
  • Ignore the DinD image for now!!!!!
    – Max N
    Jan 21, 2021 at 19:00
0

My solution to this problem was two fold.

First: As mentioned in the question, I was able to reduce the amount of concurrent Gitlab Instance runners to one, to avoid the usage of too many resource which was causing the shared runner freeze. To do this i moved all my integration tests into a single job. This didn't increase the time of the tests significantly because the largest time cost was the Gitlab instance setup.

Second: I utilized the retry functionality of the docker image, to respin the CI job if there was an error. I set the retry property to 2, that way it would catch erroneous instance issues, but still fail if a consistent error occurred.

This approach seems to be working well so far! Hopefully it holds out.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.