Set job start timeout in Gitlab CI

Question

My pipeline consists of very long jobs that build docker images by compiling a set of packages; all of them require ~ 3 hours on my runner.

I set up my CI so that only one job can run at a time, in order to reduce the disk I/O overhead. But in this way many jobs fail, supposedly because they hit a job start timeout which seems to be bout 1h, with this message:

There has been a timeout failure or the job got stuck. Check your timeout limits or try again

I have not been able to find how to increase the job start timeout, since all the documentation I found is relative to the job duration timeout.

So I’d need help with this, thanks. I'm using GitLab CE 12.8.1 and Gitlab Runner 11.5.1.

Answer 1

There are three types of timeouts in Gitlab CI:

• Project timeout
• Runner timeout
• Jobs timeout

Project timeout

According to GitLab docs:

Timeout defines the maximum amount of time in minutes that a job is able run. This is configurable under your project’s Settings > CI/CD > General pipelines settings. The default value is 60 minutes. Decrease the time limit if you want to impose a hard limit on your jobs’ running time or increase it otherwise. In any case, if the job surpasses the threshold, it is marked as failed

Runner timeout

According to GitLab docs:

For each Runner you can specify a maximum job timeout.

Runner timeout settings are defined in runner's edit page

Jobs timeout

GitLab CI/CD Pipeline Configuration Reference | GitLab

Job's timeout allows you to configure a timeout for a specific job. For example:

build:
  script: build.sh
  timeout: 3 hours 30 minutes

test:
  script: rspec
  timeout: 3h 30m

Precedence of different types of timeout

The job-level timeout can exceed the project-level timeout but can not exceed the Runner-specific timeout.

If runner timeout smaller than project defined timeout, will take the precedence.

Examples of precedence of timeout directive

See Configuring GitLab Runners | GitLab

name: **Example 1 - Runner timeout bigger than project timeout**
  project_timeout: 2h
  runner_timeout: 24h
  job_timeout: 4h
  resulting_timeout: 4h

name: **Example 2 - Runner timeout not configured**
  project_timeout: 2h
  job_timeout: 4h
  resulting_timeout: 4h

name: **Example 3 - Runner AND job timeout are not configured**
  project_timeout: 24h
  resulting_timeout: 24h

name: **Example 4 - Runner timeout smaller than project timeout**
  project_timeout: 2h
  runner_timeout: 30m
  resulting_timeout: 30m

name: **Example 5 - Runner timeout smaller than Project timeout, Job timeout is bigger than Runner timeout**
  project_timeout: 2h
  runner_timeout: 30m
  job_timeout: 1h
  resulting_timeout: 30m