Gitlab CI: change pipeline definition dynamically

Question

I need to pass info from stage A to stage B so I can change stage B dynamically.

Details:

• I have a stage (A) where I validate a terraform plan and check if it's going to destroy resources.
• I have another stage (B), which applies the terraform plan, which I want to be triggered manually if stage A detected that resources are going to be destroyed.
• When nothing is being destroyed the terraform apply stage (B) should run automatically

How can I pass this information from stage A to B?

Making an artifact in A and using "rules:exist" in stage B is not possible, issue

And if I understand the docs correctly it's not possible to use a "dotenv" var in the "rules:if" section of stage B.

It seems like Gitlab doesn't want you to change the pipeline definition dynamically. Maybe there is a practical solution that I'm not aware of to solve this usecase?

Answer 1

The reason you can't use dotenv variables in the rules clause(s) is that rules are evaluated before the pipeline even starts executing.

If you have a rules clause like

rules:
  - if: $CI_COMMIT_TAG === 'v1.0.0'
  when: never
  - when: always

Gitlab CI will check the following things:

1. Is the pipeline a Tag Pipeline (from the CI_PIPELINE_SOURCE variable)? If no, CI_COMMIT_TAG won't exist, so the clause doesn't exist, and when:always is the only rule. Add the job to the pipeline.
2. If the pipeline is a tag pipeline, is the tag === 'v1.0.0'? Then use when:never. Don't add the job to the pipeline.
3. Tag pipeline and the tag !== 'v1.0.0', use when:never and add the job to the pipeline.

All these things happen before the pipeline is in the created state, so therefore it cannot depend on a variable that is created once the pipeline starts running.

Knowing this, the way I would solve the problem is to mark the job in Stage B as a manual job like it already is, but when you want to run it automatically, call the Gitlab Jobs API from the Stage A job to run the Stage B job.

This will use a couple of API Calls:

1. First, we need to get the jobs for this particular Pipeline:

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_IID/jobs?scope=manual"

This will return all the manual jobs from the pipeline. If you have more than one, you can retrieve the correct one by looking at the value for stage and name from the response. Here's an example from the docs:

[
  {
    "commit": {
      "author_email": "[email protected]",
      "author_name": "Administrator",
      "created_at": "2015-12-24T16:51:14.000+01:00",
      "id": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd",
      "message": "Test the CI integration.",
      "short_id": "0ff3ae19",
      "title": "Test the CI integration."
    },
    "coverage": null,
    "allow_failure": false,
    "created_at": "2015-12-24T15:51:21.727Z",
    "started_at": "2015-12-24T17:54:24.729Z",
    "finished_at": "2015-12-24T17:54:24.921Z",
    "duration": 0.192,
    "queued_duration": 0.023,
    "artifacts_expire_at": "2016-01-23T17:54:24.921Z",
    "tag_list": [
      "docker runner", "ubuntu18"
    ],
    "id": 6,
    "name": "rspec:other",
    "pipeline": {
      "id": 6,
      "project_id": 1,
      "ref": "main",
      "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd",
      "status": "pending"
    },
    "ref": "main",
    "artifacts": [],
    "runner": null,
    "stage": "test",
    "status": "failed",
    "tag": false,
    "web_url": "https://example.com/foo/bar/-/jobs/6",
    "user": {
      "id": 1,
      "name": "Administrator",
      "username": "root",
      "state": "active",
      "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon",
      "web_url": "http://gitlab.dev/root",
      "created_at": "2015-12-21T13:14:24.077Z",
      "bio": null,
      "location": null,
      "public_email": "",
      "skype": "",
      "linkedin": "",
      "twitter": "",
      "website_url": "",
      "organization": ""
    }
  }
]

Once we have the Job ID, we can call the Play Job API:

curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/$ID_OF_STAGE_B_JOB/play"

Note: we don't want to use the Predefined Variable $CI_JOB_ID for the Play Job API since that will be the ID of the Stage A job, not the Stage B job.

Here's the Pipeline Jobs API, and here's the Play Job API for more information and query parameters.