0

I need to find all jobs that failed in a specific fashion, a few methods are viable

  • One method that would be viable is finding where GitLab stores its log files, and searching them with something like Grep. I asked that question here.
  • Another method would be using native GitLab search functionality to search the logs. Does such functionality exist?
1

1 Answer 1

0

This functionality doesn't yet exist in the UI, though internally job logs are upload as a job artifact. And you can search them with a little bit of work. The backing store is configured in your, /etc/gitlab/gitlab.rb.

  • If artifacts_object_store_enabled is enabled and you're using the Object Storage, the artifacts will store in whatever the value is of artifacts_object_store_remote_directory under a file called job.log. Assuming that value is artifacts, and you're using something like minio, you can search across jobs for a specific failure with

    • mc find and mc cat,

      # In the example below, I'm looking for failures that contain Client.Timeout
      mc find minio/artifacts \
        --name 'job.log'      \
        --exec 'sh -c "mc cat {} | grep Client\.Timeout | xargs -r echo {time} {}; exit 0"' \
        | tee /tmp/client_timeout.log
      

      In my experience, this is just too slow as mc find isn't indexed it's just looking for any bucket with the name job.log.

    • A faster method is to lean on the database. Gitlab ships a PostgreSQL client wrapper called gitlab-psql which allows you to connect to underlying PostgreSQL datastore. After you connect with gitlab-psql can query for all file.log and get the relative path to the bucket from the result. You can easily tool this by creating a batch file of commands for s5cmd log,

      \t
      \a
      \o /tmp/pull_logs.s5cmd
      SELECT FORMAT(
        'cp s3://artifacts/%s /tmp/job_%s.log',
        to_bucket_path(ci_job_artifacts),
        id
      )
      FROM ci_job_artifacts
      WHERE file = 'job.log'
      AND expire_at IS NULL
      AND file_store = 2;
      

      Which will generate a file you can run with something like s5cmd to pull down all of your logs from the Gitlab Object store. You'd run it by setting up your ~/.aws/credentials and running (example with minio).

      s5cmd --endpoint-url https://minio.acme.net run /tmp/pull_logs.s5cmd
      
  • You can also use NFS. If you're using NFS for artifact storage, it's as easy as going to the mount point and searching with ripgrep.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.