Does GitLab support searching logs across jobs for specific strings?

Question

I need to find all jobs that failed in a specific fashion, a few methods are viable

• One method that would be viable is finding where GitLab stores its log files, and searching them with something like Grep. I asked that question here.
• Another method would be using native GitLab search functionality to search the logs. Does such functionality exist?

Answer 1

This functionality doesn't yet exist in the UI, though internally job logs are upload as a job artifact. And you can search them with a little bit of work. The backing store is configured in your, /etc/gitlab/gitlab.rb.

• If artifacts_object_store_enabled is enabled and you're using the Object Storage, the artifacts will store in whatever the value is of artifacts_object_store_remote_directory under a file called job.log. Assuming that value is artifacts, and you're using something like minio, you can search across jobs for a specific failure with

  • mc find and mc cat,

    # In the example below, I'm looking for failures that contain Client.Timeout
    mc find minio/artifacts \
      --name 'job.log'      \
      --exec 'sh -c "mc cat {} | grep Client\.Timeout | xargs -r echo {time} {}; exit 0"' \
      | tee /tmp/client_timeout.log
    

    In my experience, this is just too slow as mc find isn't indexed it's just looking for any bucket with the name job.log.

  • A faster method is to lean on the database. Gitlab ships a PostgreSQL client wrapper called gitlab-psql which allows you to connect to underlying PostgreSQL datastore. After you connect with gitlab-psql can query for all file.log and get the relative path to the bucket from the result. You can easily tool this by creating a batch file of commands for s5cmd log,

    \t
    \a
    \o /tmp/pull_logs.s5cmd
    SELECT FORMAT(
      'cp s3://artifacts/%s /tmp/job_%s.log',
      to_bucket_path(ci_job_artifacts),
      id
    )
    FROM ci_job_artifacts
    WHERE file = 'job.log'
    AND expire_at IS NULL
    AND file_store = 2;
    

    Which will generate a file you can run with something like s5cmd to pull down all of your logs from the Gitlab Object store. You'd run it by setting up your ~/.aws/credentials and running (example with minio).

    s5cmd --endpoint-url https://minio.acme.net run /tmp/pull_logs.s5cmd
    

• You can also use NFS. If you're using NFS for artifact storage, it's as easy as going to the mount point and searching with ripgrep.