0

I want to create a Kubernetes Job object like the example: https://kubernetes.io/docs/concepts/workloads/controllers/job/

Now imagine I want to have the result (in this case the value of Pi: 3.14159...) available in Prometheus.

Is this possible?

In a more complex example, imagine the output of my pod was JSON:

{
  "foo": 200,
  "bar": 10,
  "baz": 999
}

I somehow need to denote that I'm interested in foo and baz (but not bar) "being available" in Prom (note I'm not opinionated on whether the solution is push or pull.

Option 1

The first option I've thought of is somehow attaching the result 3.14159 to the job and Prom can scrape it as normal.

Option 2

Don't run the Pi-generator container at all, instead run something like a Python script that runs the Pi process. Then we can push from Python to Prom.

Option 3

Some other way?

1
  • Kudos to you for taking the commenter's feedback on Stack overflow and "moving" the question here.
    – DazWilkin
    Commented Aug 9, 2022 at 2:02

1 Answer 1

1

I think your need may be an anti-pattern but I'm unsure.

Metrics are used to measure things and Metrics are correlated with time.

In your example, the job's output is a constant and not correlated with a time.

Metrics are often (!) measurements of the health (of the state) of a system rather than the output (product) of a system.

The job's duration, CPU, memory, success|failure etc. are conventional (!) measurements.

While it's entirely reasonable to want to capture time-series data (from Jobs), it may be (!) that a database or some other persistence mechanism would be a better sink for your data.

Answering your question as stated: Jobs are challenging because they are potentially unscrapeable (because they don't live sufficiently long) by Prometheus as part of its 'pull' mechanism.

Batch jobs are a valid use-case for Pushgateway. So, your Jobs could push metrics to Pushgateway to ensure that these are captured.

If the logs produced by your Jobs are persisted beyond the life of the Job, another approach is to derive metrics (by parsing) logs. This approach biases towards aggregation of log data. The examplar is counting HTTP 500s in log entries to determine a failure rate.

2
  • Thanks Dan, I agree the question (as stated) is a bit strange but I did that to simplify the question. The real world usecase would indeed be tracking something dynamic like a job that processes a number of DB rows and I need to know how many DB rows I processed at 03:14 last night (for example). Thanks for confirming my initial thought that the push gateway is a good option here :)
    – Ezariah A
    Commented Aug 9, 2022 at 4:22
  • DB rows processed makes sense as a metric. I think you should, at least, explore using Pushgateway for your Jobs. Good luck!
    – DazWilkin
    Commented Aug 9, 2022 at 4:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.