I had a situation yesterday where my K8S master api cert has expired and I had Prometheus-operator active in the past weeks, but neither the 24 hours, neither the 7 days alert has triggered. I went to the dashboard and made the original querry, I got nothing. I cut it out and mod it to apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0, I got the result of all nodes, I checked the timers and moded it to apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 14000000 so I wanted one that is off to be cut and it was, that doesn't help me, but I think there is a problem with the alert, anyone else fought that? https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration If this is supposed to monitor all endpoints that make a request to the api server for their certificates then this failed as I didn't get an alert for the expired certificate.

1 Answer 1


The alert is broken, it is fixed in further release version of the mixins, we need to apply on(job) infront of the histogram to fix it. Example: apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) < 60432131800 and yes, that means your certs are not being tracked...

  • What is the significance of 60432131800 ?
    – zaTricky
    Commented Aug 21, 2020 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.