Prometheus-operator - detection of expiring certs failing

Question

I had a situation yesterday where my K8S master api cert has expired and I had Prometheus-operator active in the past weeks, but neither the 24 hours, neither the 7 days alert has triggered. I went to the dashboard and made the original querry, I got nothing. I cut it out and mod it to apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0, I got the result of all nodes, I checked the timers and moded it to apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 14000000 so I wanted one that is off to be cut and it was, that doesn't help me, but I think there is a problem with the alert, anyone else fought that? https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration If this is supposed to monitor all endpoints that make a request to the api server for their certificates then this failed as I didn't get an alert for the expired certificate.

Answer 1

The alert is broken, it is fixed in further release version of the mixins, we need to apply on(job) infront of the histogram to fix it. Example: apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) < 60432131800 and yes, that means your certs are not being tracked...