0

I had a situation yesterday where my K8S master api cert has expired and I had Prometheus-operator active in the past weeks, but neither the 24 hours, neither the 7 days alert has triggered. I went to the dashboard and made the original querry, I got nothing. I cut it out and mod it to apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0, I got the result of all nodes, I checked the timers and moded it to apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 14000000 so I wanted one that is off to be cut and it was, that doesn't help me, but I think there is a problem with the alert, anyone else fought that? https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration If this is supposed to monitor all endpoints that make a request to the api server for their certificates then this failed as I didn't get an alert for the expired certificate.

0

The alert is broken, it is fixed in further release version of the mixins, we need to apply on(job) infront of the histogram to fix it. Example: apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) < 60432131800 and yes, that means your certs are not being tracked...

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.