I am running a local Kubernetes cluster with Minikube on a Windows machine using --driver=hyperv.
I am trying to set up an HPA (HorizontalPodAutoscaler) based on Pods custom metrics. Using Helm, I deployed the kube-prometheus-stack to scrape more advanced metrics and everything is working fine.
I then deployed the prometheus-adapter with Helm and defined rules so that the adapter monitors nodes/pods resources and a set of custom metrics. After deploying the adapter, I'm able to query the metrics API with kubectl top pod as well as the custom metrics API with kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1.

However the HPA and myself are unable to get any metrics out of the custom metrics API. kubectl describe hpa/my-hpa displays the following warning in the events section:
FailedGetPodsMetric horizontal-pod-autoscaler unable to get metric bytes_per_second: unable to fetch metrics from custom metrics API: Internal error occurred: unable to fetch metrics

The prometheus adapter is correctly linked to Prometheus, I'm using the default K8S namespace and the custom rules defined for the adapter are as follow:

  default: false
  - seriesQuery: '{__name__=~"^container_network_receive_bytes_total$"}'
          resource: "namespace"
          resource: "pod"
      matches: ""
      as: "bytes_per_second"
    metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>})) by (<<.GroupBy>>)

What am I missing ?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.