0

I am attempting to setup metrics-server to server my 'horizon pod autoscaling' and manage my app pods scaling based on CPU. I am unable to get the metrics-server to run properly which fails with Failed probe" probe="metric-storage-ready" err="no metrics to serve. You can tell I am using EKS and managing the cluster using Terraform. Can anyone offer any insights?

Versions
  - EKS 1.27
  - Helm 3.12.0

This is my terraform script showing this specific piece of my cluster:

resource "kubernetes_namespace" "metric-server" {
  metadata {
    name = "${var.cluster_name}-metric-server"
  }
}

resource "helm_release" "metric_server" {
  name       = "metric-server"
  repository = "https://charts.bitnami.com/bitnami"
  chart      = "metric-server"
  namespace  = "${var.cluster_name}-metric-server"
  
  values = [
    "${file("./values/metrics-server.yaml")}"
  ]
}

The contents of ./values/metrics-server.yaml as follows.

spec:
  containers:
  - arg:
      - --metric-resolution=18s

This is my cluster (kube-system)

kubectl get po -n kube-system
NAME                                  READY   STATUS    RESTARTS       AGE
NAMESPACE                 NAME                                     READY   STATUS      RESTARTS        AGE
default                   reloader-reloader-7568f766fb-mkdzh       1/1     Running     0               3d8h
kube-system               aws-node-8w7lc                           2/2     Running     0               4d5h
kube-system               aws-node-927jv                           2/2     Running     0               4d5h
kube-system               aws-node-hhj66                           2/2     Running     0               4d5h
kube-system               aws-node-hpb57                           2/2     Running     0               4d5h
kube-system               aws-node-ng7nc                           2/2     Running     0               4d5h
kube-system               coredns-79b5589f65-fb64k                 1/1     Running     0               52d
kube-system               coredns-79b5589f65-fkkrw                 1/1     Running     0               52d
kube-system               ebs-csi-controller-77fcc9d855-55555      6/6     Running     9 (3d8h ago)    21d
kube-system               ebs-csi-controller-77fcc9d855-crrrr      6/6     Running     0               21d
kube-system               ebs-csi-node-d2qsw                       3/3     Running     0               21d
kube-system               ebs-csi-node-flw5z                       3/3     Running     0               21d
kube-system               ebs-csi-node-p5r5n                       3/3     Running     0               21d
kube-system               ebs-csi-node-tmdzd                       3/3     Running     0               21d
kube-system               ebs-csi-node-xpq88                       3/3     Running     0               21d
kube-system               efs-csi-controller-6f7f5f5787-ccccc      3/3     Running     0               52d
kube-system               efs-csi-controller-6f7f5f5787-xxxxx      3/3     Running     0               52d
kube-system               efs-csi-node-7zgx2                       3/3     Running     0               52d
kube-system               efs-csi-node-8hcmx                       3/3     Running     0               52d
kube-system               efs-csi-node-mhr9m                       3/3     Running     0               21d
kube-system               efs-csi-node-mzbfq                       3/3     Running     0               52d
kube-system               efs-csi-node-w859z                       3/3     Running     0               21d
kube-system               kube-proxy-28s6c                         1/1     Running     0               52d
kube-system               kube-proxy-4fjdc                         1/1     Running     0               21d
kube-system               kube-proxy-h47q6                         1/1     Running     0               52d
kube-system               kube-proxy-h5sjf                         1/1     Running     0               21d
kube-system               kube-proxy-pg5fx                         1/1     Running     0               52d
kube-system               metrics-server-5b787c6cf8-21356          1/1     Running     0               6h40m
dev-opensearch            opensearch-cluster-master-0              1/1     Running     0               52d
dev-opensearch            opensearch-cluster-master-1              1/1     Running     0               52d
dev-opensearch            opensearch-cluster-master-2              1/1     Running     0               52d
dev-rabbitmq              rabbitmq-0                               1/1     Running     0               9d
dev-rabbitmq              rabbitmq-1                               1/1     Running     0               9d
dev-rabbitmq              rabbitmq-2                               1/1     Running     0               9d
dev-redis                 redis-master-0                           1/1     Running     0               10d
dev-redis                 redis-replicas-0                         1/1     Running     0               3d8h
dev-redis                 redis-replicas-1                         1/1     Running     0               3d8h
dev-app                   app-app-enterprise-0                     2/2     Running     0               3d8h 
dev-app                   app-app-enterprise-1                     2/2     Running     0               3d8h
dev                       example-97f846598-hl6cc                  1/1     Running     0               28h
dev                       example-cron-28418300-d6xmw              0/1     Completed   0               12m
dev                       example-cron-28418305-c9hhj              0/1     Completed   0               7m4s
dev                       example-cron-28418310-chbfl              0/1     Completed   0               2m4s
dev                       example-post-upgrade-hlpmg               0/1     Completed   0               32h
dev                       ubuntu                                   1/1     Running     1 (3d11h ago)   10d
zero-trust                cloudflared-65bcdff7c6-gnp29             1/1     Running     0               2d13h
zero-trust                cloudflared-65bcdff7c6-grnmw             1/1     Running     0               2d13h

This is my hpa:

kubectl get hpa --all-namespaces
NAMESPACE   NAME      REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
dev      example   Deployment/example   <unknown>/25%   1         11        1          24h

When I check my cluster logs:

kubectl describe hpa --all-namespaces
Name:                                                  example
Namespace:                                             dev
Labels:                                                app.kubernetes.io/instance=example
                                                       app.kubernetes.io/managed-by=Helm
                                                       app.kubernetes.io/name=example
                                                       helm.sh/chart=example-23.0.1
Annotations:                                           meta.helm.sh/release-name: example
                                                       meta.helm.sh/release-namespace: dev
CreationTimestamp:                                     Thu, 11 Jan 2024 17:57:10 +0000
Reference:                                             Deployment/example
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 25%
Min replicas:                                          1
Max replicas:                                          11
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                   From                       Message
  ----     ------                   ----                  ----                       -------
  Warning  FailedGetResourceMetric  19s (x5776 over 24h)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

metrics-server logs:

kubectl logs deployment/metrics-server -n kube-system
I0112 15:51:32.519856       1 serving.go:342] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0112 15:51:32.996832       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0112 15:51:32.996894       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0112 15:51:32.996976       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0112 15:51:32.997058       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0112 15:51:32.997119       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0112 15:51:32.997164       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0112 15:51:32.997877       1 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key"
I0112 15:51:32.998109       1 secure_serving.go:267] Serving securely on [::]:8443
I0112 15:51:32.998136       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
W0112 15:51:33.000474       1 shared_informer.go:372] The sharedIndexInformer has started, run more than once is not allowed
I0112 15:51:33.002798       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:51:33.097938       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0112 15:51:33.097937       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0112 15:51:33.097969       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0112 15:51:33.939463       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:51:37.866094       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:51:47.866103       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:51:57.866445       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:52:07.867043       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:52:17.865920       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:52:27.866276       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"

Further queries yield no metrics

kubectl top pod -A
error: Metrics API not available
kubectl top pods -n kube-system
error: Metrics API not available

Checking the metrics API, shows nothing:

$kubectl api-versions | grep metrics
external.metrics.k8s.io/v1beta1
Improve this question
3
  • That means your metrics-server cannot reach kubelet for getting metrics. Make sure metrics-server can reach nodes on port 10250. You can also try to add --kubelet-insecure-tls=true to metrics-server args.
    – Vasilii Angapov
    Commented Jan 14 at 23:00
  • Thank you @VasiliiAngapov for your suggestions, I can confirm this port is allowed. I had added the additional parameter of --kubelet-insecure-tls=true. Unfortunately, I am still seeing this error: 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
    – Don Draper
    Commented Jan 15 at 14:58
  • @VasiliiAngapov I should point out that the changes you suggested had a positive effect as I can now see the metric api. I've included that option at the very bottom of the post.
    – Don Draper
    Commented Jan 15 at 15:14

1 Answer 1

Reset to default
0

I resolved this by using a different helm chart. I believe my original problem was caused by a version incompatibility.

resource "helm_release" "metrics_server" {
  name       = "metrics-server"
  repository = "https://kubernetes-sigs.github.io/metrics-server"
  chart      = "metrics-server"
  namespace  = "kube-system"
}
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.