0
# kubectl logs  kube-state-metrics-76cd57957f-h4q87 -n monitoring 
I0529 19:07:28.127093       1 main.go:86] Using default collectors
I0529 19:07:28.127733       1 main.go:98] Using all namespace
I0529 19:07:28.128006       1 main.go:139] metric white-blacklisting: blacklisting the following items: 
W0529 19:07:28.128272       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0529 19:07:28.132102       1 main.go:184] Testing communication with server
F0529 19:07:58.134533       1 main.go:147] Failed to create client: error while trying to communicate with apiserver: Get "https://10.96.0.1:443/version?timeout=32s": dial tcp 10.96.0.1:443: i/o timeout

I do have all the pods in kube-system namespace up and running.

# kubectl get pods -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
calico-kube-controllers-86bddfcff-2blpm   1/1     Running   6          30d
calico-node-ckpv2                         1/1     Running   5          27d
calico-node-jpfwm                         1/1     Running   3          27d
calico-node-prbh7                         1/1     Running   1          6d6h
coredns-74ff55c5b-8l9bj                   1/1     Running   6          30d
coredns-74ff55c5b-fmlp7                   1/1     Running   6          30d
etcd-rockpix                              1/1     Running   6          30d
kube-apiserver-rockpix                    1/1     Running   9          30d
kube-controller-manager-rockpix           1/1     Running   9          30d
kube-proxy-9x7mj                          1/1     Running   0          9h
kube-proxy-blrcq                          1/1     Running   5          27d
kube-proxy-v5src                          1/1     Running   3          27d
kube-scheduler-rockpix                    1/1     Running   9          30d

Here is the result of pod state .

# kubectl describe pod kube-state-metrics-76cd57957f-h4q87 -n monitoring 
Name:         kube-state-metrics-76cd57957f-h4q87
Namespace:    monitoring
Priority:     0
Node:         rockpi/192.168.1.10
Start Time:   Sat, 29 May 2021 15:44:35 +0100
Labels:       app=kube-state-metrics
              pod-template-hash=76cd57957f
Annotations:  cni.projectcalico.org/podIP: 192.168.42.131/32
              cni.projectcalico.org/podIPs: 192.168.42.131/32
Status:       Running
IP:           192.168.42.131
IPs:
  IP:           192.168.42.131
Controlled By:  ReplicaSet/kube-state-metrics-76cd57957f
Containers:
  kube-state-metrics:
    Container ID:   containerd://a1beda2985bf00b69cfb13a1f8279147f1870c4a1180b5c586cc06ebcd9784fe
    Image:          carlosedp/kube-state-metrics:v1.9.6
    Image ID:       docker.io/carlosedp/kube-state-metrics@sha256:0dccb8d0307e8a76dca499ecbf6b55b3780399e987da12d55d4e6bc8924821a1
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Sat, 29 May 2021 23:35:02 +0100
      Finished:     Sat, 29 May 2021 23:35:32 +0100
    Ready:          False
    Restart Count:  46
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from prometheus-token-lpmc8 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  prometheus-token-lpmc8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-token-lpmc8
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  board=rockpi
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Warning  BackOff  3m4s (x974 over 3h57m)  kubelet  Back-off restarting failed container
3
  • Are you on a cloud provider? Most cloud providers do not allow to scrape metrics. – rohatgisanat May 30 at 13:51
  • no sorry forgot to mention this setup is on bare metal nodes. – Chang Zhao May 30 at 17:42
  • i/o timeout means there is a connectivity problem.IP might have changed or it doesn't respond to the port. – rohatgisanat May 31 at 1:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.