Getting error while trying to communicate with apiserver in kube-state-metrics pod

Question

I am getting error:

# kubectl logs  kube-state-metrics-fdc49658-g98sl -n monitoring
I0523 11:30:25.206556       1 main.go:86] Using default collectors
I0523 11:30:25.206766       1 main.go:98] Using all namespace
I0523 11:30:25.206800       1 main.go:139] metric white-blacklisting: blacklisting the following items: 
W0523 11:30:25.206886       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0523 11:30:25.216265       1 main.go:184] Testing communication with server
F0523 11:30:55.217773       1 main.go:147] Failed to create client: error while trying to communicate with apiserver: Get "https://10.96.0.1:443/version?timeout=32s": dial tcp 10.96.0.1:443: i/o timeout

Here is the manifest for Kube-state-metrics:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
spec:
  selector:
    matchLabels:
      app: kube-state-metrics
  replicas: 1
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      nodeSelector:
        boardType: aarch64
      serviceAccountName: prometheus
      containers:
      - name: kube-state-metrics
        image: carlosedp/kube-state-metrics:v1.9.6
        ports:
        - containerPort: 8080
          name: monitoring
---
kind: Service
apiVersion: v1
metadata:
  name: kube-state-metrics
spec:
  selector:
    app: kube-state-metrics
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080

Answer 1

This doesn't look like kube-state-metrics but your node configuration/networking issue. You will need to debug node to node communication. I would start by checking on which nodes the kube-state-metrics is scheduled and launch a temporary pod on that node to see if you can access the API Server pod correctly. The error could be in subnet, security groups/ firewalls, or other networking and security-related components.

Answer 2

The problem I encountered was that I had adjusted the secrets before the certificate was deleted.

 kubectl -n monitoring delete pods kube-state-metrics-xxxxx