0

Using helm chart :

helm install monitor prometheus-community/prometheus -n monitoring -f ./values.yaml

I cannot get prometheus server to running.

# kubectl get pods -owide -n monitoring
NAME                                              READY   STATUS             RESTARTS   AGE   IP               NODE              NOMINATED NODE   READINESS GATES
monitor-kube-state-metrics-578cdbb5b7-ffnbc       0/1     Running            0          19s   192.168.23.10    kube-worker-vm3   <none>           <none>
monitor-prometheus-alertmanager-c94f7b6b7-znfp5   1/2     Running            0          19s   192.168.222.53   kube-worker-vm2   <none>           <none>
monitor-prometheus-node-exporter-5vksg            1/1     Running            0          20s   192.168.1.36     kube-worker-vm2   <none>           <none>
monitor-prometheus-pushgateway-8694884546-qgqd2   0/1     Running            0          19s   192.168.222.54   kube-worker-vm2   <none>           <none>
monitor-prometheus-server-5587c7f464-z9tcj        1/2     CrashLoopBackOff   1          19s   192.168.222.55   kube-worker-vm2   <none>           <none>

Even though I had 1 replicaSet out of 2 running or crashloopbakcoff.

# kubectl describe pod monitor-prometheus-server-5587c7f464-z9tcj -n monitoring
Name:         monitor-prometheus-server-5587c7f464-z9tcj
Namespace:    monitoring
Priority:     0
Node:         kube-worker-vm2/192.168.1.36
Start Time:   Sun, 09 May 2021 22:14:12 +0100
Labels:       app=prometheus
              chart=prometheus-13.8.0
              component=server
              heritage=Helm
              pod-template-hash=5587c7f464
              release=monitor
Annotations:  cni.projectcalico.org/podIP: 192.168.222.55/32
              cni.projectcalico.org/podIPs: 192.168.222.55/32
Status:       Running
IP:           192.168.222.55
IPs:
  IP:           192.168.222.55
Controlled By:  ReplicaSet/monitor-prometheus-server-5587c7f464
Containers:
  prometheus-server-configmap-reload:
    Container ID:  docker://d4fdd420ec67f42857a4f20dd56de146307c65c49cd8ffccf6153f0b54082e91
    Image:         jimmidyson/configmap-reload:v0.4.0
    Image ID:      docker-pullable://jimmidyson/configmap-reload@sha256:17d34fd73f9e8a78ba7da269d96822ce8972391c2838e08d92a990136adb8e4a
    Port:          <none>
    Host Port:     <none>
    Args:
      --volume-dir=/etc/config
      --webhook-url=http://127.0.0.1:9090/-/reload
    State:          Running
      Started:      Sun, 09 May 2021 22:14:19 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/config from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from monitor-prometheus-server-token-njbp9 (ro)
  prometheus-server:
    Container ID:  docker://b53d3a56e2985594046a3b0018e9828597678268119c2dd7b07b1c7b315246e3
    Image:         prom/prometheus:v2.20.1
    Image ID:      docker-pullable://prom/prometheus@sha256:788260ebd13613456c168d2eed8290f119f2b6301af2507ff65908d979c66c17
    Port:          9090/TCP
    Host Port:     0/TCP
    Args:
      --storage.tsdb.retention.time=15d
      --config.file=/etc/config/prometheus.yml
      --storage.tsdb.path=/data
      --web.console.libraries=/etc/prometheus/console_libraries
      --web.console.templates=/etc/prometheus/consoles
      --web.enable-lifecycle
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Sun, 09 May 2021 22:20:24 +0100
      Finished:     Sun, 09 May 2021 22:20:24 +0100
    Ready:          False
    Restart Count:  6
    Liveness:       http-get http://:9090/-/healthy delay=30s timeout=30s period=15s #success=1 #failure=3
    Readiness:      http-get http://:9090/-/ready delay=30s timeout=30s period=5s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /data from storage-volume (rw)
      /etc/config from config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from monitor-prometheus-server-token-njbp9 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      monitor-prometheus-server
    Optional:  false
  storage-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  prometheus-pv-claim
    ReadOnly:   false
  monitor-prometheus-server-token-njbp9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  monitor-prometheus-server-token-njbp9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  boardType=x86vm
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                     From               Message
  ----     ------       ----                    ----               -------
  Normal   Scheduled    9m19s                   default-scheduler  Successfully assigned monitoring/monitor-prometheus-server-5587c7f464-z9tcj to kube-worker-vm2
  Warning  FailedMount  9m18s                   kubelet            MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition
  Normal   Pulled       9m13s                   kubelet            Container image "jimmidyson/configmap-reload:v0.4.0" already present on machine
  Normal   Created      9m13s                   kubelet            Created container prometheus-server-configmap-reload
  Normal   Started      9m12s                   kubelet            Started container prometheus-server-configmap-reload
  Normal   Pulled       8m24s (x4 over 9m12s)   kubelet            Container image "prom/prometheus:v2.20.1" already present on machine
  Normal   Created      8m23s (x4 over 9m12s)   kubelet            Created container prometheus-server
  Normal   Started      8m22s (x4 over 9m12s)   kubelet            Started container prometheus-server
  Warning  BackOff      4m16s (x32 over 9m10s)  kubelet            Back-off restarting failed container
1

monitor-prometheus-server-5587c7f464-z9tcj 1/2 CrashLoopBackOff

1/2 here means that 1 of 2 containers in the Pod is "Ready"

When you describe the Pod, you see the status of the containers:

Containers:
  prometheus-server-configmap-reload:
    Ready:          True
...
  prometheus-server:
    Ready:          False

You probably need to check the logs for the crashing container to see the reason.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.