0

I followed the steps here https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-ingest-metrics-new-Prometheus.html

In a nutshell, I installed the Prometheus community helm chart on my EKS cluster in AWS https://github.com/prometheus-community/helm-charts.

My Kubernetes version is 1.27

The cluster node group has two t3.large nodes (8gb memory, 2vCPU)

After creation, I have:

Pods:

kubectl get pods -n prometheus

NAME                                                        READY   STATUS             RESTARTS         AGE
prometheus-alertmanager-0                            1/1     Running            0                142m
prometheus-kube-state-metrics-69b5df7dc4-kfqc2       1/1     Running            0                142m
prometheus-prometheus-node-exporter-dkksq            1/1     Running            0                142m
prometheus-prometheus-node-exporter-fzr5z            1/1     Running            0                142m
prometheus-prometheus-pushgateway-847c6f4d57-dz5wx   1/1     Running            0                142m
prometheus-server-5fbb47548d-sxm72                   1/2     CrashLoopBackOff   32 (3m58s ago)   142m

Pod logs:

 kubectl logs prometheus-server-5fbb47548d-sxm72 -n prometheus

....
level=error ts=2023-08-04T17:45:27.636671778Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://127.0.0.1:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"

Deployments:

kubectl get deployments -n prometheus
NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-kube-state-metrics       1/1     1            1           144m
prometheus-prometheus-pushgateway   1/1     1            1           144m
prometheus-server                   0/1     1            0           144m

Describe failed deployment:

kubectl describe deployments prometheus-server -n prometheus

...
 Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    False   ProgressDeadlineExceeded

The EBS volumes have successfully bound.

It seems that the prometheus-server pod is stuck in CrashLoopBackOff because it can't connect to itself on 127.0.0.1:9090

I have been searching for an answer all day and trying different things but can't solve the issue.

My helm values looks like this:

## The following is a set of default values for prometheus server helm chart which enable remoteWrite to AMP
## For the rest of prometheus helm chart values see: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml
##
serviceAccounts:
  server:
    name: amp-iamproxy-ingest-service-account
    annotations: 
      eks.amazonaws.com/role-arn: arn:aws:iam:###########:role/service-role/Amazon_EventBridge_Invoke_Api_Destination_###########
server:
  remoteWrite:
    - url: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/#############/api/v1/remote_write
      sigv4:
        region: eu-west-1
      queue_config:
        max_samples_per_send: 1000
        max_shards: 200
        capacity: 2500

I can't figure this out, please help! I'm sure someone else has struggled with the same issue because I can't find much info on this. Why is it failing? What Can I do to fix it? Thanks in advance

2 Answers 2

0

I've had this issue in the past as well and I ended up noticing that my nodes were not joining the master.

Ensure that your nodes have joined the master and are in a ready state.

0

kubectl logs [pod-name] -c prometheus-server -n prometheus

level=error msg="Failed to apply configuration" err="could not get SigV4 credentials: WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for

check whether you missed the openID

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.