Got "Error: chart requires kubeVersion: >=1.19.0-0 which is incompatible with Kubernetes v1.16.0" when deploy kube-prometheus-stack using Argo CD

Question

My Amazon EKS is using Kuberentes 1.30:

➜ kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0-eks-036c24b

However, when I deploy kube-prometheus-stack by Argo CD, I got error:

➜ argocd app create horizon-monitoring
--file=manifests/horizon-monitoring.yaml FATA[0001] rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: Unable to generate manifests in : rpc error: code = Unknown desc = `helm template . --name-template kube-prometheus-stack --namespace production-horizon-monitoring --api-versions v1 --api-versions apiregistration.k8s.io/v1 --api-versions apps/v1 --api-versions events.k8s.io/v1 --api-versions authentication.k8s.io/v1
--api-versions authorization.k8s.io/v1 --api-versions autoscaling/v2 --api-versions autoscaling/v1 --api-versions batch/v1 --api-versions certificates.k8s.io/v1 --api-versions networking.k8s.io/v1
--api-versions policy/v1 --api-versions rbac.authorization.k8s.io/v1 --api-versions storage.k8s.io/v1 --api-versions admissionregistration.k8s.io/v1 --api-versions apiextensions.k8s.io/v1
--api-versions scheduling.k8s.io/v1 --api-versions coordination.k8s.io/v1 --api-versions node.k8s.io/v1 --api-versions discovery.k8s.io/v1 --api-versions flowcontrol.apiserver.k8s.io/v1
--api-versions flowcontrol.apiserver.k8s.io/v1beta3 --api-versions argoproj.io/v1alpha1 --api-versions bitnami.com/v1alpha1
--api-versions crd.k8s.amazonaws.com/v1alpha1 --api-versions networking.k8s.aws/v1alpha1 --api-versions vpcresources.k8s.aws/v1beta1 --api-versions vpcresources.k8s.aws/v1alpha1 --include-crds` failed exit status 1: Error: chart requires kubeVersion: >=1.19.0-0 which is incompatible with Kubernetes v1.16.0

Use --debug flag to render out invalid YAML

which says I am on Kubernetes v1.16.0 while I am on Kubernetes v1.30.0 actually.

My my-monitoring.yaml looks like:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-monitoring
  namespace: production-my-argo-cd
  labels:
    app.kubernetes.io/name: my-monitoring
spec:
  project: default
  source:
    chart: kube-prometheus-stack
    repoURL: https://prometheus-community.github.io/helm-charts
    targetRevision: 60.1.0
    helm:
      releaseName: kube-prometheus-stack
  destination:
    namespace: production-my-monitoring
    server: https://kubernetes.default.svc

I didn't find any useful info at https://github.com/prometheus-community/helm-charts/issues

Any idea how to solve this error?

Answer 1

It turns out the issue was caused by I deploying an old Argo CD server v 2.11.3 instead of the new v7.1.3. Because initially I got confused by these two Argo CD related versions at https://artifacthub.io/packages/helm/argo/argo-cd

I got the hint from @jhughes2112 who mentioned old Helm version may cause similar issue at https://github.com/helm/helm/issues/10819#issuecomment-1435727947 (Thanks!)

You can see this old Argo CD server v2.11.3 is related to some old Kubectl v1.17.8 and Helm v3.2. I am guessing it may make kube-prometheus-stack get confused during deployment.

➜ argocd version
argocd: v2.11.2+25f7504
  BuildDate: 2024-05-23T15:31:09Z
  GitCommit: 25f7504ecc198e7d7fdc055fdb83ae50eee5edd0
  GitTreeState: clean
  GoVersion: go1.22.3
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v1.7.6+b04c25e
  BuildDate: 2020-09-19T00:52:04Z
  GitCommit: b04c25eca8f1660359e325acd4be5338719e59a0
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.17.8

After deploying the new Argo CD server v7.1.3:

➜ argocd version
argocd: v2.11.2+25f7504
  BuildDate: 2024-05-23T15:31:09Z
  GitCommit: 25f7504ecc198e7d7fdc055fdb83ae50eee5edd0
  GitTreeState: clean
  GoVersion: go1.22.3
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.11.3+3f344d5
  BuildDate: 2024-06-06T08:42:00Z
  GitCommit: 3f344d54a4e0bbbb4313e1c19cfe1e544b162598
  GitTreeState: clean
  GoVersion: go1.21.9
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.14.4+g81c902a
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0

I can succeed deploying kube-prometheus-stack as well.