how to change a rancher UI installed prometheus server config

Question

I am learning rancher (still) and after deploying the monitoring chart (prom, grafana etc) i need to add more targets to the prometheus config (such as other bare-metal server and some QEMU too), I know I can update the config by entering the pods via shell but these changes would be lost after redeploy or upgrade of the chart.

I tends to think that the chart uses configMap to handle the prometheus config but (and i'm feeling noobish here) I cannot find a way to update this configMap, well I just cannot find this specific configMap.

I went through the helm quickstart and chart update docs but this did not help, same goes for the rancher docs.

Any hint will be highly apreciated.

For example in the cattle-monitoring-system namespace I can see these configmaps :

pierre@latpopo:~$ k get cm -n cattle-monitoring-system
NAME                                                   DATA   AGE
grafana-nginx-proxy-config                             1      2d23h
kube-root-ca.crt                                       1      2d23h
prometheus-nginx-proxy-config                          1      2d23h
prometheus-rancher-monitoring-prometheus-rulefiles-0   26     2d22h
rancher-monitoring-crd-manifest                        1      2d22h
rancher-monitoring-grafana                             1      2d23h
rancher-monitoring-grafana-config-dashboards           1      2d23h
rancher-monitoring-grafana-datasource                  1      2d23h
rancher-monitoring-grafana-test                        1      2d23h
rancher-monitoring-prometheus-adapter                  1      2d23h

None of them seems to be hosting the prometheus config file

In the describe for what I think is the actual prom server deployment I can see there is a use of a configMap :

k describe statefulsets prometheus-rancher-monitoring-prometheus -n cattle-monitoring-system
[...]
--config.file=/etc/prometheus/config_out/prometheus.env.yaml
[...]
    Mounts:
      /etc/prometheus/config from config (rw)
[...]

But still, I can't find how to change in a persistent manner

EDIT : Ok by reviewing again the rancher docs i finally saw :

For information on configuring custom scrape targets and rules for Prometheus, please refer to the upstream documentation for the Prometheus Operator.

But following that documentation path led me to the prometheus-operator which seems to be the correct place for editing the prometheus config, but this also seems to be only usable for kubernetes based resources: Looks like this is not the correct place to add external targets and services monitoring.

So the question still stand:

How can I add new targets (external to kubernetes) to the prometheus "chart deployed" ?

Answer 1

If you could let us know what helm chart you have used to install prom-operator, that might help refine the answer.

But in case of kube-prometheus-stack, in the values.yaml of the , use the additionalScrapeConfigs section to describe jobs that are external to k8s system and re-deploy.

Or use serviceMonitors as described in this article