I am learning rancher (still) and after deploying the monitoring chart (prom, grafana etc) i need to add more targets to the prometheus config (such as other bare-metal server and some QEMU too), I know I can update the config by entering the pods via shell but these changes would be lost after redeploy or upgrade of the chart.
I tends to think that the chart uses configMap to handle the prometheus config but (and i'm feeling noobish here) I cannot find a way to update this configMap, well I just cannot find this specific configMap.
I went through the helm quickstart and chart update docs but this did not help, same goes for the rancher docs.
Any hint will be highly apreciated.
For example in the cattle-monitoring-system namespace I can see these configmaps :
pierre@latpopo:~$ k get cm -n cattle-monitoring-system
NAME DATA AGE
grafana-nginx-proxy-config 1 2d23h
kube-root-ca.crt 1 2d23h
prometheus-nginx-proxy-config 1 2d23h
prometheus-rancher-monitoring-prometheus-rulefiles-0 26 2d22h
rancher-monitoring-crd-manifest 1 2d22h
rancher-monitoring-grafana 1 2d23h
rancher-monitoring-grafana-config-dashboards 1 2d23h
rancher-monitoring-grafana-datasource 1 2d23h
rancher-monitoring-grafana-test 1 2d23h
rancher-monitoring-prometheus-adapter 1 2d23h
None of them seems to be hosting the prometheus config file
In the describe for what I think is the actual prom server deployment I can see there is a use of a configMap :
k describe statefulsets prometheus-rancher-monitoring-prometheus -n cattle-monitoring-system
[...]
--config.file=/etc/prometheus/config_out/prometheus.env.yaml
[...]
Mounts:
/etc/prometheus/config from config (rw)
[...]
But still, I can't find how to change in a persistent manner
EDIT : Ok by reviewing again the rancher docs i finally saw :
For information on configuring custom scrape targets and rules for Prometheus, please refer to the upstream documentation for the Prometheus Operator.
But following that documentation path led me to the prometheus-operator which seems to be the correct place for editing the prometheus config, but this also seems to be only usable for kubernetes based resources: Looks like this is not the correct place to add external targets and services monitoring.
So the question still stand:
How can I add new targets (external to kubernetes) to the prometheus "chart deployed" ?