This is more of an opinion based question but let me try to answer it.
installing Prometheus itself in the very same k8s cluster (as another running set of pods) is a normal thing.Am I right or just got a wrong idea from my research?
Considering good practices, shouldn't the monitoring tool be outside of the system that you actually intend to monitor?
This is a correct practice to have. Monitoring systems are usually setup to look into the object being monitored. (outside looking in). But with self healing systems like k8s, installing monitoring within the system can be done with minimum issues. There are a few cases you do need to look into for proper stability.
- Pods get deleted :- Run Prometheus in a HA mode (2 replicas minimum)
so as not to lose any data.
- Pods are replaced due to resource crunch:- Add priority classes
to prevent pod form being evicted.
- Use Persistent volumes:- Look into volume node affinity
conflict. Also configure the data retention period to make
optimised use of storage.
- Cluster goes down. See below.
Also, if I set up multiple Kubernetes clusters, isn't it more cumbersome to have one Prometheus installation per cluster?
Not at all, using a combination of IaC(Terraform/Chef),Config management(Ansible),CI/CD (Spinnaker),GitOps and Helm charts. Configuring/maintaining Prometheus instances across multiple instances will not be an issue.
Is it possible to set up a single Prometheus installation and centralise all k8s clusters metrics on it?
Prometheus although highly optimised, will not be able to handle all the metrics thrown at it from multiple clusters.Also Prometheus does not scale horizontally on its own, therefore it would need huge resources. Centralising all metrics to one place will cause other issues as well.
- Network Connectivity/Latency:- Running a centralised Prometheus
instance receiving data from over the public internet may cause data
loss/false alerts due to network drops and latency.
- Cost:- Running tens of exporter all sending data over the internet
will cause high internet costs for cloud deployments.
- Cluster managing:- You may face issues with exporter/job
configuration of such a large number of exporters.
Its a good choice to run Prometheus per cluster. wherein you can have all the data locally till the time is required.
Grafana does not consume much resources so its safe to run inside the cluster. However it faces a different set of issues:-
- User/Permission Management :- Each local instance would need to be
maintained separately which is a huge task.
- Charts Management:- Same as above.
- L1/DevOps will have to keep n number of tabs open to monitor your
Other than this, setting up alerts on all the clusters separately would also be a never-ending task.
My advice would be to setup Prometheus clusters locally (inside the cluster or right next to it) and connect all of them via Thanos or Cortex
and set up a HA mode Grafana centrally on top of Thanos along with alerts and external storage.This will alert if the local Prometheus instance goes down and will help with the outside looking in factor
scenarios in which a longer history period might help. Any comments on this matter?
Depends upon the metrics being stored, A quarterly SLI/SLO and incident reports seem like a good use case to store data for longer than 15 days.