1

I'm considering using Prometheus to monitor and analyze different metrics from my AWS EKS Kubernetes cluster running in Fargate, as it seems to be the standard tool during these times. However, something that really impressed me was that installing Prometheus itself in the very same k8s cluster (as another running set of pods) is a normal thing. Am I right or just got a wrong idea from my research? Considering good practices, shouldn't the monitoring tool be outside of the system that you actually intend to monitor? Also, if I set up multiple Kubernetes clusters, isn't it more cumbersome to have one Prometheus installation per cluster? Is it possible to set up a single Prometheus installation and centralize all k8s clusters metrics on it?

Same question for Grafana's installation... Should it be installed inside or outside the cluster?

On a different topic, Prometheus history retention period is set to 15 days by default (storage.tsdb.retention.time). Even though cluster metrics insight should be useful for a short period of time so you can troubleshoot issues, I wonder if there may be scenarios in which a longer history period might help. Any comments on this matter?

Thanks!

1

This is more of an opinion based question but let me try to answer it.

installing Prometheus itself in the very same k8s cluster (as another running set of pods) is a normal thing.Am I right or just got a wrong idea from my research?

Considering good practices, shouldn't the monitoring tool be outside of the system that you actually intend to monitor?

This is a correct practice to have. Monitoring systems are usually setup to look into the object being monitored. (outside looking in). But with self healing systems like k8s, installing monitoring within the system can be done with minimum issues. There are a few cases you do need to look into for proper stability.

  • Pods get deleted :- Run Prometheus in a HA mode (2 replicas minimum) so as not to lose any data.
  • Pods are replaced due to resource crunch:- Add priority classes to prevent pod form being evicted.
  • Use Persistent volumes:- Look into volume node affinity conflict. Also configure the data retention period to make optimised use of storage.
  • Cluster goes down. See below.

Also, if I set up multiple Kubernetes clusters, isn't it more cumbersome to have one Prometheus installation per cluster?

Not at all, using a combination of IaC(Terraform/Chef),Config management(Ansible),CI/CD (Spinnaker),GitOps and Helm charts. Configuring/maintaining Prometheus instances across multiple instances will not be an issue.

Is it possible to set up a single Prometheus installation and centralise all k8s clusters metrics on it?

Prometheus although highly optimised, will not be able to handle all the metrics thrown at it from multiple clusters.Also Prometheus does not scale horizontally on its own, therefore it would need huge resources. Centralising all metrics to one place will cause other issues as well.

  • Network Connectivity/Latency:- Running a centralised Prometheus instance receiving data from over the public internet may cause data loss/false alerts due to network drops and latency.
  • Cost:- Running tens of exporter all sending data over the internet will cause high internet costs for cloud deployments.
  • Cluster managing:- You may face issues with exporter/job configuration of such a large number of exporters.

Its a good choice to run Prometheus per cluster. wherein you can have all the data locally till the time is required.

Grafana does not consume much resources so its safe to run inside the cluster. However it faces a different set of issues:-

  • User/Permission Management :- Each local instance would need to be maintained separately which is a huge task.
  • Charts Management:- Same as above.
  • L1/DevOps will have to keep n number of tabs open to monitor your infra.

Other than this, setting up alerts on all the clusters separately would also be a never-ending task.

My advice would be to setup Prometheus clusters locally (inside the cluster or right next to it) and connect all of them via Thanos or Cortex and set up a HA mode Grafana centrally on top of Thanos along with alerts and external storage.This will alert if the local Prometheus instance goes down and will help with the outside looking in factor

scenarios in which a longer history period might help. Any comments on this matter?

Depends upon the metrics being stored, A quarterly SLI/SLO and incident reports seem like a good use case to store data for longer than 15 days.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.