Enterprise Ready Prometheus

Question

I am new to Prometheus and have been reading about its popularity as a tool for monitoring applications, containers and K8s objects. However, most of the blogs that I have come across have at best done a very rudimentary introduction to using Prometheus for mission critical applications and I am trying to compile a list of everything Prometheus that an SRE should be aware of when setting Prometheus as the tool of your choice for monitoring.

I am listing out my current understanding, so feel free to correct if I may have got something wrong. But would really like to hear from the community on how best have they used Prometheus.

a) Installation of Prometheus -> Where is the ideal location to install Prometheus? Some of the blog posts talk about installing this on the same K8s cluster that you have deployed your applications on. To me, that sounds counterintuitive, since now the monitoring stack eats up into your application resources. OR Should we deploy prometheus on a separate K8s cluster of its own. In this case, how does the configuration file changes to automatically scrape say two clusters which have two different applications deployed.

b) What all do you scrape using Prometheus? So far, I have gathered these

1- cAdvisor [ for collecting resource and usage data for running containers]

2- kube-state-metrics [ for gathering stats about state of K8s objects].

3- Node Exporters [ For gathering node related metrics]

4- Custom Exporters [ For your applications not emitting metrics in prometheus format]

Would be interested to hear, what else should one be monitoring ??

c) How does prometheus config.yaml file look like, when monitoring these above plugins/exporters. Do you manually add them to the scrape target? How would automatic service discovery work in the above cases? Can someone share a sample <kubernetes_sd_config> for their production scenarios [ Obviously obfuscating the application or specific details]

d) What is the ideal backend storage to be used for storing this time series data and how long should one be storing these data for? I think some of it may be driven from the compliance requirements of your application/organization, so in case where long term storage is needed, what are the practices adopted for metrics retention.

e) How is the DR strategy planned for Prometheus? I understand that since you are now monitoring your application using Prometheus, how do you ensure that this system has an effective DR plan? How would one design one?

I would appreciate inputs from anyone/everyone who has experience with Prometheus and how has this tool helped you in your monitoring pursuits.

Answer 1

My current client is looking at embedding Prometheus in the application cluster, along with EFK, Vault, OPA, Consul, MongoDB and RedPanda. We call that the core architecture. We will have one cluster per 100 tenants. We will have a dev and a few test clusters too, each based on the same core. There will also be a pipeline cluster to build and validate source-code, which will have the same core tooling, plus tekton and argocd, and a container registry and nexus. I'd like to put the git repo in there too, but we're addicted to co-pilot and code-spaces, so we're going for github.

I think we'll eventually need to aggregate all this information in a single pane of glass, so we will need to create a cluster just for prom/graf (and probably efk too) and forward cluster-specific metrics to it along with some app specific stuff. But, we'll still go to an individual cluster's graf in order to view most of that cluster's application-specific stuff.