I am new to Prometheus and have been reading about its popularity as a tool for monitoring applications, containers and K8s objects. However, most of the blogs that I have come across have at best done a very rudimentary introduction to using Prometheus for mission critical applications and I am trying to compile a list of everything Prometheus that an SRE should be aware of when setting Prometheus as the tool of your choice for monitoring.
I am listing out my current understanding, so feel free to correct if I may have got something wrong. But would really like to hear from the community on how best have they used Prometheus.
a) Installation of Prometheus -> Where is the ideal location to install Prometheus? Some of the blog posts talk about installing this on the same K8s cluster that you have deployed your applications on. To me, that sounds counterintuitive, since now the monitoring stack eats up into your application resources. OR Should we deploy prometheus on a separate K8s cluster of its own. In this case, how does the configuration file changes to automatically scrape say two clusters which have two different applications deployed.
b) What all do you scrape using Prometheus? So far, I have gathered these
1- cAdvisor [ for collecting resource and usage data for running containers]
2- kube-state-metrics [ for gathering stats about state of K8s objects].
3- Node Exporters [ For gathering node related metrics]
4- Custom Exporters [ For your applications not emitting metrics in prometheus format]
Would be interested to hear, what else should one be monitoring ??
c) How does prometheus config.yaml file look like, when monitoring these above plugins/exporters. Do you manually add them to the scrape target? How would automatic service discovery work in the above cases? Can someone share a sample <kubernetes_sd_config> for their production scenarios [ Obviously obfuscating the application or specific details]
d) What is the ideal backend storage to be used for storing this time series data and how long should one be storing these data for? I think some of it may be driven from the compliance requirements of your application/organization, so in case where long term storage is needed, what are the practices adopted for metrics retention.
e) How is the DR strategy planned for Prometheus? I understand that since you are now monitoring your application using Prometheus, how do you ensure that this system has an effective DR plan? How would one design one?
I would appreciate inputs from anyone/everyone who has experience with Prometheus and how has this tool helped you in your monitoring pursuits.