1

We want to start our monitoring off the cluster on a local server. Is this practical and will it be feasible and reasonable to hook local Prometheus and Grafana installations into a Kubernetes cluster?

Improve this question

1 Answer 1

Reset to default
3

It's possible but after doing many replatforming projects for clients I can't say I would advise it. Have you seen kube-prometheus? It's a one-shot install that will get you 90% of where you need to be for your cluster.

The only way I can advise it would be if you're going to run 5 or more clusters and want central monitoring, but I'm a bigger fan of having a Grafana per cluster.

Edit: Getting started with kube-prometheus is a one command install. kubectl apply -f manifests/

Improve this answer
7
  • I did a 15 minute Prometheus install from a Medium article. It was amazing. Still we have an ask to have the monitoring outside the cluster so that when the cluster goes down the monitoring is there. Is there a way to do that? Is the ROI high enough? Or do you still recommend in cluster?
    – David West
    Commented Apr 4, 2018 at 12:24
  • @DavidWest What will you monitor? Are you running ETCD on servers off the cluster as well? See when you lose the API servers, you are going to lose monitoring of your services eventually. This could be immediately or over time. This is because Prometheus relies on the API servers for service discovery and internally some meta_tags provided by the API servers for configuration. You can use certain POD labels to ensure your Prometheus pod is the "last man standing". I believe the recommended approach is to run Prometheus on top of Kubernetes.
    – Levi
    Commented Apr 4, 2018 at 14:41
  • 1
    It seems to be working well in cluster. Plus, it was so easy to do it that way.
    – David West
    Commented Apr 10, 2018 at 17:41
  • 1
    How do you get the application metrics to Prometheus?
    – avi
    Commented Feb 24, 2019 at 13:30
  • 1
    @avi You should probably start a new question ;). Are you talking about custom metrics the application may provide? Like how long a certain function is taking or size of a queue? In that case you can add a JSON endpoint in the format Prometheus accepts, add a tag to your pod and then configure Prometheus to scrape that tag. Now you will see the metrics in Prometheus and be able to graph them in Grafana. If you mean container metrics like CPU and Ram, most Prometheus installations on Kubernetes will handle that for you.
    – Levi
    Commented Feb 24, 2019 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.