The metrics for deleted pods still exists on the Prometheus Time Series Database. The problem is that the Grafana dashboards imported from kubernetes-mixin create the pod list when loading the dashboards.
To change this in Grafana, open the Variables menu, then change $pod variable to refresh On Time Range Change:
Go back to the panels and change the time ...
It's possible but after doing many replatforming projects for clients I can't say I would advise it. Have you seen kube-prometheus? It's a one-shot install that will get you 90% of where you need to be for your cluster.
The only way I can advise it would be if you're going to run 5 or more clusters and want central monitoring, but I'm a bigger fan of having ...
I don't think there's an ideal solution at all. However, there may be a preferred one, depending on circumstances.
There're several factors which I think one has to consider:
How cheap/expensinve a dedicated instance is?
What are security concerns/considerations?
What is expected volume of data from every environment?
This table can resume them :
You can copy the actual logs from /var/lib/docker/containers/<container id>/<container id>-json.log to archive them then clear the logs manually (since you have only 2 services, influxdb and grafana). The containers won't stop with this procedure.
There is a full and clear answer on this StackOverflow question about clearing container logs. It ...
You can update the daemon.json config to specify your normal syslog driver (assuming linux) to use the normal syslog logrotation method for keeping log files or shipping off-box
This is more of an opinion based question but let me try to answer it.
installing Prometheus itself in the very same k8s cluster (as another running set of pods) is a normal thing.Am I right or just got a wrong idea from my research?
Considering good practices, shouldn't the monitoring tool be outside of the system that you actually intend to monitor?
In terms of embedding in an external page, you have some options out of the box with both Kibana (docs here) and Grafana (docs here).
If I understand correctly, you are trying to generate realistic clickable thumbnails - in that case, you should try Grafana first since it better supports server-side image rendering
I think to have found a suitable grafana plugin that can activate/deacttivate an actuator by posting a value with a call api rest. The name of this plugin that can be easily donwloaded and istalled in the grafana is cloudspout button panel plugin. and can be found at grafana.com/grafana/plugins/cloudspout-button-panel
I've found out that it is a possible and fine approach. Prometheus support different methods of service discovery, this is explained in the docs: Configuration. As I understood the Loki way of work, it just uses Promtail which sends logs from services, so there is no need of service discovery. An example of Promtail installation is here. Also, more details ...
I haven't found any way to do that, so I created a simple CLI based on Python, which can assist with the task - make sure you don’t scrape metrics in Prometheus, which you don’t present in Grafana dashboards.
frigga - https://github.com/unfor19/frigga
Okay, it took time and help from someone else. I was going down the path of installing nginx, but that won't end up being necessary.
- name: Prometheus
# <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
Grafana was not really built to view logs from the get go.
They however introduced this feature in v6.0 via the Explore tab
This can be leveraged in 2 ways.
Via Elasticsearch data source (in v6.3+)
Check the Elasticsearch docs for more.
If you want, you can still integrate this with Cloudwatch. Check out the log subscirber-consumer solution & this
You should use https://github.com/helm/charts/tree/master/stable/prometheus-operator helm chart , also you can write Grafana dashboard script that would create dashboard dynamically , can be update as well later on.
You covered the obvious security issue, use HTTPS for all your connections.
While I won't try to tell you how to implement your environment, I will share with you the design in which I am rolling out at this time ...
Grafana with Prometheus Federated cluster consuming data from Prometheus Servers deployed in each product. Thus, my data storage needs for ...
After trying many things, I came to know that I needed to expose the port of my application's container in order to let Prometheus or other deployments to know. After exposing the port, I could see my application under targets on Prometheus and I could scrape all JMX and JVM metrics. Hope this would help someone in future...
Sadly nobody answered, so I will provide feedback about how i did it so far.
Graphite was not a good solution for me due to its retention policy. Performance tests sent data to the database every millisecond, and due to retention Graphite did not store all the results but only the last one per the period, meaning if you sent 100 data points in the last ...