12

As far as I know, Prometheus doesn't mind high-cardinality data. What Prometheus doesn't like is high-cardinality labels. Let's start with Prometheus official documentation, it gives a good high-level explanation why: CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase ...


10

The CPUThrottlingHigh is an alert created by the kubernetes-mixin project. There is an open issue (#108) to discuss this alert. I suggest that you read all the comments on this issue to better understand the problem. In short, the problem is: When working with low CPU limits, spiky workloads can have low averages and still be being throttled. Also, take a ...


8

Prometheus use a timeseries databases with vacuum, the documentation gives some maths to plan your disk consumption: On average, Prometheus uses only around 1-2 bytes per sample. Thus, to plan the capacity of a Prometheus server, you can use the rough formula: needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample ...


6

Yes, you are interpreting that correctly. When you express your SLO as a percentage, you should express your error rate as a percentage too. So, in you example your error rate of 1 is 100%. That would make the equation: (100-95/100) * 1 hour * 1 = 0.05 hours = 3 minutes The period in that equation is the reporting period that you have chosen for your SLO. ...


5

I think your can do some kind of alerting on a metric rate with something like this: ALERT DropInMetricsFromExporter IF rate(<metric_name>[1m]) == 0 FOR 3m ANNOTATIONS { summary = "Rate of metrics is 0 {{ $labels.<your_label> }}", description = "Rate of metric dropped, actually: {{ $value }}%", } The main idea is to alert whenever ...


4

The metrics for deleted pods still exists on the Prometheus Time Series Database. The problem is that the Grafana dashboards imported from kubernetes-mixin create the pod list when loading the dashboards. To change this in Grafana, open the Variables menu, then change $pod variable to refresh On Time Range Change: Go back to the panels and change the time ...


3

I don't think there's an ideal solution at all. However, there may be a preferred one, depending on circumstances. There're several factors which I think one has to consider: How cheap/expensinve a dedicated instance is? What are security concerns/considerations? What is expected volume of data from every environment? This table can resume them : | ...


3

The important part of the blog post you link to isn't testing the syntax of the YAML file; it's testing the semantics of the Prometheus configuration. The core configuration in that post includes a fragment: - alert: MyAlert expr: avg without(instance)(up) < 0.75 for: 2m labels: severity: page How do you know this is the right configuration? ...


3

You just have to use the hostname on your docker compose file like this: hostname: whatever-{{.Node.Hostname}} Note that you have to prepend something to the hostname to avoid hostnames duplicates.


3

It's possible but after doing many replatforming projects for clients I can't say I would advise it. Have you seen kube-prometheus? It's a one-shot install that will get you 90% of where you need to be for your cluster. The only way I can advise it would be if you're going to run 5 or more clusters and want central monitoring, but I'm a bigger fan of having ...


3

What about Name = "prometheus", Value = ".*". That is something that worked for me. An alternative can be Name = "severity", Value = ".*" also.


2

Have a look to solutions implementing a service mesh. For example ISTIO, Consul, Ambasador + Envoy...


2

Weave Scope is an observability tool that provides a dashboard that shows all network connections between pods. https://github.com/weaveworks/scope


2

Have a look at the Summary Display plugin. Some more details about it (= quote from the linked page): ... allows an easy and fully customized build report display. Features Allow a rich summary report visible from both project and build page Reports must be written in an XML format according to the syntax described in section Syntax Description This plugin ...


2

For Application specific calls you need to use a Prometheus client and create the metrics yourself. The Prometheus client will allow you to create a /metrics endpoint where you can output the current metric values which Prometheus will scrape. For metrics that are not unique to your application, you can use Prometheus exporters that will run on either the ...


2

The best way to deploy prometheus to kubernetes is with the helm chart: https://github.com/helm/charts/tree/master/stable/prometheus If you haven't used helm yet, its a very simple install to your laptop/etc and the cluster (and the newest version doesn't even require a cluster side install). It takes like 30 seconds. It's a package manager for ...


2

To calculate disk space required by Prometheus v2.20 in bytes, use the query: retention_time_seconds * rate(prometheus_tsdb_head_samples_appended_total[2h]) * (rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[2h]) / rate(prometheus_tsdb_compaction_chunk_samples_sum[2h])) Where retention_time_seconds is the value you've configured for --storage.tsdb....


2

I think your math may be off. If you use the ~18600 number you found I get a very different result from 68TiB: 2,592,000 (seconds) * 18600 (samples/second) * 1.3 (bytes/sample) = 62,674,560,000 (bytes). This is ~62.675 Gigabytes (divide the digital storage value by 1e+9 ) and I'm guessing a more reasonable number for your infrastructure.


2

Answering my own question (after a few days of research). I have yet to test this but https://github.com/Cleafy/promqueen seems to be designed for this purpose: PromQueen made possible to record prometheus metrics offline. PromQueen can, > therefore, backfill the recorded data inside a native prometheus database. PromQueen is composed of two primary ...


2

After researching this, i've found an open issue on grafana's github. At the time of writing its still in the backlog.


1

In the end solved it myself, I had to just define jira.summary and description respectively and copy/paste the contents from the slack integration in the given fields. {{ define "jira.summary" }}] {{ .Alerts.Firing | len }} {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }} {{- if gt (len .CommonLabels) (len .GroupLabels) -}} {{" "}}( ...


1

You should use https://github.com/helm/charts/tree/master/stable/prometheus-operator helm chart , also you can write Grafana dashboard script that would create dashboard dynamically , can be update as well later on.


1

label_values(kube_node_role, role) sum(kube_pod_container_resource_requests_cpu_cores * on (node) group_left (role) kube_node_role{role="$node_role"}) kube_node_role out of kube-state-metrics can give you a relationship to join


1

You covered the obvious security issue, use HTTPS for all your connections. While I won't try to tell you how to implement your environment, I will share with you the design in which I am rolling out at this time ... Grafana with Prometheus Federated cluster consuming data from Prometheus Servers deployed in each product. Thus, my data storage needs for ...


1

After trying many things, I came to know that I needed to expose the port of my application's container in order to let Prometheus or other deployments to know. After exposing the port, I could see my application under targets on Prometheus and I could scrape all JMX and JVM metrics. Hope this would help someone in future...


1

To piggy back off of @eduardo-baitello answer, A third option is to increase the CPUThrottlingPercent config here


1

As we can see from kube-state-metrics/docs/horizontalpodautoscaler-metrics.md We just need to have kube-state-metrics along with HPA to start getting metrics in prometheus.


1

Do you use grafana for the dashboard? If so, you can customize with the prometheus plugin for Jenkins See also: https://piotrminkowski.wordpress.com/2017/08/29/visualizing-jenkins-pipeline-results-in-grafana/


1

Sadly nobody answered, so I will provide feedback about how i did it so far. Graphite was not a good solution for me due to its retention policy. Performance tests sent data to the database every millisecond, and due to retention Graphite did not store all the results but only the last one per the period, meaning if you sent 100 data points in the last ...


1

Another option is to use a trace-based monitoring tool, eg something OpenTracing compatible. These will show requests as they flow through the entire system, allowing you to visualize dependencies.


Only top voted, non community-wiki answers of a minimum length are eligible