Scraping different Prometheus metrics at different rates

Question

I am running a Tendermint network. I am using Prometheus to track various metrics, and to track node health. For performance metrics such as tendermint_consensus_total_txs, I want updates every second. For health metrics such as go_gc_duration_seconds_sum, I only want updates every minute.

How can I configure Prometheus to scrape metric A every second and metric B every minute? Or alternatively how can I configure it to only keep every 60th sample of metric B?

Frequently updating performance metrics is important for business reasons, but I need to avoid ballooning my storage costs by storing lots of samples for metrics I don't need that frequently.

Answer 1

You can define a different scrape_interval in the scrape config of two different jobs: a job for slow polling and one for fast polling.

If your metrics are on different endpoints you can simply adjust which endpoint should be scraped more frequently than the other.

If your metrics are on the same endpoint, you can define the same endpoint in both job and use the metric_relabel_configs of your target to adapt the metrics you want to ingest:

On the fast job, keep only the metrics you want to keep

metric_relabel_configs:
- source_labels: [__name__]
  regex: go_gc_duration_seconds_sum
  action: keep

If the duplication of metrics between the jobs is an issue, you can drop them in the slow job:

metric_relabel_configs:
- source_labels: [__name__]
  regex: go_gc_duration_seconds_sum
  action: drop

And optionally rename the job label in the fast job to match the slow job (exercise left to the reader).