Prometheus and Azure WebApps: How to monitor multiple instances with prometheus

Question

I have an application that has two instances running and each request runs on a different machine.

I am monitoring an event with a counter in my application and each instance runs it's own counter using prometheus-net. It would be great if prometheus-net was using a caching mechanism other that in-memory, maybe a redis, so whenever prometheus scrap /metrics, it gets the same data each time.

Unfortunately, each scrap request from prometheus runs on a different machine and it's like I have two counters.

I need a way to tell to prometheus to scrap two urls and I will find a way to combine them.

Unfortunately, the way to access directly the web app instance is by using parameters.

The urls looks like that:

https://example.com/metrics?instance=dba246887a8c3c07e1a9cc331d7372c43937e6acd56073246ffdf0e95597480c
https://example.com/metrics?instance=be007691832f4002a86cf0148768a3b039304e9b1fa7ae91a032b0beddc152b2

That way I cannot use the following code since I cannot add full url here.

 static_configs:
      - targets: ['example.com']

I tried the following but it didn't created two 'instances' at the label instance

- job_name: 'example'
    scrape_interval: 30s
    params:
      -instance: ['372c43937e6acd56073246ffdf0e95597480cdba24688cc331d77a8c3c07e1a9','be007691832f4002a86cf0148768a3b039304e9b1fa7ae91a032b0beddc152b2']
    static_configs:
      - targets: ['example.com']
    metrics_path: /metrics
    scheme: https

Answer 1

You can do this similarly to how it's done in blackbox_exporter: list identifying information of your targets in the target block, and then use relabeling to replace scraping address with address of your actual exporter and to place targets into parameters of scraping request.

scrape_configs:
  - job_name: 'myjob'
    static_configs:
      - targets:  # Targets to probe using parameters
        - 372c43937e6acd56073246ffdf0e95597480cdba24688cc331d77a8c3c07e1a9   
        - be007691832f4002a86cf0148768a3b039304e9b1fa7ae91a032b0beddc152b2
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_instance
      - source_labels: [__param_instance]
        target_label: instance
      - target_label: __address__
        replacement: example.com  # The exporter's real address

please notice here: __param_instance is a label that will be used during scraping as URL parameter with name instance.

Here replacement config does:

1. puts ids from target list into parameter instance,
2. puts content of that parameter into final label instance (otherwise content of __address__ will be used),
3. puts address of actual target into label __address__ to be used bu scraping.