Add more static instance labels in prometheus config

Question

I'm adding serves to my prometheus config like this:

  - job_name: cluster1
    static_configs:
      - targets:
        - server1.example.com
        - server2.example.com
        - server3.example.com

but now I would like to label the instances with additional static labels, like the server1 and server2 with "database" and server3 with "mail" and maybe server1 additionally with "mysql" and server2 with "postgres".

I would then like to use this labels in grafana to filter on like here:

free_space_in_bytes{job=~'$job', server_type='database', database_type='mysql' }

Is this achievable within the prometheus config? Filtering on the jobname or instance is indeed possible, but I would like to extend this to more labels.

Answer 1

If I'm understanding your question, the answer comes straight out of the docs for static_configs:

A static_config allows specifying a list of targets and a common label set for them. It is the canonical way to specify static targets in a scrape configuration.

So,

 - job_name: cluster1
    static_configs:
      - targets:
        - server1.example.com
        - server2.example.com
        - server3.example.com
        labels:
          database: mysql
          some_carefully_chosen_property: its_value_for_these_targets

If you want to play around with a prometheus.yml in your local directory, to check syntax, you can run

docker run -it --rm --name prometheus_configcheck \
  -v ${PWD}/prometheus.yml:/prometheus.yml \
  --entrypoint /bin/promtool \
  prom/prometheus:v2.40.6 \
  check config /prometheus.yml

if you don't have prometheus installed locally, or

promtool check config prometheus.yml

if you do, and it came with promtool, which it should have.

However Please do not do this in general and certainly not until you fully understand the issue of cardinality, and why high cardinality will overload your system, and is unnecessary in light of PromQL. You can destroy your setup by choosing either a) too many labels or b) labels with a high or non-bounded set of values, since each combination of labels is stored as a separate metric stream. This article explains it well https://grafana.com/blog/2020/08/27/the-concise-guide-to-labels-in-loki/ (it's about Loki, but labels work in the same way in Prometheus).

Also, your examples for your label values indicate that you may not have identified what property (key) you're talking about. e.g. is it server_type? Saying that your label is mysql doesn't make sense; the only values would then be true or false. Try to find out what properties really describe your infrastructure, then don't label things with them, and use them in PromQL queries instead, or add them sparingly as labels.