Concat multiple meta labels on prometheus for relabling

Question

How can I change the value of the label __param_target to resemble how I would interact with a Kubernetes service in-cluster, following the format: http://serviceName.namespace.svc:servicePort/path ?

I am aware of the necessary meta labels for this task, but I am unsure how to relabel it using serviceMonitor. The meta labels I intend to use are:

__scheme__
__meta_kubernetes_service_name
__meta_kubernetes_namespace
__meta_kubernetes_pod_container_port_number

the path can be hardcoded since it is not something that can be on the meta labels :) but how can it be concatenated

Answer 1

So i figured it out

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: blackbox-exporter-test
  namespace: monitoring
  labels:
    prometheus: apps
spec:
  endpoints:
  - interval: 15s
    targetPort: 3000
    path: /probe
    scrapeTimeout: 10s
    params:
      module: [http_2xx]
    relabelings:
    - sourceLabels: [__scheme__,__meta_kubernetes_service_name]
      targetLabel: __param_target
      separator: ://
    - sourceLabels: [__param_target, __meta_kubernetes_namespace]
      targetLabel: __param_target
      separator: .
    - sourceLabels: [__address__]
      targetLabel: svclabel
      replacement: "svc.cluster.local" 
    - sourceLabels: [__param_target, svclabel]
      targetLabel: __param_target
      separator: .
    - sourceLabels: [__param_target, __meta_kubernetes_pod_container_port_number]
      targetLabel: __param_target
      separator: ':'
    - sourceLabels: [__address__]
      targetLabel: path
      replacement: "zorkiness" 
    - sourceLabels: [__param_target, path]
      targetLabel: __param_target
      separator: /
    - sourceLabels: [__param_target]
      targetLabel: instance
    - targetLabel: __address__
      replacement:  prometheus-blackbox-exporter:9115
  selector:
    matchLabels:
      probe: "blackbox" # monitor the services only with this label
  namespaceSelector:
    matchNames:
      - zorki

so basically it rewrite the __probe_target to

http://serviceName.namespace.svc:servicePort/path