How to add an aditional ServiceMonitor for prometheus-operator?

Question

I installed prometheus-operator using kube-prometheus-stack and so far everything has worked fine, but I need to add one more service to monitoring. This is a Windows Server that is not running on the Kubernetes cluster.

I can't add the target to prometheus.yml because the storage is read-only. I saw that I could use ServiceMonitor for this purpose, but even following the steps in the official documentation, the exporter is not detected.

Answer 1

I don't know if you managed to get this working, but I had the exact same issue.

Running a K8s cluster with kube-prometheus-stack installed via Helm, and a CRD prometheus-operator. I also needed to monitor an external Windows SQL VM outside the cluster, in the same network CIDR, with the K8s cluster & SQL VM being in separate subnets.

I ended up installing the Windows-Exporter on the SQL VM. How I got Prometheus to pick up and scrape this external target is as follows:

1. I defined a ServiceMonitor.
2. I defined a headless Service (it has no label selector specified)
3. I then defined a custom Endpoints object that the headless Service uses.

Note: Because we define a headless Service we need to manually create/manage the underlying Endpoints, as Services create those by default. Also, labelling your resources correctly are super important.

Here are the relevant YAML files:

1. ServiceMonitor (SM) yaml:

  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: sql-server-metrics # This name is usually important
    namespace: monitoring
    labels:
      # The below meta labels are NB! 
      # It is used to match the SVC to SM and the EP to the SVC.
      kubernetes.io/service-name: sql-server-metrics 
      app.kubernetes.io/name: sql-server-metrics
  spec:
    endpoints:
      - port: sql
        interval: 30s
        path: /metrics
        targetPort: 9182 
    # The below selector tells SM to pick up all SVC's with these meta labels
    selector:
      matchLabels:
        kubernetes.io/service-name: sql-server-metrics
        app.kubernetes.io/name: sql-server-metrics

2. Service (SVC) yaml:

  apiVersion: v1
  kind: Service
  metadata:
    name: sql-server-metrics
    namespace: monitoring
    labels:
      kubernetes.io/service-name: sql-server-metrics
      app.kubernetes.io/name: sql-server-metrics
  spec:
    # This can be one of 4 types, check the Service docs for K8s
    type: ExternalName
    # This can be the IP or FQDN/Partial-FQDN
    externalName: {IP_OR_HOSTNAME}
    ports:
      - name: sql
        protocol: TCP
        port: 9182
        targetPort: 9182

3. Endpoints (EP) yaml:

  apiVersion: v1
  kind: Endpoints
  metadata:
    name: sql-server-metrics
    namespace: monitoring
    labels:
      # This is needed to match the selector-less External Service to this Endpoint
      kubernetes.io/service-name: sql-server-metrics
      app.kubernetes.io/name: sql-server-metrics
  subsets:
    - addresses:
      - ip: {VM_IP}
        hostname: {VM_HOSTNAME}
      ports:
        - name: sql
          port: 9182
          protocol: TCP

I do hope the above response helps you or anyone in the future, please do feel free to mark this as the answer or to ask for any clarity and I will respond ASAP.

Chow!