1

I installed prometheus-operator using kube-prometheus-stack and so far everything has worked fine, but I need to add one more service to monitoring. This is a Windows Server that is not running on the Kubernetes cluster.

I can't add the target to prometheus.yml because the storage is read-only. I saw that I could use ServiceMonitor for this purpose, but even following the steps in the official documentation, the exporter is not detected.

1 Answer 1

1

I don't know if you managed to get this working, but I had the exact same issue.

Running a K8s cluster with kube-prometheus-stack installed via Helm, and a CRD prometheus-operator. I also needed to monitor an external Windows SQL VM outside the cluster, in the same network CIDR, with the K8s cluster & SQL VM being in separate subnets.

I ended up installing the Windows-Exporter on the SQL VM. How I got Prometheus to pick up and scrape this external target is as follows:

  1. I defined a ServiceMonitor.
  2. I defined a headless Service (it has no label selector specified)
  3. I then defined a custom Endpoints object that the headless Service uses.

Note: Because we define a headless Service we need to manually create/manage the underlying Endpoints, as Services create those by default. Also, labelling your resources correctly are super important.

Here are the relevant YAML files:

  1. ServiceMonitor (SM) yaml:
  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: sql-server-metrics # This name is usually important
    namespace: monitoring
    labels:
      # The below meta labels are NB! 
      # It is used to match the SVC to SM and the EP to the SVC.
      kubernetes.io/service-name: sql-server-metrics 
      app.kubernetes.io/name: sql-server-metrics
  spec:
    endpoints:
      - port: sql
        interval: 30s
        path: /metrics
        targetPort: 9182 
    # The below selector tells SM to pick up all SVC's with these meta labels
    selector:
      matchLabels:
        kubernetes.io/service-name: sql-server-metrics
        app.kubernetes.io/name: sql-server-metrics
  1. Service (SVC) yaml:
  apiVersion: v1
  kind: Service
  metadata:
    name: sql-server-metrics
    namespace: monitoring
    labels:
      kubernetes.io/service-name: sql-server-metrics
      app.kubernetes.io/name: sql-server-metrics
  spec:
    # This can be one of 4 types, check the Service docs for K8s
    type: ExternalName
    # This can be the IP or FQDN/Partial-FQDN
    externalName: {IP_OR_HOSTNAME}
    ports:
      - name: sql
        protocol: TCP
        port: 9182
        targetPort: 9182
  1. Endpoints (EP) yaml:
  apiVersion: v1
  kind: Endpoints
  metadata:
    name: sql-server-metrics
    namespace: monitoring
    labels:
      # This is needed to match the selector-less External Service to this Endpoint
      kubernetes.io/service-name: sql-server-metrics
      app.kubernetes.io/name: sql-server-metrics
  subsets:
    - addresses:
      - ip: {VM_IP}
        hostname: {VM_HOSTNAME}
      ports:
        - name: sql
          port: 9182
          protocol: TCP

I do hope the above response helps you or anyone in the future, please do feel free to mark this as the answer or to ask for any clarity and I will respond ASAP.

Chow!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.