1

A few weeks ago, I installed ingress-nginx in my Kubernetes cluster. Now, I realize that was a mistake because that did not come with Prometheus monitoring enabled automatically, and even worse, it doesn't use Helm. So now my ingress-nginx-controller is all set up and working, but I am unable to change settings using Helm to create the ServiceMonitor and scrape metrics from it.

I've been searching everywhere I possibly can for this and I can't find any documentation on how to get Prometheus to monitor ingress-nginx without using Helm. I understand I need to make a ServiceMonitor resource, but I can't figure out how to make one specifically for ingress-nginx because every solution I find online pretty much boils down to: "oh just use Helm bro".

I attempted to create my own ServiceMonitor resource like so:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: ingress-nginx
    release: kube-prometheus-stack
  name: ingress-nginx-metrics
  namespace: ingress-nginx
spec:
  endpoints:
    - interval: 30s
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx-controller

It doesn't work, but I have no clue if this is correct or not because again, absolutely no examples online. It seems like everyone is just expected to use Helm to do all the work for you. Does my ServiceMonitor need to be in the kube-prometheus-stack namespace, the namespace of the service being monitored, or does it not matter?

  • My ingress-nginx-controller deployment is in the ingress-nginx namespace.
  • kube-prometheus-stack is installed in the kube-prometheus-stack namespace.

Can someone point me in the right direction for getting Prometheus to discover the ingress controller without Helm?

Edit: I moved it to the kube-prometheus-stack namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the /metrics path in the endpoint? I've tried with and without and it doesn't seem to change anything.

Edit: I created a ClusterIP service pointing to the prometheus port on my ingress-nginx-controller and I am able to port forward that and see the /metrics route which shows the Prometheus logs. Pointing my ServiceMonitor directly at this new ClusterIP service doesn't seem to be doing anything either.

Do I need to point the ServiceMonitor directly at the ingress-nginx-controller deployment? Or does it need to point at a service? I've tried both and didn't see any other results. I can still see ingress-nginx in the Prometheus Service Discovery screen, but 0/8 active targets.

Edit: Here's what I have in my resources now:

NGINX Controller (relevant items only):

kind: Deployment
apiVersion: apps/v1
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  labels:
    app: ingress-nginx-controller
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
  annotations:
    prometheus.io/port: '10254'
    prometheus.io/scrape: 'true'
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      ports:
        - name: http
          containerPort: 80
          protocol: TCP
        - name: https
          containerPort: 443
          protocol: TCP
        - name: webhook
          containerPort: 8443
          protocol: TCP
        - name: prometheus
          containerPort: 10254
          protocol: TCP

NGINX metrics service:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx-metrics
  namespace: ingress-nginx
  labels:
    app: ingress-nginx-metrics
spec:
  ports:
    - name: prometheus
      protocol: TCP
      port: 10254
      targetPort: prometheus
  selector:
    app: ingress-nginx-controller
  type: ClusterIP

NGINX Service Monitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ingress-nginx-metrics
  namespace: kube-prometheus-stack
  labels:
    app: ingress-nginx-metrics
    release: kube-prometheus-stack
spec:
  endpoints:
    - interval: 30s
      path: /metrics
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx-metrics

I cannot port-forward to the metrics service I created, the connection times out.

This is what I'm seeing in Prometheus:

enter image description here

2 Answers 2

2

Your ServiceMonitor seems to be correct with the exception that you have defined no path under spec.endpoints.

You also need to define the port named prometheus under the ingress-nginx service or deployment.

ServiceMonitors can be debugged this way:-

  1. Check if the ServiceMonitor created is being read by the Prometheus:- Look at the /targets URL. (There should be a target in 0/0 state at least and use the old Prometheus UI) If not, that means the ServiceMonitor itself is not being picked up by Prometheus.I suggest looking into the following configuration in your kube-prometheus-stack configuration.

        serviceMonitorSelectorNilUsesHelmValues: false  # is true by default
        serviceMonitorSelector: {} #labels which help Prom identify which servicemonitors to pick up
        serviceMonitorNamespaceSelector: {} # If this is empty then all servicemonitors in any namespace will be picked up.
    

    The default ServiceMonitor has the Helm metadata attached to it which is used by the Prometheus Operator to filter/choose the ServiceMonitors to monitor. Setting serviceMonitorSelectorNilUsesHelmValues:false will ignore any such selection.

  2. If the ServiceMonitor is visible in targets but there are no targets.:- In this case the issue lies between the ServiceMonitor and the pods it is trying to scrape.Check if the ports you mentioned are accessible and the pods fulfill the selectors mentioned (all selectors.matchLabels must match).

My advice would be to start another dummy ServiceMonitor by following this and then modifying the ServiceMonitor one step at a time till it starts monitoring the pods

Edit:-

Moving all servicemonitors to the kube-prometheus-stack namespace is not a good practice. Multiple Helm charts generate their own ServiceMonitor objects which are applied in their own namespace.Enabling the prometheus operator to monitor all namespaces for servicemonitors is a better choice.

For nginx side (ports and label would vary based on your implementations, update the same in the yaml below.):-

  1. Add the port in the nginx deployment

     - containerPort: 10254
       name: prometheus
       protocol: TCP
    
  2. Create a service to map to the monitoring port.

    apiVersion: v1
    kind: Service
    metadata:
      labels: [] # labels of your choice
      name: {} # correct name
    spec:
      clusterIP: ""
      ports:
        - name: prometheus
          port: 9913
          targetPort: prometheus
      selector: <all labels in your pods should match>
      type: ClusterIP
    
    
  3. Create the ServiceMonitor

    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
        annotations: [] # as required.
        name: ingress-nginx-monitoring
    spec:
      endpoints:
      - interval: 30s
        path: /metrics
        port: prometheus
      namespaceSelector:
        matchNames:
        - <namespace where nginx is deployed>
      selector:
        matchLabels: [] # all labels in the given in the service above should match.
    

19
  • I do see the ServiceMonitor being discovered within the Prometheus UI, but I see 0/8 active targets. I do have a port named prometheus going to 9113 on the ingress-nginx-controller. I'm not sure if that's the right port or if ingress-nginx even has the metrics exposed to begin with. Jun 7, 2021 at 15:17
  • I've done some more digging, check my edits. Jun 7, 2021 at 15:49
  • If I move the ServiceMonitor out of the kube-prometheus-stack namespace, I can no longer see it in the Prometheus UI. I'd rather keep all my resources in a namespace if possible anyway. Jun 7, 2021 at 17:59
  • That should be okay. The rest of the steps would remain the same. Jun 7, 2021 at 18:07
  • I don't see the ServiceMonitor showing up again in Prometheus when I move it back to the kube-prometheus-stack namespace, it's totally gone again. I created a ClusterIP service to expose the prometheus port on the controller deployment but that doesn't seem to be finding it either, and I tested that it works by port-fowarding the service and I can confirm that I do see the metrics displayed at /metrics. Jun 7, 2021 at 18:20
0

The service monitor seems right, only thing is that the service monitor should be in the namespace of the Prometheus stack.

Try the below service monitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: ingress-nginx
    release: kube-prometheus-stack
  name: ingress-nginx-metrics
  namespace: kube-prometheus-stack
spec:
  endpoints:
    - interval: 30s
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx

Also when you do, try port-forwarding the Prometheus

 kubectl port-forward svc/<SERVICE_NAME_OF_PROMETHEUS> -n kube-prometheus-stack 9090:9090

and check in Targets and Service Discovery options in Prometheus. enter image description here You can see the service monitor in Service Discovery and exposed metrics in Targets.

2
  • Okay so I moved it to the kube-prometheus-stack namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the /metrics path in the endpoint? I've tried with and without and it doesn't seem to change anything. Jun 7, 2021 at 14:24
  • I've done some more digging, check my edits. Jun 7, 2021 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.