Enable ingress-nginx Prometheus monitoring without Helm

Question

A few weeks ago, I installed ingress-nginx in my Kubernetes cluster. Now, I realize that was a mistake because that did not come with Prometheus monitoring enabled automatically, and even worse, it doesn't use Helm. So now my ingress-nginx-controller is all set up and working, but I am unable to change settings using Helm to create the ServiceMonitor and scrape metrics from it.

I've been searching everywhere I possibly can for this and I can't find any documentation on how to get Prometheus to monitor ingress-nginx without using Helm. I understand I need to make a ServiceMonitor resource, but I can't figure out how to make one specifically for ingress-nginx because every solution I find online pretty much boils down to: "oh just use Helm bro".

I attempted to create my own ServiceMonitor resource like so:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: ingress-nginx
    release: kube-prometheus-stack
  name: ingress-nginx-metrics
  namespace: ingress-nginx
spec:
  endpoints:
    - interval: 30s
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx-controller

It doesn't work, but I have no clue if this is correct or not because again, absolutely no examples online. It seems like everyone is just expected to use Helm to do all the work for you. Does my ServiceMonitor need to be in the kube-prometheus-stack namespace, the namespace of the service being monitored, or does it not matter?

• My ingress-nginx-controller deployment is in the ingress-nginx namespace.
• kube-prometheus-stack is installed in the kube-prometheus-stack namespace.

Can someone point me in the right direction for getting Prometheus to discover the ingress controller without Helm?

Edit: I moved it to the kube-prometheus-stack namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the /metrics path in the endpoint? I've tried with and without and it doesn't seem to change anything.

Edit: I created a ClusterIP service pointing to the prometheus port on my ingress-nginx-controller and I am able to port forward that and see the /metrics route which shows the Prometheus logs. Pointing my ServiceMonitor directly at this new ClusterIP service doesn't seem to be doing anything either.

Do I need to point the ServiceMonitor directly at the ingress-nginx-controller deployment? Or does it need to point at a service? I've tried both and didn't see any other results. I can still see ingress-nginx in the Prometheus Service Discovery screen, but 0/8 active targets.

Edit: Here's what I have in my resources now:

NGINX Controller (relevant items only):

kind: Deployment
apiVersion: apps/v1
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  labels:
    app: ingress-nginx-controller
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
  annotations:
    prometheus.io/port: '10254'
    prometheus.io/scrape: 'true'
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      ports:
        - name: http
          containerPort: 80
          protocol: TCP
        - name: https
          containerPort: 443
          protocol: TCP
        - name: webhook
          containerPort: 8443
          protocol: TCP
        - name: prometheus
          containerPort: 10254
          protocol: TCP

NGINX metrics service:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx-metrics
  namespace: ingress-nginx
  labels:
    app: ingress-nginx-metrics
spec:
  ports:
    - name: prometheus
      protocol: TCP
      port: 10254
      targetPort: prometheus
  selector:
    app: ingress-nginx-controller
  type: ClusterIP

NGINX Service Monitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ingress-nginx-metrics
  namespace: kube-prometheus-stack
  labels:
    app: ingress-nginx-metrics
    release: kube-prometheus-stack
spec:
  endpoints:
    - interval: 30s
      path: /metrics
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx-metrics

I cannot port-forward to the metrics service I created, the connection times out.

This is what I'm seeing in Prometheus:

Answer 1

Your ServiceMonitor seems to be correct with the exception that you have defined no path under spec.endpoints.

You also need to define the port named prometheus under the ingress-nginx service or deployment.

ServiceMonitors can be debugged this way:-

1. Check if the ServiceMonitor created is being read by the Prometheus:- Look at the /targets URL. (There should be a target in 0/0 state at least and use the old Prometheus UI) If not, that means the ServiceMonitor itself is not being picked up by Prometheus.I suggest looking into the following configuration in your kube-prometheus-stack configuration.

       serviceMonitorSelectorNilUsesHelmValues: false  # is true by default
       serviceMonitorSelector: {} #labels which help Prom identify which servicemonitors to pick up
       serviceMonitorNamespaceSelector: {} # If this is empty then all servicemonitors in any namespace will be picked up.
   

   The default ServiceMonitor has the Helm metadata attached to it which is used by the Prometheus Operator to filter/choose the ServiceMonitors to monitor. Setting serviceMonitorSelectorNilUsesHelmValues:false will ignore any such selection.

2. If the ServiceMonitor is visible in targets but there are no targets.:- In this case the issue lies between the ServiceMonitor and the pods it is trying to scrape.Check if the ports you mentioned are accessible and the pods fulfill the selectors mentioned (all selectors.matchLabels must match).

My advice would be to start another dummy ServiceMonitor by following this and then modifying the ServiceMonitor one step at a time till it starts monitoring the pods

Edit:-

Moving all servicemonitors to the kube-prometheus-stack namespace is not a good practice. Multiple Helm charts generate their own ServiceMonitor objects which are applied in their own namespace.Enabling the prometheus operator to monitor all namespaces for servicemonitors is a better choice.

For nginx side (ports and label would vary based on your implementations, update the same in the yaml below.):-

1. Add the port in the nginx deployment

    - containerPort: 10254
      name: prometheus
      protocol: TCP
   

2. Create a service to map to the monitoring port.

   apiVersion: v1
   kind: Service
   metadata:
     labels: [] # labels of your choice
     name: {} # correct name
   spec:
     clusterIP: ""
     ports:
       - name: prometheus
         port: 9913
         targetPort: prometheus
     selector: <all labels in your pods should match>
     type: ClusterIP
   
   

3. Create the ServiceMonitor

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
       annotations: [] # as required.
       name: ingress-nginx-monitoring
   spec:
     endpoints:
     - interval: 30s
       path: /metrics
       port: prometheus
     namespaceSelector:
       matchNames:
       - <namespace where nginx is deployed>
     selector:
       matchLabels: [] # all labels in the given in the service above should match.
   

Answer 2

The service monitor seems right, only thing is that the service monitor should be in the namespace of the Prometheus stack.

Try the below service monitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: ingress-nginx
    release: kube-prometheus-stack
  name: ingress-nginx-metrics
  namespace: kube-prometheus-stack
spec:
  endpoints:
    - interval: 30s
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx

Also when you do, try port-forwarding the Prometheus

 kubectl port-forward svc/<SERVICE_NAME_OF_PROMETHEUS> -n kube-prometheus-stack 9090:9090

and check in Targets and Service Discovery options in Prometheus. You can see the service monitor in Service Discovery and exposed metrics in Targets.