A few weeks ago, I installed
ingress-nginx in my Kubernetes cluster. Now, I realize that was a mistake because that did not come with Prometheus monitoring enabled automatically, and even worse, it doesn't use Helm. So now my
ingress-nginx-controller is all set up and working, but I am unable to change settings using Helm to create the
ServiceMonitor and scrape metrics from it.
I've been searching everywhere I possibly can for this and I can't find any documentation on how to get Prometheus to monitor
ingress-nginx without using Helm. I understand I need to make a
ServiceMonitor resource, but I can't figure out how to make one specifically for
ingress-nginx because every solution I find online pretty much boils down to: "oh just use Helm bro".
I attempted to create my own
ServiceMonitor resource like so:
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app: ingress-nginx release: kube-prometheus-stack name: ingress-nginx-metrics namespace: ingress-nginx spec: endpoints: - interval: 30s port: prometheus namespaceSelector: matchNames: - ingress-nginx selector: matchLabels: app: ingress-nginx-controller
It doesn't work, but I have no clue if this is correct or not because again, absolutely no examples online. It seems like everyone is just expected to use Helm to do all the work for you. Does my
ServiceMonitor need to be in the
kube-prometheus-stack namespace, the namespace of the service being monitored, or does it not matter?
ingress-nginx-controllerdeployment is in the
kube-prometheus-stackis installed in the
Can someone point me in the right direction for getting Prometheus to discover the ingress controller without Helm?
Edit: I moved it to the kube-prometheus-stack namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the
/metrics path in the endpoint? I've tried with and without and it doesn't seem to change anything.
Edit: I created a
ClusterIP service pointing to the
prometheus port on my
ingress-nginx-controller and I am able to port forward that and see the
/metrics route which shows the Prometheus logs. Pointing my
ServiceMonitor directly at this new
ClusterIP service doesn't seem to be doing anything either.
Do I need to point the
ServiceMonitor directly at the
ingress-nginx-controller deployment? Or does it need to point at a service? I've tried both and didn't see any other results. I can still see
ingress-nginx in the Prometheus Service Discovery screen, but 0/8 active targets.
Edit: Here's what I have in my resources now:
NGINX Controller (relevant items only):
kind: Deployment apiVersion: apps/v1 metadata: name: ingress-nginx-controller namespace: ingress-nginx labels: app: ingress-nginx-controller app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: ingress-nginx app.kubernetes.io/version: 0.44.0 helm.sh/chart: ingress-nginx-3.23.0 annotations: prometheus.io/port: '10254' prometheus.io/scrape: 'true' spec: replicas: 1 selector: matchLabels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx template: metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx annotations: prometheus.io/port: '10254' prometheus.io/scrape: 'true' spec: ports: - name: http containerPort: 80 protocol: TCP - name: https containerPort: 443 protocol: TCP - name: webhook containerPort: 8443 protocol: TCP - name: prometheus containerPort: 10254 protocol: TCP
NGINX metrics service:
kind: Service apiVersion: v1 metadata: name: ingress-nginx-metrics namespace: ingress-nginx labels: app: ingress-nginx-metrics spec: ports: - name: prometheus protocol: TCP port: 10254 targetPort: prometheus selector: app: ingress-nginx-controller type: ClusterIP
NGINX Service Monitor:
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: ingress-nginx-metrics namespace: kube-prometheus-stack labels: app: ingress-nginx-metrics release: kube-prometheus-stack spec: endpoints: - interval: 30s path: /metrics port: prometheus namespaceSelector: matchNames: - ingress-nginx selector: matchLabels: app: ingress-nginx-metrics
I cannot port-forward to the metrics service I created, the connection times out.
This is what I'm seeing in Prometheus: