A few weeks ago, I installed ingress-nginx
in my Kubernetes cluster. Now, I realize that was a mistake because that did not come with Prometheus monitoring enabled automatically, and even worse, it doesn't use Helm. So now my ingress-nginx-controller
is all set up and working, but I am unable to change settings using Helm to create the ServiceMonitor
and scrape metrics from it.
I've been searching everywhere I possibly can for this and I can't find any documentation on how to get Prometheus to monitor ingress-nginx
without using Helm. I understand I need to make a ServiceMonitor
resource, but I can't figure out how to make one specifically for ingress-nginx
because every solution I find online pretty much boils down to: "oh just use Helm bro".
I attempted to create my own ServiceMonitor
resource like so:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app: ingress-nginx
release: kube-prometheus-stack
name: ingress-nginx-metrics
namespace: ingress-nginx
spec:
endpoints:
- interval: 30s
port: prometheus
namespaceSelector:
matchNames:
- ingress-nginx
selector:
matchLabels:
app: ingress-nginx-controller
It doesn't work, but I have no clue if this is correct or not because again, absolutely no examples online. It seems like everyone is just expected to use Helm to do all the work for you. Does my ServiceMonitor
need to be in the kube-prometheus-stack
namespace, the namespace of the service being monitored, or does it not matter?
- My
ingress-nginx-controller
deployment is in theingress-nginx
namespace. kube-prometheus-stack
is installed in thekube-prometheus-stack
namespace.
Can someone point me in the right direction for getting Prometheus to discover the ingress controller without Helm?
Edit: I moved it to the kube-prometheus-stack namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the /metrics
path in the endpoint? I've tried with and without and it doesn't seem to change anything.
Edit: I created a ClusterIP
service pointing to the prometheus
port on my ingress-nginx-controller
and I am able to port forward that and see the /metrics
route which shows the Prometheus logs. Pointing my ServiceMonitor
directly at this new ClusterIP
service doesn't seem to be doing anything either.
Do I need to point the ServiceMonitor
directly at the ingress-nginx-controller
deployment? Or does it need to point at a service? I've tried both and didn't see any other results. I can still see ingress-nginx
in the Prometheus Service Discovery screen, but 0/8 active targets.
Edit: Here's what I have in my resources now:
NGINX Controller (relevant items only):
kind: Deployment
apiVersion: apps/v1
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
labels:
app: ingress-nginx-controller
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/version: 0.44.0
helm.sh/chart: ingress-nginx-3.23.0
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
- name: webhook
containerPort: 8443
protocol: TCP
- name: prometheus
containerPort: 10254
protocol: TCP
NGINX metrics service:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx-metrics
namespace: ingress-nginx
labels:
app: ingress-nginx-metrics
spec:
ports:
- name: prometheus
protocol: TCP
port: 10254
targetPort: prometheus
selector:
app: ingress-nginx-controller
type: ClusterIP
NGINX Service Monitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: ingress-nginx-metrics
namespace: kube-prometheus-stack
labels:
app: ingress-nginx-metrics
release: kube-prometheus-stack
spec:
endpoints:
- interval: 30s
path: /metrics
port: prometheus
namespaceSelector:
matchNames:
- ingress-nginx
selector:
matchLabels:
app: ingress-nginx-metrics
I cannot port-forward to the metrics service I created, the connection times out.
This is what I'm seeing in Prometheus: