I have spring boot powered microservices deployed in my local kubernetes cluster. The microservices are using micrometer and prometheus registry but due to our company policy the actuator is available on another port:
- 8080 for "business" http requests
- 8081/manage for actuator. So, I can access
http://host:8081/manage/prometheus
and see the metrics when running the process locally (without kubernetes).
Now, I'm a beginner in Prometheus and have a rather limited knowledge in kubernetes (I'm coming with a Java developer background).
I've created a POD with my application and succesfully run it in kubernetes. It works and I can access it (for 8080 I've created a service to map the ports) and I can execute "business" level http requests it from the same PC.
But I haven't find any examples of adding a prometheus into the picture. Prometheus is supposed to be deployed in the same kubernetes cluster just as another pod. So I've started with:
FROM @docker.registry.address@/prom/prometheus:v2.15.2
COPY entrypoint.sh /
USER root
RUN chmod 755 /entrypoint.sh
ADD ./prometheus.yml /etc/prometheus/
ENTRYPOINT ["/entrypoint.sh"]
entrypoint.sh
looks like:
#!/bin/sh
echo "About to run prometheus"
/bin/prometheus --config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/prometheus \
--storage.tsdb.retention.time=3d \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.console.templates=/etc/prometheus/consoles
My question is about how exactly I should define prometheus.yml
so that it will get the metrics from my spring boot pod (and other microservices that I have, all spring boot driven with the same actuator setup).
I've started with (prometheus.yml
):
global:
scrape_interval: 10s
evaluation_interval: 10s
scrape_configs:
- job_name: 'prometheus'
metrics_path: /manage/prometheus
kubernetes_sd_configs:
- role: pod
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
action: keep
regex: sample-pod-app(.*)|another-pod-app(.*)
But apparently it doesn't work, so I've asking for the advices:
- If someone has a working example it would be the best :)
- Intuitively I understand I need to specify the port mapping for my
8081
port but I'm not exactly know how - Since prometheus is supposed to run on another port, am I supposed to expose a kubernetes service for port 8081 at the kubernetes level?
- Do I need any security related resources in kubernetes to be defined?
As a side note, at this point I don't care about scalability issues, I believe one prometheus server will do the job, but I'll have to add Grafana into the picture.
P.S. Originally asked this question on SO but was suggested to ask this here as well.