I have spring boot powered microservices deployed in my local kubernetes cluster. The microservices are using micrometer and prometheus registry but due to our company policy the actuator is available on another port:
- 8080 for "business" http requests
- 8081/manage for actuator. So, I can access
http://host:8081/manage/prometheusand see the metrics when running the process locally (without kubernetes).
Now, I'm a beginner in Prometheus and have a rather limited knowledge in kubernetes (I'm coming with a Java developer background).
I've created a POD with my application and succesfully run it in kubernetes. It works and I can access it (for 8080 I've created a service to map the ports) and I can execute "business" level http requests it from the same PC.
But I haven't find any examples of adding a prometheus into the picture. Prometheus is supposed to be deployed in the same kubernetes cluster just as another pod. So I've started with:
FROM @docker.registry.address@/prom/prometheus:v2.15.2 COPY entrypoint.sh / USER root RUN chmod 755 /entrypoint.sh ADD ./prometheus.yml /etc/prometheus/ ENTRYPOINT ["/entrypoint.sh"]
entrypoint.sh looks like:
#!/bin/sh echo "About to run prometheus" /bin/prometheus --config.file=/etc/prometheus/prometheus.yml \ --storage.tsdb.path=/prometheus \ --storage.tsdb.retention.time=3d \ --web.console.libraries=/etc/prometheus/console_libraries \ --web.console.templates=/etc/prometheus/consoles
My question is about how exactly I should define
prometheus.yml so that it will get the metrics from my spring boot pod (and other microservices that I have, all spring boot driven with the same actuator setup).
I've started with (
global: scrape_interval: 10s evaluation_interval: 10s scrape_configs: - job_name: 'prometheus' metrics_path: /manage/prometheus kubernetes_sd_configs: - role: pod bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_pod_label_app] action: keep regex: sample-pod-app(.*)|another-pod-app(.*)
But apparently it doesn't work, so I've asking for the advices:
- If someone has a working example it would be the best :)
- Intuitively I understand I need to specify the port mapping for my
8081port but I'm not exactly know how
- Since prometheus is supposed to run on another port, am I supposed to expose a kubernetes service for port 8081 at the kubernetes level?
- Do I need any security related resources in kubernetes to be defined?
As a side note, at this point I don't care about scalability issues, I believe one prometheus server will do the job, but I'll have to add Grafana into the picture.
P.S. Originally asked this question on SO but was suggested to ask this here as well.