Using Prometheus to monitor Spring Boot Applications in Kubernetes Cluster

Question

I have spring boot powered microservices deployed in my local kubernetes cluster. The microservices are using micrometer and prometheus registry but due to our company policy the actuator is available on another port:

• 8080 for "business" http requests
• 8081/manage for actuator. So, I can access http://host:8081/manage/prometheus and see the metrics when running the process locally (without kubernetes).

Now, I'm a beginner in Prometheus and have a rather limited knowledge in kubernetes (I'm coming with a Java developer background).

I've created a POD with my application and succesfully run it in kubernetes. It works and I can access it (for 8080 I've created a service to map the ports) and I can execute "business" level http requests it from the same PC.

But I haven't find any examples of adding a prometheus into the picture. Prometheus is supposed to be deployed in the same kubernetes cluster just as another pod. So I've started with:

FROM @docker.registry.address@/prom/prometheus:v2.15.2

COPY entrypoint.sh /
USER root
RUN chmod 755 /entrypoint.sh

ADD ./prometheus.yml  /etc/prometheus/

ENTRYPOINT ["/entrypoint.sh"]

entrypoint.sh looks like:

#!/bin/sh
echo "About to run prometheus"
/bin/prometheus --config.file=/etc/prometheus/prometheus.yml \
                --storage.tsdb.path=/prometheus \
                --storage.tsdb.retention.time=3d \
                --web.console.libraries=/etc/prometheus/console_libraries \
                --web.console.templates=/etc/prometheus/consoles

My question is about how exactly I should define prometheus.yml so that it will get the metrics from my spring boot pod (and other microservices that I have, all spring boot driven with the same actuator setup).

I've started with (prometheus.yml):

global:
  scrape_interval: 10s
  evaluation_interval: 10s

scrape_configs:

  - job_name: 'prometheus'
    metrics_path: /manage/prometheus
    kubernetes_sd_configs:
      - role: pod
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token  
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: sample-pod-app(.*)|another-pod-app(.*)

But apparently it doesn't work, so I've asking for the advices:

• If someone has a working example it would be the best :)
• Intuitively I understand I need to specify the port mapping for my 8081 port but I'm not exactly know how
• Since prometheus is supposed to run on another port, am I supposed to expose a kubernetes service for port 8081 at the kubernetes level?
• Do I need any security related resources in kubernetes to be defined?

As a side note, at this point I don't care about scalability issues, I believe one prometheus server will do the job, but I'll have to add Grafana into the picture.

P.S. Originally asked this question on SO but was suggested to ask this here as well.

Answer 1

The best way to deploy prometheus to kubernetes is with the helm chart:

https://github.com/helm/charts/tree/master/stable/prometheus

If you haven't used helm yet, its a very simple install to your laptop/etc and the cluster (and the newest version doesn't even require a cluster side install). It takes like 30 seconds. It's a package manager for kubernetes and it allows you to install, upgrade, list, and delete "charts" which are basically simple wrappers around kubernetes YAML files (with some nice templating). It would even be good to wrap your spring boot apps in a helm chart =).

Once you've got prometheus running well, there is a good tutorial here on setting up actuator's prometheus endpoint in spring-boot and getting it picked up by prometheus by editing its values.yaml.

https://www.callicoder.com/spring-boot-actuator-metrics-monitoring-dashboard-prometheus-grafana/

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'spring-actuator'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s
    static_configs:
    - targets: ['HOST_IP:8080']

There's also some info at the end of the tutorial about surfacing all this in Grafana for slick visualization.