Initial Configuration: Grafana and Prometheus inside docker and then AWS

Question

I'm a bit of a newbie with this, but I haven't been able to find an explanation. Here's my issue. I'm trying to get Prometheus and Grafana configured on an AWS Fargate instance. I'm testing inside a Docker instance on my desktop. Both environments were coming up, but Grafana was reporting no data.

My Datasource:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: direct
    orgId: 1
    url: http://localhost:9090
    isDefault: true

I started the container with this:

docker run -d \
    --name prometheus \
    --hostname prometheus \
    -p 8110:22 \
    -p 9091:9091 \
    prometheus:latest

When I couldn't figure out why Grafana reported no data, I restarted the container, adding -p 9090:9090 so that I could access Prometheus's graphing interface. And suddenly Grafana shows data.

Which has me deeply puzzled. I don't know how Grafana works under the hood, and so far I haven't found an explanation. So I'm wondering why port-mapping fixed the problem.

Do I need to configure Grafana to access Prometheus via an externally-resolved URL (that is, one that resolves in my browser, not from Grafana itself)?

I hadn't planned on exposing Prometheus itself to the world, but maybe I have to.

Additional Info: Within AWS, my Grafana charts all have a red warning indicator for each chart, and the rollover says "Not allowed to request resource". I haven't found any docs yet on what can cause this. Still digging.

Answer 1

Okay, it took time and help from someone else. I was going down the path of installing nginx, but that won't end up being necessary.

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
    access: proxy
    orgId: 1
    url: http://localhost:9090
    isDefault: true

This is the end change. originally I had access: direct. As I understand it, that means access to the metrics is via the browser. Switching it to proxy means that Grafana will look up the data. Which means the URL doesn't need to be externally-useful.

To a newbie, none of this was obvious, so maybe my two days of pain will help someone out in the future.