What is the Prometheus and Grafana ideal setup?

Question

I am just wondering. If I have many environments monitored via Prometheus, what will be the best configuration? The security and efficiency are already important for on-premise installations but they are much more important and unavoidable when deployed on a cloud provider.

From your experience, what is the most common and useful configuration? I see three main scenarios feel free to propose something else.

Thanks.

Solution I, Duplicate nothing

It is easy for the monitoring user because there is only one Grafana with one Datasource. However, from a security point of view, I have to open a port for each service in each environment. I will, of course, configure one reverse proxy in each environment to secure the exporters over Https and Basic authentication.

+---------+       +------------+       +- Environment A -------------+
| Grafana |       | Prometheus |       |     +-----------+           |
|         |------>|            |--+--+------>| Service 1 |           |
+---------+       +------------+  |  | |     |           |           |
                                  |  | |     +-----------+           |
                                  |  | |                             |
                                  |  | |     +-----------+           |
                                  |  +------>| Service 2 |           |
                                  |    |     |           |           |
                                  |    |     +-----------+           |
                                  |    +-----------------------------+
                                  | 
                                  |    +- Environment B -------------+
                                  |    |     +-----------+           |
                                  +--------->| Service 1 |           |
                                  |    |     |           |           |
                                  |    |     +-----------+           |
                                  |    |                             |
                                  |    |     +-----------+           |
                                  +--------->| Service 2 |           |
                                       |     |           |           |
                                       |     +-----------+           |
                                       +-----------------------------+

Solution II, Duplicate everything

This looks like a popular solution and may have many benefits from a deployment point of view. I find it quite elegant because we can have on docker-compose.yml that set up everything. But while it can be interesting for security segregation, there will be many Grafana (each with one Prometheus data source) and that may not be useful for the monitoring user.

+- Environment A ---------------------------------------------------+
| +---------+       +------------+                                  |
| | Grafana |       | Prometheus |                                  |
| |         |------>|            |                                  |
| +---------+       +------------+    +-----------+                 |
|                             |       | Service 1 |                 |
|                             +------>|           |                 |
|                             |       +-----------+                 |
|                             |                                     |
|                             |       +-----------+                 |
|                             |       | Service 2 |                 |
|                             +------>|           |                 |
|                                     +-----------+                 |
+-------------------------------------------------------------------+

Solution III, Duplicate Prometheus

This is almost the same as Solution II except that I keep one Grafana with many Prometheus data sources. This looks neat too but I still have to secure my Prometheus instance behind a reverse proxy.

I am also thinking to create a Solution III Bis with the use of a kind of proxy or push-exporter in place of Prometheus in each environment.

a) The usage of a proxy does not bring much value except having only one data source in Grafana.

b) The usage of a push-exporter may remove the external entry point in each environment however I do not know how I will be "informed" when one environment is down (since it will not push anything).

                 +- Environment A ----------------------------------+
 +---------+     | +------------+                                   |
 | Grafana |     | | Prometheus |                                   |
 |         |-+---->|            |                                   |
 +---------+ |   | +------------+    +-----------+                  |
             |   |           |       | Service 1 |                  |
             |   |           +------>|           |                  |
             |   |           |       +-----------+                  |
             |   |           |                                      |
             |   |           |       +-----------+                  |
             |   |           |       | Service 2 |                  |
             |   |           +------>|           |                  |
             |   |                   +-----------+                  |
             |   +--------------------------------------------------+
             |
             |   +- Environment B ----------------------------------+
             |   | +------------+                                   |
             |   | | Prometheus |                                   |
             +---->|            |                                   |
                 | +------------+    +-----------+                  |
                 |           |       | Service 1 |                  |
                 |           +------>|           |                  |
                 |           |       +-----------+                  |
                 |           |                                      |
                 |           |       +-----------+                  |
                 |           |       | Service 2 |                  |
                 |           +------>|           |                  |
                 |                   +-----------+                  |
                 +--------------------------------------------------+

Answer 1

I don't think there's an ideal solution at all. However, there may be a preferred one, depending on circumstances.

There're several factors which I think one has to consider:

• How cheap/expensinve a dedicated instance is?
• What are security concerns/considerations?
• What is expected volume of data from every environment?

This table can resume them :

 |    Factor \ |  Solution I  | Solution II  | Solution III |
 |____________\|______________|______________|______________|
 | Data Volume |     Low      |     High     |     High     |
 |-------------+--------------+--------------+--------------|
 |    Security |      No      |     Yes      |     Yes      |
 |-------------+--------------+--------------+--------------|
 |Instance Cost|     High     |     Low      |    Medium    |
 |_____________|______________|______________|______________|

Based on the analysis of one's particular circumstances a preferred solution may be selected.

I would love to know what others think about the subject, though

Answer 2

You covered the obvious security issue, use HTTPS for all your connections.

While I won't try to tell you how to implement your environment, I will share with you the design in which I am rolling out at this time ...

Grafana with Prometheus Federated cluster consuming data from Prometheus Servers deployed in each product. Thus, my data storage needs for these 'child' clusters is minimal and no need to worry about redundancy. My Federated Cluster will be storing data in a PostgreSQL server (agents available to handle the Prometheus TSDB data are available). I could then replicate this PostgreSQL server across regions for DR with a Grafana Interface per region for redundancy. I have chosen the PostgreSQL Server option over disk volumes for ease of replication.

As you have realized, there are several options on how to perform this task.

Background: My use case includes multiple stand-alone SaaS products as options within a bundled solution.

Answer 3

With the use case III, have you looked into Thanos or Cortex ? Both of these will provide:-

• Horizontal scalability (with deduplication)
• Support for secure communication with the central Thanos querier.
• Long term data storage support in S3 buckets and such, if required.
• Single place to manage all your Prometheus instances rather than adding individually into Grafana.