4

I am just wondering. If I have many environments monitored via Prometheus, what will be the best configuration? The security and efficiency are already important for on-premise installations but they are much more important and unavoidable when deployed on a cloud provider.

From your experience, what is the most common and useful configuration? I see three main scenarios feel free to propose something else.

Thanks.

Solution I, Duplicate nothing

It is easy for the monitoring user because there is only one Grafana with one Datasource. However, from a security point of view, I have to open a port for each service in each environment. I will, of course, configure one reverse proxy in each environment to secure the exporters over Https and Basic authentication.

+---------+       +------------+       +- Environment A -------------+
| Grafana |       | Prometheus |       |     +-----------+           |
|         |------>|            |--+--+------>| Service 1 |           |
+---------+       +------------+  |  | |     |           |           |
                                  |  | |     +-----------+           |
                                  |  | |                             |
                                  |  | |     +-----------+           |
                                  |  +------>| Service 2 |           |
                                  |    |     |           |           |
                                  |    |     +-----------+           |
                                  |    +-----------------------------+
                                  | 
                                  |    +- Environment B -------------+
                                  |    |     +-----------+           |
                                  +--------->| Service 1 |           |
                                  |    |     |           |           |
                                  |    |     +-----------+           |
                                  |    |                             |
                                  |    |     +-----------+           |
                                  +--------->| Service 2 |           |
                                       |     |           |           |
                                       |     +-----------+           |
                                       +-----------------------------+

Solution II, Duplicate everything

This looks like a popular solution and may have many benefits from a deployment point of view. I find it quite elegant because we can have on docker-compose.yml that set up everything. But while it can be interesting for security segregation, there will be many Grafana (each with one Prometheus data source) and that may not be useful for the monitoring user.

+- Environment A ---------------------------------------------------+
| +---------+       +------------+                                  |
| | Grafana |       | Prometheus |                                  |
| |         |------>|            |                                  |
| +---------+       +------------+    +-----------+                 |
|                             |       | Service 1 |                 |
|                             +------>|           |                 |
|                             |       +-----------+                 |
|                             |                                     |
|                             |       +-----------+                 |
|                             |       | Service 2 |                 |
|                             +------>|           |                 |
|                                     +-----------+                 |
+-------------------------------------------------------------------+

Solution III, Duplicate Prometheus

This is almost the same as Solution II except that I keep one Grafana with many Prometheus data sources. This looks neat too but I still have to secure my Prometheus instance behind a reverse proxy.

I am also thinking to create a Solution III Bis with the use of a kind of proxy or push-exporter in place of Prometheus in each environment.

a) The usage of a proxy does not bring much value except having only one data source in Grafana.

b) The usage of a push-exporter may remove the external entry point in each environment however I do not know how I will be "informed" when one environment is down (since it will not push anything).

                 +- Environment A ----------------------------------+
 +---------+     | +------------+                                   |
 | Grafana |     | | Prometheus |                                   |
 |         |-+---->|            |                                   |
 +---------+ |   | +------------+    +-----------+                  |
             |   |           |       | Service 1 |                  |
             |   |           +------>|           |                  |
             |   |           |       +-----------+                  |
             |   |           |                                      |
             |   |           |       +-----------+                  |
             |   |           |       | Service 2 |                  |
             |   |           +------>|           |                  |
             |   |                   +-----------+                  |
             |   +--------------------------------------------------+
             |
             |   +- Environment B ----------------------------------+
             |   | +------------+                                   |
             |   | | Prometheus |                                   |
             +---->|            |                                   |
                 | +------------+    +-----------+                  |
                 |           |       | Service 1 |                  |
                 |           +------>|           |                  |
                 |           |       +-----------+                  |
                 |           |                                      |
                 |           |       +-----------+                  |
                 |           |       | Service 2 |                  |
                 |           +------>|           |                  |
                 |                   +-----------+                  |
                 +--------------------------------------------------+

3 Answers 3

3

I don't think there's an ideal solution at all. However, there may be a preferred one, depending on circumstances.

There're several factors which I think one has to consider:

  • How cheap/expensinve a dedicated instance is?
  • What are security concerns/considerations?
  • What is expected volume of data from every environment?

This table can resume them :

 |    Factor \ |  Solution I  | Solution II  | Solution III |
 |____________\|______________|______________|______________|
 | Data Volume |     Low      |     High     |     High     |
 |-------------+--------------+--------------+--------------|
 |    Security |      No      |     Yes      |     Yes      |
 |-------------+--------------+--------------+--------------|
 |Instance Cost|     High     |     Low      |    Medium    |
 |_____________|______________|______________|______________|

Based on the analysis of one's particular circumstances a preferred solution may be selected.

I would love to know what others think about the subject, though

1
  • "I would love to know what others think about the subject", yes me too.
    – gervais.b
    Feb 18, 2020 at 8:38
1

You covered the obvious security issue, use HTTPS for all your connections.

While I won't try to tell you how to implement your environment, I will share with you the design in which I am rolling out at this time ...

Grafana with Prometheus Federated cluster consuming data from Prometheus Servers deployed in each product. Thus, my data storage needs for these 'child' clusters is minimal and no need to worry about redundancy. My Federated Cluster will be storing data in a PostgreSQL server (agents available to handle the Prometheus TSDB data are available). I could then replicate this PostgreSQL server across regions for DR with a Grafana Interface per region for redundancy. I have chosen the PostgreSQL Server option over disk volumes for ease of replication.

As you have realized, there are several options on how to perform this task.

Background: My use case includes multiple stand-alone SaaS products as options within a bundled solution.

4
  • Thanks. We have less requirements than you but this looks like our preferred solution which is the "Solution III". I was not aware of the federation in Prometheus, that's while I was thinking to use something like github.com/pambrose/prometheus-proxy that push metrics in Prometheus instead of letting it pulling them. What would be the advantage of using the Federation or the proxy ?
    – gervais.b
    Feb 19, 2020 at 22:15
  • Prometheus Federation allows you to effectively define parent / child relationship to allow the parent to poll data from the children into a centralized repository. Whereas, Prometheus Proxy allows you to pass your Grafana connection through a firewall to your Prometheus Server. These are entirely different concepts. Feb 20, 2020 at 0:23
  • I fear that we are not speaking about the same proxy, the one in my link uses one agent that push metrics to a proxy through a firewall. So I guess that the main difference with the federation is that the "Pambrose proxy" push the metrics instead of letting another pulling them.
    – gervais.b
    Feb 20, 2020 at 7:46
  • Apologies, i misread that page. Yes, you are correct, an agent gathers metrics and pushes them through a firewall to the Prometheus server. Feb 20, 2020 at 12:53
0

With the use case III, have you looked into Thanos or Cortex ? Both of these will provide:-

  • Horizontal scalability (with deduplication)
  • Support for secure communication with the central Thanos querier.
  • Long term data storage support in S3 buckets and such, if required.
  • Single place to manage all your Prometheus instances rather than adding individually into Grafana.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.