I am just wondering. If I have many environments monitored via Prometheus, what will be the best configuration? The security and efficiency are already important for on-premise installations but they are much more important and unavoidable when deployed on a cloud provider.
From your experience, what is the most common and useful configuration? I see three main scenarios feel free to propose something else.
Thanks.
Solution I, Duplicate nothing
It is easy for the monitoring user because there is only one Grafana with one Datasource. However, from a security point of view, I have to open a port for each service in each environment. I will, of course, configure one reverse proxy in each environment to secure the exporters over Https and Basic authentication.
+---------+ +------------+ +- Environment A -------------+
| Grafana | | Prometheus | | +-----------+ |
| |------>| |--+--+------>| Service 1 | |
+---------+ +------------+ | | | | | |
| | | +-----------+ |
| | | |
| | | +-----------+ |
| +------>| Service 2 | |
| | | | |
| | +-----------+ |
| +-----------------------------+
|
| +- Environment B -------------+
| | +-----------+ |
+--------->| Service 1 | |
| | | | |
| | +-----------+ |
| | |
| | +-----------+ |
+--------->| Service 2 | |
| | | |
| +-----------+ |
+-----------------------------+
Solution II, Duplicate everything
This looks like a popular solution and may have many benefits from a deployment point of view. I find it quite elegant because we can have on docker-compose.yml
that set up everything. But while it can be interesting for security segregation, there will be many Grafana (each with one Prometheus data source) and that may not be useful for the monitoring user.
+- Environment A ---------------------------------------------------+
| +---------+ +------------+ |
| | Grafana | | Prometheus | |
| | |------>| | |
| +---------+ +------------+ +-----------+ |
| | | Service 1 | |
| +------>| | |
| | +-----------+ |
| | |
| | +-----------+ |
| | | Service 2 | |
| +------>| | |
| +-----------+ |
+-------------------------------------------------------------------+
Solution III, Duplicate Prometheus
This is almost the same as Solution II except that I keep one Grafana with many Prometheus data sources. This looks neat too but I still have to secure my Prometheus instance behind a reverse proxy.
I am also thinking to create a Solution III Bis with the use of a kind of proxy or push-exporter in place of Prometheus in each environment.
a) The usage of a proxy does not bring much value except having only one data source in Grafana.
b) The usage of a push-exporter may remove the external entry point in each environment however I do not know how I will be "informed" when one environment is down (since it will not push anything).
+- Environment A ----------------------------------+
+---------+ | +------------+ |
| Grafana | | | Prometheus | |
| |-+---->| | |
+---------+ | | +------------+ +-----------+ |
| | | | Service 1 | |
| | +------>| | |
| | | +-----------+ |
| | | |
| | | +-----------+ |
| | | | Service 2 | |
| | +------>| | |
| | +-----------+ |
| +--------------------------------------------------+
|
| +- Environment B ----------------------------------+
| | +------------+ |
| | | Prometheus | |
+---->| | |
| +------------+ +-----------+ |
| | | Service 1 | |
| +------>| | |
| | +-----------+ |
| | |
| | +-----------+ |
| | | Service 2 | |
| +------>| | |
| +-----------+ |
+--------------------------------------------------+