How to create a centralised dashboard to monitor multiple cloud (Amazon and Google)?

Question

I'm looking for a single product or service which can be used as a centralized dashboard to monitor cloud infrastructure provide by Amazon Web Services and Google Cloud Platform in addition to the application.

I'm looking at few options like sensu which support all of the services in Amazon Web Services, but I didn't find any plugins to monitor Google Cloud Platform.

Are there any other services or products that allow me to create this, my preference is Open Source?

Note: Everyone has access to GCP and AWS and if anyone launches something it will be easy to have one dashboard for monitoring.

Answer 1

I think what you are looking for is an Open Source project that can take inputs from both Amazon CloudWatch and Google StackDriver; there isn't a huge amount out there at the moment, but I will detail what I know.

I have made the assumption that you know how to import your application telemetry into the solutions below.

Open Source

Open source solutions to this problem have been lagging behind as late due to favourable prices of SaaS and built-in solutions. The products below are constantly evolving so if they don't do what you need now they may be in the future.

• Elastic Stack, formerly known as the ELK stack is made up of three individual products from Elastic.co: ElasticSearch, LogStash and Kibana. It is possible to ingest data from CloudWatch with a plugin and to export StackDriver data to Google Pub/Sub and then ingest into LogStash.
• Grafana, is slightly better than Kibana at displaying time-series data. However, dosn't have as robust an ecosystem around it. Grafana has built in support for Amazon CloudWatch, Google StackDriver is not supported as yet. However, someone is working on it in the form of a plugin.

Software as a Service (SaaS)

Software as a Service monitoring solutions is a multi-billion dollar market, as large enterprises will readily pay for a SaaS or off-the-shelf solution to their dashboarding challenge.

• New Relic, a very mature product with extensive cloud support, including ingesting of CloudWatch metrics and Google Cloud Storage Metrics.
• Splunk, a bit more focused on logs than metrics, however, has good support for AWS CloudWatch and Google Cloud Platform.
• Google Stackdriver, obviously has good support for Google Cloud Platform, also it is capable of reading performance metrics in from AWS.
• DataDog is my personal favourite, it has a vast range of integrations including support for Amazon Web Services and Google Cloud Platform.

Single Pane of Glass

In the last three years, various products have gone to market providing a "Single Pane of Glass". The advantages of these products are:

• Heterogeneous Platforms - you can use many different monitoring tools and products and then unify them with a Single Pane of Glass.
• Data Normalization - most monitoring tools have idiosyncrasies in the way they present data a Single Pane of Glass helps to normalise these so that metrics from Google, Amazon and your App make sense when displayed in the same graph.
• Business View - a Single Pane of Glass usually has the capability to integrate with non-technical data sources. As an example displaying data from financial tools to show a measure of business success. This view makes for some compelling C-Level dashboards that demonstrate how technology is providing value to the firm.