Google Cloud Storage vs Redis / ElasticSearch

Question

We're building an application that reads data from an API and displays it in a frontend. Due to API limits, we have to cache the API calls meaning that we save the result of the API calls (both user information (first name, country, email etc.) and big sets of activity log data) in a database and only request new data from the API if the stored data is older than x hours.

This application should also work for multiple users that can log in and see their relevant data. For each user, the application connects to a different API endpoint (e.g. "user1.api.com/data/whatever" and "user2.api.com/data/whatever").

So, in the end there will be two databases. One for caching the API calls of an external tool and another one for the application's user management. However, we are not sure what would be best to use.

Is it generally a good idea to use cloud databases for either caching or user management? As the application will run on Google Container Engine, we could probably use Google Cloud Storage (for caching) and Firebase (for user management). Or should we rather use something like Redis / ElasticSearch (for caching) and something different for the different user data?

Answer 1

I don't know much about Redis/ElasticSearch, but GCS is not really a database-like solution, it is closer to a file storage solution.

If you're looking for database-like storage Google Cloud offers:

• Cloud SQL
• Cloud Bigtable
• Cloud Datastore
• Cloud Spanner

If you expect a modest app traffic or alternating high/low traffic it might be more cost-effective (and simpler) to have a Google App Engine app instead of a GCE-based solution.

Answer 2

We had the similar requirement as there needs to be a set of data cached in a server for serving the application (fast processing) and also by end of the day, we would need to sync the data from the cache server to the origin server.

In the longer run, I would suggest to go for Redis as you can have Redis as an intermediate database (caching) and it will help you in scaling.

You can write scripts to run at an interval or at specific time of the day so that you can pull/push the data from the origin server (external application).

Your requirement wouldn't require Elastic Search at any point of time as it will help you in speeding up search results based on caching, it is just a search engine.