Please help me here based on your experiences.
We have launched a desktop application where users can download all their online data from several platforms at one place. Our motto is to provide users access to their digital footprint so that they can utilize it in their own different ways.The whole idea of desktop application is to put privacy first, Our servers only have access to the users login information and never had access to the users data, which is also encrypted at rest on users machine with user's mnemonic.
Based on the users feedback, We have realized that most of the users did like our platform but they don't want to bear the hassle of installing a desktop application. For them, Making use of their data (search, organize etc) is more important than military grade privacy.
As a result, we have decided to provide both the options to the user, Desktop application as well as web app.
Now shifting such a huge data on users machine (around 50GB to 1 TB for single user) to common database on cloud would be a cumbersome task, In terms of servers required for replication, sharding , caching etc, their costs and complexities which comes with data at scale.
We are thinking to maintain docker containers for every user with their storage on Elastic file system on AWS. Data on rest on EFS for a container will always be encrypted with user's mnemonic and will only be decrypted on successful users login.
Every user docker container will have a small SQLite database to make this data searchable and indexed. In future, containers can also have their own Elastic search database with any other NoSql or Sql database.
The identities of the users can be put of AWS Cognito or Any decentralized identity system.
One other approach could be: Instead of running dockers we can keep different encrypted SQLITE DB for different users.
It would be a problem because then we have to keep track of all the SQLite db files present being accessed by the user, because SQLite doesnt support multi threading, we cannot open different instances of same SQLite database,
In case of docker, We will only spin docker when user logs in, that ways we always know that only one instance of SQLite is being accessed.
Our worry: We are still trying to figure out loopholes in this approach. Please help us in identifying the same.