I am sorry for the newbye question. Im trying to figure out the best architecture / set of technologies to implement some very specific system:
I have several databases indexes (used for searching data elements) that are broken down into small pieces. Each datababase index is actually a serialized data structure of mine and can be composed by N index file pieces. All pieces of all databases are in a s3 bucket and can be properly identified (pieces of one same database can be found by filename pattern).
For searching into one specific database, we need to load all the N file pieces from s3 into RAM memory for fast search. But all pieces might not fit one machine RAM, so theoretically, each database can / will be distributed over several machines.
Once a query is requested for a specific database, we need to send the query to all machines that contains any piece of the database (some might contain more than one piece, no problem), and then the machines will returns if it was found or not.
Some database might get to be unused for a while and will get unloaded from machines (the database index file pieces will be unloaded from RAM).
As any database index piece is not loaded when a query arrives, it must be loaded into one of the service machines that stores the pieces in memory.
What I think it need to be done:
A master node will contain a map of loaded/unloaded databases and which pieces are loaded in which machine.
A request will arrive to master for a specific database, and it will check if all pieces for that database are loaded, if not, it will request loading sending a command to one of the available machines.
More machines are put available as soon as memory of the whole set of service machines get low (auto scaling)
After all pieces are loaded in some machines, the master updates the map of pieces and send the query to all of the corresponding machines that contains the database pieces.
The master will also have timestamps or a LRU for the pieces so it can unload after some time not being used.
So for this solution we would have a master and service slaves. Slaves contains api to load/unload a piece and also for listing which piece it has loaded.
Drawbacks from this solution:
This solution involves a master node that will be a bottleneck and a context that reflects the service nodes and its stored in master. If master fails, how this context can be restored? Not so trivial. Possible fixes would be having lots of masters and many more service slaves. Also if a master dies it can recover the state by quering the slaves.
Can anyone give me any hint on the technologies I could use? I think amazon ECS can help.
Thanks in advance.