1

I am sorry for the newbye question. Im trying to figure out the best architecture / set of technologies to implement some very specific system:

  1. I have several databases indexes (used for searching data elements) that are broken down into small pieces. Each datababase index is actually a serialized data structure of mine and can be composed by N index file pieces. All pieces of all databases are in a s3 bucket and can be properly identified (pieces of one same database can be found by filename pattern).

  2. For searching into one specific database, we need to load all the N file pieces from s3 into RAM memory for fast search. But all pieces might not fit one machine RAM, so theoretically, each database can / will be distributed over several machines.

  3. Once a query is requested for a specific database, we need to send the query to all machines that contains any piece of the database (some might contain more than one piece, no problem), and then the machines will returns if it was found or not.

  4. Some database might get to be unused for a while and will get unloaded from machines (the database index file pieces will be unloaded from RAM).

  5. As any database index piece is not loaded when a query arrives, it must be loaded into one of the service machines that stores the pieces in memory.

What I think it need to be done:

IDEA

  1. A master node will contain a map of loaded/unloaded databases and which pieces are loaded in which machine.

  2. A request will arrive to master for a specific database, and it will check if all pieces for that database are loaded, if not, it will request loading sending a command to one of the available machines.

  3. More machines are put available as soon as memory of the whole set of service machines get low (auto scaling)

  4. After all pieces are loaded in some machines, the master updates the map of pieces and send the query to all of the corresponding machines that contains the database pieces.

  5. The master will also have timestamps or a LRU for the pieces so it can unload after some time not being used.

So for this solution we would have a master and service slaves. Slaves contains api to load/unload a piece and also for listing which piece it has loaded.

Drawbacks from this solution:

This solution involves a master node that will be a bottleneck and a context that reflects the service nodes and its stored in master. If master fails, how this context can be restored? Not so trivial. Possible fixes would be having lots of masters and many more service slaves. Also if a master dies it can recover the state by quering the slaves.

Can anyone give me any hint on the technologies I could use? I think amazon ECS can help.

Thanks in advance.

Improve this question
1
  • Would Elasticsearch do the trick? It does index sharding across multiple machines. You could decide on all-flash storage, RAMDisk, or Tier III storage for the shards and Elasticsearch has jobs to take indexes and prune out old data, unneeded segments of data or just take that data from every 5 minutes, to an hour summary.
    – James Shewey
    Jul 26 at 3:40

1 Answer 1

Reset to default
0

You might want to have a look at this video:

"Why We Built Our Own Distributed Column Store" by Sam Stokes

It illustrates how you could build a highly scalable columnar data store using SSDs rather than loading everything into RAM. They got really high performance out of this solution (try the Honeycomb demo to see it).

It might fit your use case?

Improve this answer
3
  • Thanks for the reference. However, I will not use a traditional database, the index is a custom data structure serialized that I can only read if I load into RAM. Loading even from SSD its orders or magnitude slower than from RAM, so I also need short latency. Im looking for something that can store things in RAM and be horizontally scaled.
    – Pablo Carneiro Elias
    Oct 29, 2021 at 1:11
  • If you don't have to perform complex operations - e.g. only count, sum and average - then you could create a distributed system to perform partial operations to be merged into a final result. Otherwise you'll need plenty of RAM. You could also check out Snowflake and see if their approach fits your use case.
    – Erik Norman
    Oct 29, 2021 at 6:11
  • I need to use an specialized algorithm for performing a KNN search. The index is actually a serialized index of a kdtree for binary descriptors (using hamming distance).
    – Pablo Carneiro Elias
    Oct 29, 2021 at 8:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.