DynamoDB vs RDS for a counter store?

Question

I have multiple lambda functions, which a file passes through for processing, and I want to store the status of each computation in a counter, for better logging and monitoring.

Current plan is to log it like this:

file-name  Lambda1  Lambda2 Lambda3
Name1      Done     Done    In-process

And, I'll be updating the values as Received, In-process and Done under the respective Lambda functions for the file.

So, two doubts:

1. Is this a good practice? If not, then is there a better alternative for the same?
2. If it is, then what would make a better/cheaper store? RDS or DynamoDB?

Answer 1

Why not something like Redis? (can use it through ElastiCache). Hard to answer this without knowing the operational requirements of this project. Is this data strictly for logging/research, or is it operational (i.e. this "table" is consulted in order to know which lambda to invoke).

Dynamo vs. RDS - depends how spiky your traffic is. Dynamo is very scalable and will give you a guaranteed throughput. But if you traffic can grow 10x over hours (spiky) Dynamo might be too expensive, since you'll need the maximum throughput provisioned or else you'll get throttled.

RDS is basically managed RDBMS on an instance type that you choose, so it can handle larger spikes, but ultimately not as scalable as Dynamo.

Again, I would consider Redis, for it's simplicity (seems like you just need a KV/counter store), and incredible speed/efficiency. If it's just for logging, it's probably good enough.

Answer 2

For question #1)

I think this could perhaps be better answered on SoftwareEngineering SE.

Nonetheless, I'll risk an answer: for this kind of information (states) coming from that kind of architecture (distributed), I recommend event sourcing. It removes the complexity and many headaches that come with the nature of distributed systems if someone try to use traditional database paradigms.

For question #2: both would work. To know what is better and cheaper we'll need to know more about your use case.

Answer 3

The cost will depend on the number of messages you write per second. Most often DDB is cheaper. If you have a large throughput, 10k requests per second, then your DDB costs will need to be carefully considered vs. RDS.

In this case however, I would actually reccommend using S3 to output your log messages to a file. I think technically this will be a good enough solutin, at a fraction of the cost to setup and maintain compared to either DDB or RDS.