How to monitor AWS S3 cross-region replication process?

Question

Amazon S3 has a cross-region replication which will handle copy of new/updated objects to additional region.

The problem is that solution does not provide visibility on state for replication process, for example at the moment there's no way to easily monitor missing objects on destination or any possible permission issues that can interfere with the process and can result with replication not doing it's job, as well missing feedback for replication lag.

We found several tools similar to CRR Monitor https://aws.amazon.com/answers/infrastructure-management/crr-monitor/ but we didn't have good experience with it as we manage tens of millions of objects which can cost several thousand of dollars per month for CloudWatch events, as well as additional management for services required by solution (DinamoDB, Kinesis etc.) another layer of "monitoring the monitor" required.

For now we are using daily comparison of S3 objects inventory between source and destination using AWS Athena query, which also feels like a hack and it's not optimal for problem detection as inventory generated with daily frequency.

I hope to see some feedback, suggestions, ideas or experience for alternatives. Ideal solution would be managed or "serverless" solution which will improve visibility on the process and will provide fast detection of errors.

Answer 1

My solution was based on the one of @rombob. I used S3 and created a daily S3 inventory file of the source bucket (which is replicated). Then I created a Athena DB and table using Glue (in CloudFormation). Then I implemented an S3 event which triggers a lambda. Every time a new manifest.checksum file was uploaded (= new inventory is finished). The lambda executes a query on the Athena table and checks if there are objects which replication status is FAILED. If there are then the lambda publishes to a SNS topic to send an email.

Answer 2

Still no answers, and still no CRR monitoring in place on AWS, here's minimalistic solution, any feedback and suggestions are welcome. This monitor will only show which S3 objects are missing from destination, it will not provide visibility on versions of replicated objects.

Prerequisites

---

1. Enable inventory reports on src (Source) and dst (Destination) buckets
2. Create Athena tables (src_inventory, dst_inventory) based on inventory reports for each bucket

Monitor

---

Run Athena query after both inventory reports generated e.g. on daily basis, can be executed from Lambda, Step Function, build server or any type of cron job.

Example of Athena query which should show number of missing objects on destination bucket at the time of generation of inventory reports:

FROM src_inventory src 
LEFT JOIN dst_inventory dst 
ON src.key = dst.key 
WHERE dst.key is NULL
AND src.dt = '2018-11-11-08-00';

The result can be reported as CloudWatch metric and monitored with CloudWatch alarm.

Answer 3

At this point using S3 inventory to monitor replication is the best solution for large buckets, and the solution suggested by AWS themselves. Since AWS don't report on the status directly, any other solution is likely to be both expensive and time consuming when that many objects are involved. It seems like you're on the right (and only?) track.

An obvious idea would be to talk to AWS Support about it, ask them to add such a feature. But that's obvious.