DevOps Stack Exchange is a question and answer site for software engineers working on automated testing, continuous delivery, service integration and monitoring, and building SDLC infrastructure. It only takes a minute to sign up.
Sign up to join this community
Amazon S3 has an option of cross-region replication which should be pretty fault-tolerant against region/zone outages.
Does that mean those who are ranting about the outage did not make use of this aspect?
Or is that cross-region replication is not completely fool-proof and would not have helped?
The drawback when there's replication come from the note below:
Amazon S3 routes any virtual hosted–style requests to the US East (N. Virginia) region by default if you use the US East (N. Virginia) endpoint (s3.amazonaws.com), instead of the region-specific endpoint (for example, s3-eu-west-1.amazonaws.com).
When you use replication you usualy let AWS takes care of routing the alias to one region by targetting s3.amazonaws.com in your REST request from your servers and let the redirect do it's job.
Whenever N.Virginia is down, the magic cease to work and you're out of luck to access your data and have to update your configuration to choose a specific region endpoint.
The problem does not come from the DNS (a request to the bucket itself will work) but from S3 clients, which will connect to the S3 API endpoint before accessing the bucket, in this case the dns resolution is done on s3.amazonaws.com and this is us-east-1 endpoint.
When you use regions alias, you loose the ease of load balancing over regions with the health check from AWS included.
If you use DNS cname targeting the regions to switch quickly, you're responsible of your DNS TTL but nothing guarantee cache servers of client ISP will honor your value (one of many cache your client may encounter).
And lastly, if you try to load balance yourself you'll probably create the same SPOF than AWS already have with the added burden of maintaining it.
AWS is working on it but that's all the information I have at time of writing.
example-bucket bucket exists, example-bucket.s3.amazonaws.com already points to US East in DNS. Within a few minutes of initial bucket creation, this permanently changes to point to the correct regional endpoint. The warning here is that this hostname may initially be briefly misrouted immediately after bucket creation -- not later.
Mar 2, 2017 at 12:21
s3.amazonaws.com zone within a few minutes of bucket creation, and this change persists independently of us-east-1. Create a bucket in another region and watch how your-bucket-name.s3.amazonaws.com resolves before, during, and a few minutes after bucket creation. The information is pushed to the s3-1.amazonaws.com zone in Route 53 after bucket creation and persists there, without further reliance on us-east-1.
Mar 2, 2017 at 12:29
Many big companies would be at fault for not using this feature. It does add additional cost, and historically any kind of real disaster recovery solution is untested even if implemented.
Other than the cost issue, companies who are actively using cross-region replication can offer a valid concern regarding the latency it takes for an object to replicate. S3 does not allow (as far as I know) read-after-write consistency on replicated objects, while it does allow it for a bucket in a single region.
This SE question raises a concern where objects are not being replicated properly, or take too long to replicate. Provided that cross-region replication is done in a mode of eventual consistency, there are a lot of concerns to address.
