What is a good strategy to keep my site online when S3 goes offline?

Question

What is a good strategy to keep my site online when S3 goes offline?

If S3 US East 1 goes offline, how should I have my app configured/structured to prevent that taking my entire site offline?

What are the best strategies to diversify in this sort of situation?

Answer 1

In March 2015, Amazon AWS announced they support S3 replication across regions. When a certain region in S3 goes offline, you can serve files from your mirror in another region.

^{source: https://aws.amazon.com/blogs/aws/new-cross-region-replication-for-amazon-s3/}

The practice of keeping your infrastructure online by doing a switch over to another region is a complex one, but S3 is a relatively small and simple component. Netflix has a great article on their experience with Chaos Gorilla.

This also applies to service degradation, like increased latency. Not just when a service you depend upon is completely offline. Netflix has an article on this as well: Chaos Engineering Upgraded.

Answer 2

What you are asking for is, basically, high-availability. In order to make a system highly available, you need three things:

1. Eliminate single points of failure
2. A mechanism to switch from an endpoint to another
3. A way to detect failures

Eliminate single points of failure

In the case of S3, the point #1 is addressed, as Evgeny pointed out, by S3 cross-region replication.

Replication, however, is not instantaneous and you'll want to check if you want to make your application replication aware or not. In the event of an outage, it is possible that something that was written to your source bucket has not yet made it (wasn't replicated) to the destination bucket. You have to think how the application would handle such a scenario. That really depend on the type of data, what is being done with it and (potentially) the end-users or management expectations.

A mechanism to switch from an endpoint to another

For S3, that means that in the event of an outage, you want the application to stop reading and writing from/to bucket A and use bucket B instead.

How to achieve this is, as far as I know, up to you for now. Some other AWS services offer completely transparent fail-overs, but I'm not aware of such a thing for S3 at the moment.

There are various ways to achieve this. One example is using a proxy that will route the traffic to the appropriate bucket. During an outage, you would update/change the proxy to route the traffic to a bucket not affected by the outage. Another example would be to make your application configuration dynamic and store it in a key-value store. If the application reads the KV store for updated properties often enough, you can switch where you read from and write to (Spring Cloud has support for an "EnvironmentChange" listener, for example).

A way to detect failures

Well, that one is easy, I think. Simply setup a write + read loop and alert as soon as something isn't right :)

Closing notes

• If your application is writing to the bucket, you have to think about what would happen in the case of a fail-over. Have all the writes made it to the destination bucket (and can you tell)? Can you allow writes to the destination bucket (making it the new "primary")? Careful planning will avoid split-brain or lost updates scenarios.
• Depending on your SLA, you may want the points #2 and #3 to be automated or automatic. That requires additional planning, tooling and testing, but well-written scripts will always react faster and in more predictable ways than human can (failures also have the annoying habit of happening in the middle of the night when human intervention is something hazardous.
• It's worth mentioning that even cross-region replication doesn't completely eliminate single points of failure. Sure, if a region goes down, you are covered. But what if a US-wide AWS outage happens? Azure had a partial but global outage last year and one in 2014 too.