Skip to main content
1 of 2

The durability design objective is not especially relevant above some number of 9's, because it is simultaneously both extremely unlikely and non-zero.

The 99.999999999% durability is easily calculated by reading between the lines as the product of the failure probabilities of 6 copies each with an AFR between 1% and 2%. Older versions of the documentation suggested 3 AZs × 2 devices per AZ was relevant to the durability calculation. Assuming these "devices" were RAID arrays, these AFRs are well within reason.

This is raw durability. It does not take into account natural disasters or acts of war or terrorism or other sabotage (any of which would need to wipe out multiple availability zones within a region) or serious bugs in the S3 system code... all of which are also extremely unlikely but similarly difficult to quantify.

And we need to factor in bugs and human errors in your code and systems, compromise of your credentials, and malicious insiders in your organization.

The bottom line is that the risk of loss is extremely low, difficult to quantify, but still non-zero.

I have many millions of objects (dozens of terabytes) in S3 and have never lost one. A majority of them have been stored with copies of both their md5 and sha256 checksums, they are all cataloged in an external database, and extensive audits have never revealed a corrupt or missing object.

But if I lose one critical object in the future, all of that still becomes irrelevant.

Can you live with the loss of an object? If so, no problem. It's really unlikely that you will lose one.

But if the data is critical, there is a compelling argument that it should still be backed up, and not exist in only "one place." An object in an S3 bucket is physically in multiple places but logically that is still "one place" because whether the object exists without corruption ultimately resolves to a single true or false answer from your perspective.

Just as using most classes of RAID arrays reduce the likelihood of data loss but do not remove the need for backup, I would argue that S3's durability -- as high as it is -- does not elminate the requirement that critical data be backed up elsewhere.

Cross-region replication is one option, and apparently one important motivation for the introduction of the 1-Zone-Infrequent-Access storage class -- similarly durable except for the case where the availability zone is lost -- was as a very low cost target for cross-region replication. It seems to meet that objective -- it's more vulnerable to the loss of a single AZ (and you don't get to pick which AZ) but that's only relevant when data loss has occurred in another bucket, in a totally different region. A replicated bucket could also migrate its copies of objects to Glacier, further reducing long-term storage costs. (Note that very small objects may encounter storage cost increases since 1ZIA has a minimum billable object size of 128K, and Glacier bills for 40K of metadata storage in addition to the object size in bytes).

Depending on the two regions involved, cross-region data transfer charges can be as low as $0.02/GB (or even $0.01/GB for the special case of us-east-1/us-east-2) which hardly seems burdensome if you need that additional level of assurance -- and note that since CRR also supports object versioning and can be configured not to replicate object deletion, it provides some protections against the risks that you pose to your own data -- risks that arguably exceed the systemic risks in S3 itself.

Other options are cross-cloud-provider replication, copying objects to GCS, for example, which would be more expensive due to data transfer costs but then can also fit into your "beyond worst case" disaster recovery plan for when AWS (or your entire AWS account?) is completely inaccessible. The cost of cross-cloud can be partially mediated by using aggressive compression where applicable, and services like Direct Connect, which reduces the cost of exporting data from AWS, particularly if the presence of the interconnection is justified by other purposes.

Or, consider the extreme opposite: a managed local backup strategy that makes physical, tangible backups on DVD or Blu-Ray is still better than no backup at all. I take a fair amount of grief for suggesting this "low tech" approach, but as long as the lifespan of the media is factored in and aggressive checksumming and integrity measures are in place, this is a viable protection against the extremely unlikely event where data is lost from S3. The only problem with this solution is that it appears to cost less than it really does -- it can be somewhat labor-intensive, requiring frequent touching of disks even when everything else is automated -- and of course if you compare the costs of this "simple" solution to the relative set-and-forget of cross-region replication, CRR becomes a pretty obvious choice.