AWS says that their systems are designed for 99.999999999% durability, but apparently there is no guarantees in their SLA.
Yes, this can (hypothetically) be improved by using cross-region replication, but this comes at the cost of paying twice for storage, and also for data transfer costs.
How are your companies handling this? Do they go with pure trust in AWS, or do they spend (a lot) of money to (try to) improve this?
The durability design objective is not especially relevant above some number of 9's, because it is simultaneously both extremely unlikely and non-zero.
The 99.999999999% durability is easily calculated by reading between the lines as the product of the failure probabilities of 6 copies each with an AFR between 1% and 2%. Older versions of the documentation suggested 3 AZs × 2 devices per AZ was relevant to the durability calculation. Assuming these "devices" were RAID arrays, these AFRs are well within reason.
This is raw durability. It does not take into account natural disasters or acts of war or terrorism or other sabotage (any of which would need to wipe out multiple availability zones within a region) or serious bugs in the S3 system code... all of which are also extremely unlikely but similarly difficult to quantify.
And we need to factor in bugs and human errors in your code and systems, compromise of your credentials, and malicious insiders in your organization.
The bottom line is that the risk of loss is extremely low, difficult to quantify, but still non-zero.
I have many millions of objects (dozens of terabytes) in S3 and have never lost one. A majority of them have been stored with copies of both their md5 and sha256 checksums, they are all cataloged in an external database, and extensive audits have never revealed a corrupt or missing object.
But if I lose one critical object in the future, all of that still becomes irrelevant.
Can you live with the loss of an object? If so, no problem. It's really unlikely that you will lose one.
But if the data is critical, there is a compelling argument that it should still be backed up, and not exist in only "one place." An object in an S3 bucket is physically in multiple places but logically that is still "one place" because whether the object exists without corruption ultimately resolves to a single true or false answer from your perspective.
Just as using most classes of RAID arrays reduce the likelihood of data loss but do not remove the need for backup, I would argue that S3's durability -- as high as it is -- does not elminate the requirement that critical data be backed up elsewhere.
Cross-region replication is one option, and apparently one important motivation for the introduction of the 1-Zone-Infrequent-Access storage class -- similarly durable except for the case where the availability zone is lost -- was as a very low cost target for cross-region replication. It seems to meet that objective -- it's more vulnerable to the loss of a single AZ (and you don't get to pick which AZ) but that's only relevant when data loss has occurred in another bucket, in a totally different region. A replicated bucket could also migrate its copies of objects to Glacier, further reducing long-term storage costs. (Note that very small objects may encounter storage cost increases since 1ZIA has a minimum billable object size of 128K, and with the Glacier storage class, you will be paying for 40k¹ of metadata storage in addition to the object size in bytes).
Depending on the two regions involved, cross-region data transfer charges can be as low as $0.02/GB (or even $0.01/GB for the special case of us-east-1/us-east-2) which hardly seems burdensome if you need that additional level of assurance -- and note that since CRR also supports object versioning and can be configured not to replicate object deletion, it provides some protections against the risks that you pose to your own data -- risks that arguably exceed the systemic risks in S3 itself.
Other options are cross-cloud-provider replication, copying objects to GCS, for example, which would be more expensive due to data transfer costs but then can also fit into your "beyond worst case" disaster recovery plan for when AWS (or your entire AWS account?) is completely inaccessible. The cost of cross-cloud can be partially mediated by using aggressive compression where applicable, and services like Direct Connect, which reduces the cost of exporting data from AWS, particularly if the presence of the interconnection is justified by other purposes.
Or, consider the extreme opposite: a managed local backup strategy that makes physical, tangible backups on DVD or Blu-Ray is still better than no backup at all. I take a fair amount of grief for suggesting this "low tech" approach, but as long as the lifespan of the media is factored in and aggressive checksumming and integrity measures are in place, this is a viable protection against the extremely unlikely event where data is lost from S3. The only problem with this solution is that it appears to cost less than it really does -- it can be somewhat labor-intensive, requiring frequent touching of disks even when everything else is automated -- and of course if you compare the costs of this "simple" solution to the relative set-and-forget of cross-region replication, CRR becomes a pretty obvious choice.
¹ 40k is an oversimplification, based on two values, 8k + 32k, mentioned in the Amazon S3 Developer Guide. "For each archived object, Glacier adds 32 KB of storage for index and related metadata" is straightforward enough, but the explanation of the remaining 8k is a bit less intuitive: "For each object archived to Glacier, Amazon S3 uses 8 KB of storage for the name of the object and other metadata. Amazon S3 stores this metadata so that you can get a real-time list of your archived objects by using the Amazon S3 API... You are charged standard Amazon S3 rates for this additional storage." When S3 migrates an object to the Glacier storage class, you are still able to access informarion about the object in bucket listings and access all the object's metadata in real time, just as any other S3 object, even though the object payload itself is stored in Glacier and not accessible in real time. This metadata is stored at the S3 STANDARD storage class rate, rather than the Glacier rate. It is unclear whether this is a fixed 8k, or actually a maximum, depending in part on the actual size of the object metadata, which includes but is not limited to the optional x-amz-meta-* user-defined metadata for each object. As such, the importance of this 8k as a cost consideration is not entirely clear. The 32k is noteworthy whenever it is significant in comparison to the total size of a small object.
