11

I would like to know how everyone is making external backups of S3 buckets and best practices/recommendations for this. Regardless of how good AWS is it is still a single point of failure.

Especially on a more corporate scale, vast amounts of buckets that are very large.

The S3 buckets functionality seems to be very limited.

My resolution was the following:

The way I'm looking at doing this now is by downloading the data to an external server (bare-metal as EC2 instance is a lot more expensive) and then pushing that to my backups provider.

Improve this question
6
  • 4
    Have you looked into Amazon's Glacier service? When combined with AWS Lambda, you can create a very cost efficient backup service at the cost of the time to access a backup.
    – Preston Martin ♦
    Commented Oct 9, 2017 at 16:33
  • 2
    What is the actual problem that you're trying to solve? Is it cost, need for additional reliability, do you want to recover from an all-regions failure, or something else?
    – kdgregory
    Commented Oct 11, 2017 at 15:07
  • Short answer, though, is that yes, downloading and uploading is your only option. S3 is intended to be a destination for data, not a waypoint. Depending on how much data you have and how frequently you want to move it (and why), there may be different "best" approaches (and be aware that there may be a charge for data transfer).
    – kdgregory
    Commented Oct 11, 2017 at 15:11
  • 1
    @kdgregory yes, I see that this the only way of doing it and I have resolved the issue as per the third line in my question. Leaving this open to see what other people recommend.
    – Kyle Steenkamp
    Commented Oct 12, 2017 at 8:43
  • 1
    It is just to have an external backup as opposed to having everything just on S3, which is a single point of failure.
    – Kyle Steenkamp
    Commented Dec 13, 2017 at 16:50

3 Answers 3

Reset to default
1

In terms of S3, for increased durability, consider cross region replication as an option if you want to increase the durability. AWS also provides their AWS Backup service you might investigate. I'm not certain of any benefit to transferring large amounts of data for durability to another provider. You'll pay a lot more for transfer, and it won't be integrated first class with the security IAM provides.

If the concern is about someone maliciously doing something then there are security best practices, access logging, cloudtrail, and other options to provide assurance. If it's more about the possibility (as low as it seems) for a bucket to fail, then the cross-region replication might be a great option for you with minimal configuration impact.

Improve this answer
0

You can take backup of s3 using Amazon, other cloud services like google and local system.

  • Local system

S3 Browser comes with a simple Folder Sync Tool allowing you to upload or download only new and changed files and thereby significantly optimize your bandwidth usage and save you time when performing backups to Amazon S3.

This very good tool if your using window. here is the link http://s3browser.com/amazon-s3-folder-sync.aspx if you using Linux based system then i tired this tool also good one DragonDisk

  • cloud-based backup

you can take backup of s3 in google cloud as well.

The gsutil command-line tool also enables you to transfer data between Cloud Storage and other locations. While you can use gsutil to work with Amazon S3 buckets and transfer data from Amazon S3 to Google Cloud Storage, Storage Transfer Service is recommended for this use case.

  • AWS

As with any environments, the best practice is to have a backup and to put in place safeguards against malicious or accidental users errors. For S3 data, that best practice includes secure access permissions, Cross-Region Replication, versioning and a functioning, regularly tested backup.

  • How durable is Amazon S3?

Amazon S3 Standard and Standard - IA are designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years. In addition, Amazon S3 is designed to sustain the concurrent loss of data in two facilities.

https://aws.amazon.com/s3/faqs/

Improve this answer
2
  • 2
    Thanks for the information, but this does not answer the actual question being asked :)
    – Kyle Steenkamp
    Commented Oct 12, 2017 at 8:35
  • 4
    In a corporate setting, it does not matter how reliable S3 itself is, it is still one service at one provider. One engineer with access can easily drop the entire bucket and you can lose hundreds of TB of data literally overnight with no recovery. It is a single point of failure.
    – Jiri Klouda
    Commented Dec 11, 2017 at 17:27
0

AWS Datasync service will be a great way to sync S3, EFS or FSx data from AWS to another location. You can deploy the agent on a VM in Azure or GCP and then use their storage solution to receive the data from AWS. Obviously you will need to look at charges for outbound data transfer but if you have a strong business case for doing this then the costs should not be a problem.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.