We changed our privacy policy. Read more.
28

In March 2015, Amazon AWS announced they support S3 replication across regions. When a certain region in S3 goes offline, you can serve files from your mirror in another region. source: https://aws.amazon.com/blogs/aws/new-cross-region-replication-for-amazon-s3/ The practice of keeping your infrastructure online by doing a switch over to another region is ...


19

According to Amazon AWS Customers may use any AWS service in an account designated as a HIPAA account, but they should only process, store and transmit PHI in the HIPAA-eligible services defined in the BAA. There are ten HIPAA-eligible services today, including AWS Snowball, Amazon DynamoDB, Amazon EBS, Amazon EC2, Amazon Elastic MapReduce (EMR), ...


15

Resolution 1 So I solved this with the following script. I originally posted the question just in case there was an easier way that I was not aware of. #!/bin/bash aws_profile=('profile1' 'profile2' 'profile3'); #loop AWS profiles for i in "${aws_profile[@]}"; do echo "${i}" buckets=($(aws --profile "${i}" --region your_region s3 ls s3:// --recursive |...


12

AWS Import/Export is a service that accelerates transferring data into and out of AWS using physical storage appliances, bypassing the Internet. AWS Import/Export Disk was originally the only service offered by AWS for data transfer by mail. Disk supports transfers data directly onto and off of storage devices you own using the Amazon high-speed internal ...


11

What you are asking for is, basically, high-availability. In order to make a system highly available, you need three things: Eliminate single points of failure A mechanism to switch from an endpoint to another A way to detect failures Eliminate single points of failure In the case of S3, the point #1 is addressed, as Evgeny pointed out, by S3 cross-region ...


11

The drawback when there's replication come from the note below: Amazon S3 routes any virtual hosted–style requests to the US East (N. Virginia) region by default if you use the US East (N. Virginia) endpoint (s3.amazonaws.com), instead of the region-specific endpoint (for example, s3-eu-west-1.amazonaws.com). When you use replication you usualy let ...


10

Many big companies would be at fault for not using this feature. It does add additional cost, and historically any kind of real disaster recovery solution is untested even if implemented. Other than the cost issue, companies who are actively using cross-region replication can offer a valid concern regarding the latency it takes for an object to replicate. ...


10

Amazon's IAM roles generally grant a role access to a particular ARN (Amazon Resource Name). Amazon notes on their pages that for S3 a resource ...can be a bucket-name or a bucket-name/object-key. They also provide a helpful example for doing just this which appears as follows: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", ...


8

The AWS documentation is clear that automated key rotation is only supported by Amazon on a yearly basis. However, you can use the CLI to manually create new keys, and then use an automation technique such as cron to automate the manual process. In this case you can likely have cron run a bash script to read from a list of keys and credentials and then use ...


8

Yes, you can use the AWS SDK to store objects within google storage. From Simple migration: In a simple migration from Amazon S3 to Google Cloud Storage, you can use your existing tools and libraries for generating authenticated REST requests to Amazon S3, to also send authenticated requests to Google Cloud Storage. The changes you need to make to ...


7

Most of the AWS SDKs mentioned support constructing a custom endpoint, for example as described here for Node.JS. Setting the endpoint to storage.googleapis.com and using your GCP access credentials should enable you to operate the Google storage backend service "like S3".


6

I think you want Amazon Snowball, it's a HD shipping service by Amazon to/from AWS. Snowball is a petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data into and out of the AWS cloud. With Snowball, you don’t need to write any code or purchase any hardware to transfer your data. Simply create a job in the AWS ...


6

You can use a remote backend as a data source. That is working well for us thus far but this project is not very mature and we will likely refactor a time or ten. data "terraform_remote_state" "network" { backend = "s3" config { // some variables related to config would likely go here } } then for the teams not managing the network to consume ...


4

You need to use... the Internet, from your data center, to access S3 directly (not via the endpoint) or an AWS DirectConnect connection to access S3 directly (not via the endpoint) or an HTTP proxy server inside this VPC to access S3 (via the endpoint) ...because S3 endpoints are not designed to be used across VPN connections. They are only accessible ...


4

The policy will still exist. The documentation lists what is deleted when a user is deleted, any resources that the user has created will persist.


4

S3 Life cycle policy will remain and working as it is. When you use the AWS Management Console to delete an IAM user, IAM automatically deletes the following information for user: The user Any group memberships. The user is removed from any IAM groups. That the user was a member of Any password associated with the user Any access keys belonging to the ...


3

There's a generator for policies in IAM (Into IAM -> Policies -> Create), choosing connect gives me the following policy: { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "connect:DescribeInstance", "connect:CreateInstance", ...


3

Nope, credit goes to this SO answer, still valid today: How to upload files directly to Amazon S3 from a remote server?. But if you have a server in the cloud, ideally in AWS, it'll probably go faster to execute the copy operations on that server - higher bandwidth and lower latency to/from such server when compared to your computer.


3

AWS stores whole AMI as a single non-differential snapshot thus they charge for a full size of files in an image.


3

You've got a couple of options: 1. Embed the source code to the CloudFormation template I do just that in my ec2-start-stop demo. You'll see file ec2-start-stop.template.yml with these lines: StartStopLambda: Type: AWS::Lambda::Function Properties: [...] Runtime: python3.6 Code: ZipFile: Fn::...


3

No. The account that uploads an object into a bucket owns the object. All the bucket owner account can do, unless granted permission by the object owner account, is delete the object.


3

You will need to write a script that parses and queries this data because as far as I'm aware, there is not a tool or cli function that performs this. Luckily, you can gather all of this information with the CLI. List and parse all of the accounts in your org. aws organizations list-accounts For each account, list and parse all of the buckets. aws ...


3

Yes, Resource is a required element on S3 bucket policies. If the resource is omitted, the API will respond with an error resembling this one below. status code: 400 Error putting S3 policy: MalformedPolicy: Statement is missing required element The reasoning is that each statement could apply to the bucket itself, the contents of the bucket, or some ...


3

No you can't. S3 is an object storage, accessed over HTTP or REST for example. Just as you can't mount an HTTP address as a directory you can't mount a S3 bucket as a directory. Having said that there are some workarounds that expose S3 as a filesystem - e.g. 's3fs' project. How reliable and stable they are I don't know.


3

For the best performance naming many small files in an s3 bucket may not be the best idea unless they user_id is a random string. See how the S3 hashing algorithm works https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/ So replacing it by a random hashed value may not be that bad. Of course you need to store the hash in ...


3

I think you're on the right track. Just need a tiny bit more logic on the backup process, then S3 can handle the rest. Consider this: A backup is uploaded every day. If today is January 1st (for example), the backup is given a "yearly" tag. Otherwise, if today is the 1st of the month, the backup is given a "monthly" tag. Otherwise, if ...


2

You're going to have to write a script to do this unless there's a tool to do it that I can't find. If you're working with multiple aws accounts you'll need to be switching between accounts to get the bucket sizes. To get all of the buckets and their total size you can use 'aws s3api'. Using list-buckets you can get all of the buckets in one account and ...


2

Attach below policy to that user: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:ListAllMyBuckets" ], "Resource": "arn:aws:s3:::*" }, { "Effect": "Allow",...


2

Still no answers, and still no CRR monitoring in place on AWS, here's minimalistic solution, any feedback and suggestions are welcome. This monitor will only show which S3 objects are missing from destination, it will not provide visibility on versions of replicated objects. Prerequisites Enable inventory reports on src (Source) and dst (Destination) ...


2

In the CodeBuild "Artifacts" configuration, there is an "Add Artifact" button. You need to click that and reference your secondary artifact to get it to upload to S3 as well.


Only top voted, non community-wiki answers of a minimum length are eligible