The Stack Overflow podcast is back! Listen to an interview with our new CEO.
26

In March 2015, Amazon AWS announced they support S3 replication across regions. When a certain region in S3 goes offline, you can serve files from your mirror in another region. source: https://aws.amazon.com/blogs/aws/new-cross-region-replication-for-amazon-s3/ The practice of keeping your infrastructure online by doing a switch over to another region is ...


19

According to Amazon AWS Customers may use any AWS service in an account designated as a HIPAA account, but they should only process, store and transmit PHI in the HIPAA-eligible services defined in the BAA. There are ten HIPAA-eligible services today, including AWS Snowball, Amazon DynamoDB, Amazon EBS, Amazon EC2, Amazon Elastic MapReduce (EMR), ...


12

AWS Import/Export is a service that accelerates transferring data into and out of AWS using physical storage appliances, bypassing the Internet. AWS Import/Export Disk was originally the only service offered by AWS for data transfer by mail. Disk supports transfers data directly onto and off of storage devices you own using the Amazon high-speed internal ...


11

The drawback when there's replication come from the note below: Amazon S3 routes any virtual hosted–style requests to the US East (N. Virginia) region by default if you use the US East (N. Virginia) endpoint (s3.amazonaws.com), instead of the region-specific endpoint (for example, s3-eu-west-1.amazonaws.com). When you use replication you usualy let ...


10

What you are asking for is, basically, high-availability. In order to make a system highly available, you need three things: Eliminate single points of failure A mechanism to switch from an endpoint to another A way to detect failures Eliminate single points of failure In the case of S3, the point #1 is addressed, as Evgeny pointed out, by S3 cross-region ...


10

Many big companies would be at fault for not using this feature. It does add additional cost, and historically any kind of real disaster recovery solution is untested even if implemented. Other than the cost issue, companies who are actively using cross-region replication can offer a valid concern regarding the latency it takes for an object to replicate. ...


9

Resolution 1 So I solved this with the following script. I originally posted the question just in case there was an easier way that I was not aware of. #!/bin/bash aws_profile=('profile1' 'profile2' 'profile3'); #loop AWS profiles for i in "${aws_profile[@]}"; do echo "${i}" buckets=($(aws --profile "${i}" --region your_region s3 ls s3:// --recursive |...


7

Amazon's IAM roles generally grant a role access to a particular ARN (Amazon Resource Name). Amazon notes on their pages that for S3 a resource ...can be a bucket-name or a bucket-name/object-key. They also provide a helpful example for doing just this which appears as follows: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", ...


7

The AWS documentation is clear that automated key rotation is only supported by Amazon on a yearly basis. However, you can use the CLI to manually create new keys, and then use an automation technique such as cron to automate the manual process. In this case you can likely have cron run a bash script to read from a list of keys and credentials and then use ...


6

Most of the AWS SDKs mentioned support constructing a custom endpoint, for example as described here for Node.JS. Setting the endpoint to storage.googleapis.com and using your GCP access credentials should enable you to operate the Google storage backend service "like S3".


6

I think you want Amazon Snowball, it's a HD shipping service by Amazon to/from AWS. Snowball is a petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data into and out of the AWS cloud. With Snowball, you don’t need to write any code or purchase any hardware to transfer your data. Simply create a job in ...


5

You can use a remote backend as a data source. That is working well for us thus far but this project is not very mature and we will likely refactor a time or ten. data "terraform_remote_state" "network" { backend = "s3" config { // some variables related to config would likely go here } } then for the teams not managing the network to consume ...


5

Yes, you can use the AWS SDK to store objects within google storage. From Simple migration: In a simple migration from Amazon S3 to Google Cloud Storage, you can use your existing tools and libraries for generating authenticated REST requests to Amazon S3, to also send authenticated requests to Google Cloud Storage. The changes you need to make to ...


4

S3 Life cycle policy will remain and working as it is. When you use the AWS Management Console to delete an IAM user, IAM automatically deletes the following information for user: The user Any group memberships. The user is removed from any IAM groups. That the user was a member of Any password associated with the user Any access keys belonging to the ...


4

The policy will still exist. The documentation lists what is deleted when a user is deleted, any resources that the user has created will persist.


3

There's a generator for policies in IAM (Into IAM -> Policies -> Create), choosing connect gives me the following policy: { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "connect:DescribeInstance", "connect:CreateInstance", ...


3

Nope, credit goes to this SO answer, still valid today: How to upload files directly to Amazon S3 from a remote server?. But if you have a server in the cloud, ideally in AWS, it'll probably go faster to execute the copy operations on that server - higher bandwidth and lower latency to/from such server when compared to your computer.


3

You've got a couple of options: 1. Embed the source code to the CloudFormation template I do just that in my ec2-start-stop demo. You'll see file ec2-start-stop.template.yml with these lines: StartStopLambda: Type: AWS::Lambda::Function Properties: [...] Runtime: python3.6 Code: ZipFile: Fn::...


3

No. The account that uploads an object into a bucket owns the object. All the bucket owner account can do, unless granted permission by the object owner account, is delete the object.


3

AWS stores whole AMI as a single non-differential snapshot thus they charge for a full size of files in an image.


3

Yes, Resource is a required element on S3 bucket policies. If the resource is omitted, the API will respond with an error resembling this one below. status code: 400 Error putting S3 policy: MalformedPolicy: Statement is missing required element The reasoning is that each statement could apply to the bucket itself, the contents of the bucket, or some ...


2

You're going to have to write a script to do this unless there's a tool to do it that I can't find. If you're working with multiple aws accounts you'll need to be switching between accounts to get the bucket sizes. To get all of the buckets and their total size you can use 'aws s3api'. Using list-buckets you can get all of the buckets in one account and ...


2

Still no answers, and still no CRR monitoring in place on AWS, here's minimalistic solution, any feedback and suggestions are welcome. This monitor will only show which S3 objects are missing from destination, it will not provide visibility on versions of replicated objects. Prerequisites Enable inventory reports on src (Source) and dst (Destination) ...


2

You need to use... the Internet, from your data center, to access S3 directly (not via the endpoint) or an AWS DirectConnect connection to access S3 directly (not via the endpoint) or an HTTP proxy server inside this VPC to access S3 (via the endpoint) ...because S3 endpoints are not designed to be used across VPN connections. They are only accessible ...


2

Attach a policy to the source bucket Get the AWS account ID number of the destination account. From the source account, attach a policy to the source bucket that allows the destination account to get objects, similar to the following: Important: For the value of Principal, be sure to enter the AWS account ID number of the destination account. { "...


2

Download - yes. Simply set each Object ACL to be publicly readable. List - sort of. If you set the Bucket ACL to publicly readable anyone can list the content. However it's a rather human-unfriendly XML listing. You can generate your directory listings to some index.html files through an external process and configure the bucket to serve those. Basic auth ...


2

One possibility is mounting the bucket as s3fs and serving the root folder as usually done by Nginx.


2

As you point out, AWS S3 currently does not have an event type for GET calls. To do something similar to what you are trying to do, I had to do the following: Create an application/lamdba that's only purpose was to retrieve items from the S3 Bucket. This application was given exclusive privileges to the S3 bucket. The application keeps a counter of the ...


1

please attach below policy to your AWS S3 Bucket. Allow Access to Specific IP Addresses <div class="code"> { "Id": "S3PolicyId1", "Statement": [ { "Sid": "IPDeny", "Effect": "Deny", "Principal": { "AWS": "*" }, "Action": "s3:*", "Resource": "arn:aws:s3:::bucket/*",...


1

My solution was based on the one of @rombob. I used S3 and created a daily S3 inventory file of the source bucket (which is replicated). Then I created a Athena DB and table using Glue (in CloudFormation). Then I implemented an S3 event which triggers a lambda. Every time a new manifest.checksum file was uploaded (= new inventory is finished). The lambda ...


Only top voted, non community-wiki answers of a minimum length are eligible