Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [amazon-s3]

Amazon Simple Storage Service. For questions about storing and retrieving DevOps-related data in S3, or using S3 for building a code pipeline.

1
vote
1answer
8 views

Why isn't my S3 Cross Region Replication working for some files?

I've followed the documentation to set up cross-region replication to another AWS account. It looked like it was working correctly until I noticed an entire prefix was missing. Here is my ...
0
votes
1answer
24 views

How to override default aws region credentials for Terraform?

I am using Terraform to create aws resources. I practiced with a single region and with my aws cli credentials being used. worked great! however when I tried to override the region in the Terraform ....
0
votes
1answer
23 views

Artifacts mentioned in secondary-artifacts section not getting uploaded to s3 bucket

I have created a codePipeline project through which my application get build using maven and it's artifacts are uploaded to s3. I'm trying to upload multiple artifacts as a single zip file to s3 ...
0
votes
0answers
10 views

AWS CodePipeline with Artifactory

I have a CodePipeline that builds a .Net core application that I would like to store in an Artifactory repository. AWS has an article that recommends using a worker EC2 instance to continuously poll ...
0
votes
1answer
17 views

compiled opencv copy to another host

Every time I spawn a new host I need to compile OpenCV on Ubuntu following official manual. The problem is that it takes about 20-30 minutes depends on the VM type. Is it somehow possible to store the ...
0
votes
0answers
21 views

Can Terraform S3 remote state be kept in a different AWS account?

I am currently using Terraform with S3 remote state. The IAM user that I'm using to run Terraform exists in the same AWS account as the S3 bucket holding the remote state files. This is all working ...
1
vote
1answer
51 views

Is it necessary to specify resource in S3 Bucket Policies?

Is it necessary to specify a resource attribute within the statements that make up an S3 bucket policy? The Access Policy Language Overview seems to suggest that the resource attribute is always ...
5
votes
2answers
326 views

Impact on policy after deleting IAM user

I have created one IAM User named "test" with S3 and Glacier Permission. Now with user "test" i created one life cycle policy to move data from S3 to Glacier at every 5 days. Now, i delete IAM user "...
1
vote
1answer
38 views

How to Allow AWS S3 Access From only Specific IPs

Recently we were testing with AWS VPC, and a requirement for our project was that we needed to allow nodes within a VPC access to S3 buckets, but deny access from any other IP address. Is there ...
0
votes
1answer
500 views

Access S3 files from Datacenter over VPN to VPC

I am trying to access S3 files from non-EC2 machine. non-EC2 machine is connected to VPC over VPN, also can reach to other hosts in VPC. Created VPC Endpoint for S3 Main routing table of VPC has ...
0
votes
0answers
167 views

How can I configure logstash-input-s3 in Dockerfile

I installed https://hub.docker.com/r/sebp/elk/ then looked at Installing Logstash plugins. I modified the example to install logstash-input-s3: FROM sebp/elk WORKDIR ${LOGSTASH_HOME} RUN gosu ...
5
votes
2answers
223 views

How to monitor AWS S3 cross-region replication process?

Amazon S3 has a cross-region replication which will handle copy of new/updated objects to additional region. The problem is that solution does not provide visibility on state for replication process, ...
3
votes
1answer
1k views

How to “inject” AWS Lambda code into a CloudFormation?

I'm exercising an idea of not using S3 bucket to refer to the code of a lambda function. The usual example of a lambda CloudFormation code might look like: MyLambda: Type: AWS::Lambda::...
0
votes
0answers
58 views

Managing Cloud Formation Templates in SCM and S3 Buckets

I have multiple projects with a CFT stored in Git. However, these CFT files are too large (exceed Amazon's 51k byte limit) and have to be stored in S3. What is an effective method that would allow ...
0
votes
0answers
15 views

Are direct download links from a private bucket possible with s3?

We deploy our application as two separate virtual appliances in OVF formats. The OVFs are about 1.5g each and we have a few other incidental installers for specific situations. Currently we are ...
1
vote
1answer
500 views

Different download speeds from AWS S3 CLI vs Console/Websites

I'm experiencing much slower traffic for the exact same files when downloading (or uploading) them via the AWS website vs. downloading them via the CLI. Is there a known reason for this? Is there ...
1
vote
0answers
21 views

Converting s3 URL to the desired one

I have created a s3 bucket with all permissions and user access through IAM policies. I give the below URL to the customer to access that bucket https://s3.console.aws.amazon.com/s3/buckets/s3-upload-...
0
votes
1answer
29 views

Copying a file with special character & spaces

I have been trying to copy a file from my remote server to AWS s3, but the file name is quite weird and I am not able to figure it out what can I do. I tried using escape characters but that did not ...
0
votes
0answers
24 views

Managing Security in an S3 Data Lake

I am creating a design proposal for a data lake hosted in S3. My goal is to create a flat data lake that will host a variety objects within it. The majority of these objects will be made available to ...
1
vote
0answers
148 views

How Can I Easily View Build Artifacts From AWS CodePipeline / CodeBuild?

I'm creating a project with the nodejs + AWS Lambda + express template in AWS CodeStar, and I'm using CodePipeline and CodeBuild to build and deploy the project. It also runs tests, and it would be ...
-2
votes
1answer
145 views

Amazon AWS - Amazon Connect IAM policies

I was trying to lock down user account just to Amazon Connect, Lambda and S3. I was able to give permissions to S3 and Lambda but I cannot find permissions for Amazon Connect. I have went through the ...
1
vote
1answer
35 views

Is it possible to load an image into S3 via a URL

I'm going through some steps to get my ova into S3. Seems like a waste of time for me to download it and then upload it, is there a mechanism to import an S3 object from a web resource? I have a UCS ...
3
votes
1answer
1k views

How to apply using Terraform to launch multiple EC2 Resource with different configs (VPC not maintained by TF)

We are planning to use Terraform for provisioning EC2 instance which will host Oracle DB. While we are able to achieve this, there are some questions for which I am searching for suggestions / ...
1
vote
1answer
31 views

AWS S3 ACLs across accounts

I have two AWS accounts: A and B (viewable from https://console.aws.amazon.com/organizations). I have given an S3 bucket in account A a policy that allows account B to do everything in that bucket. ...
2
votes
1answer
611 views

Best practices for external backups for S3 buckets

I would like to know how everyone is making external backups of S3 buckets and best practices/recommendations for this. Regardless of how good AWS is it is still a single point of failure. Especially ...
1
vote
1answer
226 views

AWS Redshift UNLOAD not firing S3 PUT event

I am trying to use Redshift to UNLOAD data to S3 and then pick that data up with a Lambda function and process it. All of the constituent parts (the UNLOAD, the Lambda function) now work, however the ...
9
votes
4answers
15k views

View all AWS S3 buckets and list each buckets storage used

I have multiple AWS accounts and I need to list all S3 buckets per account and then view each buckets total size. Currently, I can only view the storage size of a single S3 bucket with: aws s3 ls ...
6
votes
4answers
2k views

How to provide a user to access only a particular bucket in AWS S3?

I have a list of buckets in AWS S3. I have created an IAM user. I have an option to provide S3 full or read only access for a user using groups. Is there any options to provide access only to a ...
4
votes
1answer
410 views

AWS Storage Cost for a custom AMI

How does AWS calculate the cost of a custom AMI? If I would create an AMI which is based on Ubuntu 16.04, do I need to pay for the complete size of AMI (e.g. 4GB) or just the differences of my custom ...
10
votes
2answers
1k views

Can you use `aws-sdk` to store objects in Google Storage?

Google Storage claims they support interoperability in their solution with Amazon AWS S3 and Eucalyptus Systems APIs. source: https://cloud.google.com/storage/docs/interoperability Can the aws-sdk ...
7
votes
1answer
565 views

Is there any way I can rotate AWS KMS keys every time interval < 1 year?

According AWS KMS, keys can be rotated once every year. However, there is no way I can define a custom time period for that. However, I can keep changing the same in the bucket's policy manually. So,...
19
votes
2answers
2k views

Is cross-region replication 100% fool-proof for S3 region outages?

Amazon S3 has an option of cross-region replication which should be pretty fault-tolerant against region/zone outages. Does that mean those who are ranting about the outage did not make use of this ...
21
votes
1answer
382 views

How do I work around the fact that AWS SQS is not HIPAA compliant?

I have a use case where data from S3 is queued into AWS SQS, which is in turn connected to CloudWatch, whose metrics will be triggering AWS Lambda. However, I want the architecture to be HIPAA ...
30
votes
2answers
496 views

What is a good strategy to keep my site online when S3 goes offline?

What is a good strategy to keep my site online when S3 goes offline? If S3 US East 1 goes offline, how should I have my app configured/structured to prevent that taking my entire site offline? What ...
15
votes
2answers
200 views

How to order physical backup of S3 data?

I'd like to order backup of data to be copied out from S3 to a physical drive (SATA HD) and to be shipped to the address. I know it's possible using Amazon Glacier, however I cannot find such option ...