Skip to main content

Questions tagged [amazon-s3]

Amazon Simple Storage Service. For questions about storing and retrieving DevOps-related data in S3, or using S3 for building a code pipeline.

Filter by
Sorted by
Tagged with
2 votes
2 answers
7k views

Amazon S3 bucket does not redirect folder URLs to index.html

I have uploaded my HUGO static site into AWS S3 Bucket. I have also added the domain in cloudrfront. The home index.html page works fine, but I cannot access my resources. For example https://www....
0 votes
0 answers
30 views

granting access to personal folder in s3

I'm trying a setup where Cognito users are granted access only to personal folder inside a common bucket. Everything works as long as I use "aws:userid" as folder name, i.e. s3://mybucket/${...
0 votes
0 answers
24 views

How to add a user to MinIO server using CLI?

I have MinIO server running in docker. I need to add a user from command line mc admin user add local ACCESSKEY SECRETKEY mc: <ERROR> Unable to add new user. Access Denied. why? How to add an ...
0 votes
1 answer
484 views

s3: reading parquet files with a subset of columns as argument; does this reduce data transfer?

I have set of large tables stored as parquets on s3. In python, I'm using: pd.read_parquet(...,columns=columns) I'm reading the files directly from s3, without any database engine whatsoever for ...
0 votes
0 answers
50 views

Create an S3 blobstore on Nexus fails ("The S3 bucket exists but you are not the owner")

Before mentioning my problem, I want to inform you that I'm using terraform and having nexus container running on EC2 orchestrated by Amazon ECS. Here is the objective: I want to create an S3 ...
0 votes
1 answer
38 views

How to design AWS infrastructure for my online course selling website?

I am designing an AWS architecture for hosting a website that sells online courses and merchandise products. For the purpose, I am planning to use the following architecture design: 1 VPC 1 ...
0 votes
1 answer
41 views

How to ascertain excesive use of my S3 bucket?

I have a S3 bucket that I use to store images for a website I manage. The bill is usually about $1/month and I store about 12.5 GB on the service. It's never really been an issue before, but recently ...
2 votes
2 answers
1k views

How can I use S3 as a filesystem

I'm not sure if it's something I'm doing wrong, or if the S3 web interface is just trash. There are a ton of basic capabilities that work in pretty much any other file storage product (Windows ...
0 votes
0 answers
34 views

How to assign different authorizations to different AWS Cognito groups for a static site served by AWS Cloudfront

We have implemented the following AWS solution in order to publish a static html site on the internet : https://github.com/aws-samples/cloudfront-authorization-at-edge How can we manage to assign ...
21 votes
6 answers
41k views

View all AWS S3 buckets and list each buckets storage used

I have multiple AWS accounts and I need to list all S3 buckets per account and then view each buckets total size. Currently, I can only view the storage size of a single S3 bucket with: aws s3 ls ...
0 votes
1 answer
1k views

calculating the size of objects in AWS S3 buckets

I ran into an problem recently, my AWS S3 bucket shows total size of 30 TB. However when i try to get the total size of individual folders, it is not summing up more than 2 TB. I used the below ...
0 votes
1 answer
257 views

When OnPrem with Kubernetes, what is the recommended way to do file storage buckets?

When you are running kubernetes on GKS, AKS, etc, you use your cloud provider specific storage bucket such as Amazon S3, google cloud storage, azure blob storage, to store your raw binary content in a ...
2 votes
1 answer
582 views

How to assign an ACL for each S3 Bucket in a tuple/list using Terraform?

I have a terraform file set to create a list of S3 buckets, and I was assigning acl to each bucket the old way: resource "aws_s3_bucket" "this" { count = length(var....
3 votes
1 answer
955 views

Use S3 To Store ConfigMap

I have a ConfigMap which stores Nginx configs and It is getting size exceeded. There are many site configs in there so I can't create multiple ConfigMaps. I thought of using S3 to store ConfigMap, is ...
2 votes
1 answer
999 views

Terraform Aws S3 - deny all users except for a specific user

I have a bucket which I need to restrict to a specific user, I have written the following script but it still seems to allow all users to operate on the bucket. resource "aws_s3_bucket" &...
0 votes
1 answer
3k views

How can I efficiently scale a data lake?

I am running into problems with AWS Athena data lakes. They were initially very quick, but as we've scaled, they're performing poorly and are becoming expensive and inefficient. What can we do to ...
0 votes
1 answer
110 views

How can I grant access to my cloud storage for clients?

I am developing a webshop for large digital products (multiple .tif image files). These are currently stored in an S3 bucket. What is a recommended way to grant access to my customers once they have ...
1 vote
1 answer
465 views

What is the proper AWS IAM policy for access to a single bucket that allows stacking?

I am working on a buildout where I am doing the following. Eventually I will need to make this scalable so I am not doing this manually each time. I have several clients dumping data into S3 buckets. ...
0 votes
1 answer
142 views

Error: error loading plugins: loading plugin /data/ipfs/plugins/go-ds-s3.so: not built with cgo support

After cloning this repo run make build (getting as output s3plugin.so) run make install make the appropriate changes in both /data/ipfs/config "Datastore": { "BloomFilterSize"...
10 votes
3 answers
3k views

Best practices for external backups for S3 buckets?

I would like to know how everyone is making external backups of S3 buckets and best practices/recommendations for this. Regardless of how good AWS is it is still a single point of failure. Especially ...
0 votes
1 answer
816 views

404 status code in response from Cloudfront for a single page react application

we have a single page app in ReactJS that is being served by CloudFront but the response status continues to be 404 for any nested pages (i.e pages other than the home page). How can this be resolved? ...
4 votes
4 answers
10k views

Is it possible to mount an s3 bucket as a point in a docker container?

I have a Java EE packaged as war file stored in an AWS s3 bucket. I will like to mount the folder containing the .war file as a point in my docker container. I have managed to do this on my local ...
0 votes
1 answer
196 views

AWS S3, as an organizer see empty bucket

I am an owner of organizer in on my aws account. When I enter S3 I see one folder, which is empty. But its being billed 8$ per month, so smth should be there. Is it possible that one account, of my ...
0 votes
1 answer
48 views

Migrating to AWS with Devops [closed]

What are they key stages of migrating to AWS with Devops ? Can anyone list a use case or workflow?
1 vote
2 answers
1k views

s3fs to mount S3 bucket with iamrole on non-aws machine

I am trying to mount S3 bucket on my on-premises CentOS-6 server using s3fs as below: [roots@myhost ~]# s3fs bucket-name /mnt/s3mnt -o uid=1001,gid=1001,allow_other,umask=022,iam_role=iam-role,...
4 votes
1 answer
215 views

auto delete aws S3 backups

I need to automate deletion of aws S3 backups ... evidently if I tag the backup with one of daily, weekly, monthly, yearly then aws will delete based on my desired retention counts per those periods ...
2 votes
1 answer
2k views

Troubleshooting VPC flow logs with an S3 bucket using SSE-KMS encryption with CMK

I'm using Terraform and trying to set up automatic export of VPC flow logs into an S3 bucket in the same AWS account and region (ca-central-1) that has default encryption turned on with AWS-KMS (using ...
1 vote
1 answer
296 views

How to make different permissions for different subdirectories of storage bucket

On Google Cloud Storage or Amazon S3, is there a way to give users a key to only a specific subdirectory of a storage bucket, for example: gs://images.eng.cm.com/avatars/<user-uuid>/xyz so the ...
1 vote
2 answers
2k views

Is there a policy that can limit the growth of an AWS S3 buckets total size?

We have an AWS S3 bucket that accepts uploads from untrusted sources which are then processed and moved out of the bucket. Is there a Policy that can limit the growth of the bucket to some ...
8 votes
5 answers
7k views

How to provide a user to access only a particular bucket in AWS S3?

I have a list of buckets in AWS S3. I have created an IAM user. I have an option to provide S3 full or read only access for a user using groups. Is there any options to provide access only to a ...
1 vote
1 answer
1k views

Error when creating new recource using backend AWS s3 [closed]

When I try create a new resource using terraform apply (Terraform init was success) received an error: data.terraform_remote_state.environment_data: Refreshing state... Error: Unable to find remote ...
17 votes
2 answers
1k views

How to order physical backup of S3 data?

I'd like to order backup of data to be copied out from S3 to a physical drive (SATA HD) and to be shipped to the address. I know it's possible using Amazon Glacier, however I cannot find such option ...
9 votes
1 answer
1k views

Is there any way I can rotate AWS KMS keys every time interval < 1 year?

According AWS KMS, keys can be rotated once every year. However, there is no way I can define a custom time period for that. However, I can keep changing the same in the bucket's policy manually. So,...
2 votes
0 answers
629 views

Allowing all apps/users access to folder - S3FS-FUSE

I'm running an application on AWS EC2 Ubuntu instance, here's what I've done: - I've mounted S3 bucket as folder with s3fs-fuse using this command (unfortunately as root : sudo): sudo s3fs ...
13 votes
2 answers
4k views

Can you use `aws-sdk` to store objects in Google Storage?

Google Storage claims they support interoperability in their solution with Amazon AWS S3 and Eucalyptus Systems APIs. source: https://cloud.google.com/storage/docs/interoperability Can the aws-sdk ...
20 votes
2 answers
3k views

Is cross-region replication 100% fool-proof for S3 region outages? [duplicate]

Amazon S3 has an option of cross-region replication which should be pretty fault-tolerant against region/zone outages. Does that mean those who are ranting about the outage did not make use of this ...
34 votes
2 answers
708 views

What is a good strategy to keep my site online when S3 goes offline?

What is a good strategy to keep my site online when S3 goes offline? If S3 US East 1 goes offline, how should I have my app configured/structured to prevent that taking my entire site offline? What ...
1 vote
1 answer
748 views

How to run aws cli with www-data?

I can successfully upload a file to my S3 bucket using the AWS cli version 1 with my normal linux user. However, I'm having trouble doing the same with Apache2's www-data. I have a personal project ...
2 votes
1 answer
1k views

For storing avatars in s3, which approach has the lowest cost?

On my site I have the ability for users to upload avatars (which they upload to s3). They can only have one avatar at the moment, so I currently just have the filename set to their user id. However, ...
0 votes
1 answer
1k views

S3 bucket policy errors

I have the following bucket policy for Amazon S3, which is giving the following error: Error: Invalid bucket policy syntax. Either it is a JSON syntax error or a Policy error. This is the policy ...
0 votes
1 answer
2k views

Can Terraform S3 remote state be kept in a different AWS account?

I am currently using Terraform with S3 remote state. The IAM user that I'm using to run Terraform exists in the same AWS account as the S3 bucket holding the remote state files. This is all working ...
2 votes
1 answer
68 views

Is it possible to load an image into S3 via an URL?

I'm going through some steps documented in How Do I Upload Files and Folders to an S3 Bucket? to get my ova into S3. Seems like a waste of time for me to download it and then upload it, is there a ...
0 votes
1 answer
741 views

s3 static hosting not redirecting my request

I followed these instructions to redirect: platform.demo.interos.net to platform.demo.interos.io The instructions use s3 for static site hosting and use the redirect option: https://aws.amazon....
4 votes
1 answer
903 views

Triggering a SNS notification when an object is accessed (downloaded) in AWS S3

I have managed to trigger events on my bucket when objects are uploaded (PUT) but I am surprised that there are no triggers called when objects are downloaded (GETs), as can be seen in this screenshot ...
1 vote
3 answers
524 views

How to make an AWS S3 bucket accessible over HTTP?

Does AWS S3 provide a way to make bucket's files available over HTTP preferably protected with BASIC_AUTH? I mean listing the files (index) and make it possible to download them over HTTP.
5 votes
3 answers
3k views

How to monitor AWS S3 cross-region replication process?

Amazon S3 has a cross-region replication which will handle copy of new/updated objects to additional region. The problem is that solution does not provide visibility on state for replication process, ...
1 vote
1 answer
756 views

How can I 'aws s3 sync' two buckets, which are located in different accounts

I'm trying to use 'aws s3 sync' on the awscli between two accounts. Account A, I own. Account B, Owned by a third party. Account B has given a user:jon on account A permission to a bucket through ...
1 vote
1 answer
533 views

How to override default aws region credentials for Terraform?

I am using Terraform to create aws resources. I practiced with a single region and with my aws cli credentials being used. worked great! however when I tried to override the region in the Terraform ....
2 votes
1 answer
2k views

Artifacts mentioned in secondary-artifacts section not getting uploaded to s3 bucket

I have created a codePipeline project through which my application get build using maven and it's artifacts are uploaded to s3. I'm trying to upload multiple artifacts as a single zip file to s3 ...
1 vote
1 answer
412 views

Why isn't my S3 Cross Region Replication working for some files?

I've followed the documentation to set up cross-region replication to another AWS account. It looked like it was working correctly until I noticed an entire prefix was missing. Here is my ...