auto delete aws S3 backups

Question

I need to automate deletion of aws S3 backups ... evidently if I tag the backup with one of daily, weekly, monthly, yearly then aws will delete based on my desired retention counts per those periods however I see no easy way to determine which of those tags I give to my daily backup process which is

//   Upload backup file

aws s3 cp /tmp/backup_2007020644.20200703.1248_blobs.tar  s3://foo-bar-baz-someaccount-us-east-2/backup_2007020644.20200703.1248_blobs.tar --region  us-east-2 --only-show-errors 


aws s3api put-object-tagging --bucket foo-bar-baz-someaccount-us-east-2  --key backup_2007020644.20200703.1248_blobs.tar --tagging  --region  us-east-2 TagSet=[{Key=backuptype,Value=blobs}]

//   now lets list this s3 bucket

aws s3 ls s3://foo-bar-baz-someaccount-us-east-2 --region  us-east-2 

2020-07-01 22:55:57   31904428 backup_2007010938.20200701.2233_blobs.tar
2020-07-01 22:55:43     893239 backup_2007010938.20200701.2233_mongo.tar
2020-07-02 15:30:36   34343354 backup_2007010938.20200702.1508_blobs.tar
2020-07-02 15:30:22     893676 backup_2007010938.20200702.1508_mongo.tar
2020-07-03 01:20:04   30596405 backup_2007020644.20200703.0055_blobs.tar
2020-07-03 01:19:51     893741 backup_2007020644.20200703.0055_mongo.tar
2020-07-03 12:48:44   34658003 backup_2007020644.20200703.1226_blobs.tar
2020-07-03 12:48:30     895294 backup_2007020644.20200703.1226_mongo.tar
2020-07-03 15:05:00   34657972 backup_2007020644.20200703.1248_blobs.tar
2020-07-03 15:04:46     895279 backup_2007020644.20200703.1248_mongo.tar

Alternatively I can code up my own logic to parse above listing then issue the delete command per backup file after my code keeps track of my retention policy. There must be a better way, any advice?

Answer 1

I think you're on the right track. Just need a tiny bit more logic on the backup process, then S3 can handle the rest.

Consider this:

1. A backup is uploaded every day.
2. If today is January 1st (for example), the backup is given a "yearly" tag.
3. Otherwise, if today is the 1st of the month, the backup is given a "monthly" tag.
4. Otherwise, if today is a Sunday, the backup is given a "weekly" tag.
5. Otherwise the backup is given a "daily" tag.

Next, set up S3 Bucket Lifecycle Policies with each tag as the filter.

1. Transition "yearly" backups to Glacier after 365 days (kept indefinitely)
2. Delete "monthly" backups after 365 days (1 year)
3. Delete "weekly" backups after 183 days (6 months)
4. Delete "daily" backups after 31 days (1 month)

References:

• https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html
• https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-configuration-examples.html