Backup Jenkins log files to external cloud storage

Question

I'm looking to automatically back up Jenkins log files for all jobs running on our master and slaves once a week, moving them to aws' s3 and deleting them from their respective directories when done.

My question is, is there an existing plugin that has the capabilities I need, or will I need to do this manually?

I've found the Periodic Backup Plugin but it only seems to back up locally, and is focused on config files, not log files (although I could use negative pathing to ignore everything but log files...which would be a massive pain). Likewise, ThinBackup is also config-only. I'm hoping someone might have seen something googling isn't turning up.

Answer 1

Historically, I approached this a bit differently. I added a post-build step to each job, and had it copy the output to S3 at execution time via the S3 plugin. This blog post covers the moving parts n a pretty granular detail.

You may want to reference

wget -O console-output.log ${BUILD_URL}consoleText

as the output for the job in question in your build task.

Answer 2

Corey's answer ended up inspiring me to take another look at the plugins available, this time considering anything that could be applied to jobs as a post-build step.

I found AWS CloudWatch Logs Publisher Plugin, which is a post-build step that pushes your logs to CloudWatch. Not quite the destination we originally had in mind, but it worked for us. Easy to configure too! Only downside is it doesn't currently have Pipeline functionality.

Answer 3

1. you can wrirte python script with Boto3 package to upload your logs to s3 bucket and clear the logs in master and slave.

2. Then create a job to run the python script . Make the job to run in desired time intervals.

3. you can also monitor your jenkins using some monitoring tools like Splunk where you can see the build trends in dashboards in real time and create alerts to send mail notification or trigger a jenkins jobs which cleans the log when the memory is full.