How should we automatically rebuild immutable infrastructure when new packages are available?

Question

We're going to be using Terraform to automate our infrastructure deployment and Packer to create the machine images deployed by Terraform. By following immutable infrastructure design principles, we will implement patching by creating a new image with the patch applied and then redeploy our infrastructure.

With this setup, are there any additional tools we can use to automatically detect when a package or the OS itself in our base image needs updating and trigger the build pipeline?

Chef Automate seems close to what I'm looking for, however, it seems to scan running nodes for compliance rather than analyze the image manifest itself.

Thanks!

Answer 1

Part of adopting the Immutable Infrastructure Pattern is decomposing your system into small manageable pieces that can move through CI/CD Pipeline very quickly, this means that OS patches can be done quickly and in a controlled manner. I often see clients ending up with a halfway house where infrastructure is mostly immutable.

However, there are a few approaches to this which I have used in large-scale deployments of cloud architecture, typically I implement more than one as part of a Defense in Depth strategy:

• Security Information and Event Management (SIEM): These products, for example, LogRhythm Security Intelligence Platform and more generic products such as ElasticStack take feeds from the operating system, which includes the output from a frequent check for updates. The trick here is to get the information about what to patch quickly and automatically to inform decisions about when to roll those through your pipeline.

• Vulnerability Management systems are more tailored than SIEM in that they are only focusing on vulnerabilities across the system so could catch issues with libraries installed as part of the software deployed to the system but not managed by the Operating System. This might highlight vulnerabilities for which there is no patch (yet... hopefully).

• Dependency Check tools form part of your pipeline and can be configured to fail the build if check-ins add vulnerabilities, this also works if a new vulnerability is added to the tool since the last check-in.

Answer 2

You could do a vulnerability scan with AWS Inspector and look for CVE vulnerabilities. Inspector has a CloudWatch metric to cause an action when there are findings. Would be tricky to determine WHICH packages need to be updated, but you could just update everything. You don’t have to use Inspector, any vulnerability scanning tool you can hook into would work. Vulnerability scanning is a best practice in general.

Another idea would be to setup a cron job that does apt-get update && apt-get upgrade —dry-run > stuff-that-should-be-updated.txt and parse the text file for a list of packages to be updated and feed that back into your build process.

Answer 3

This approach looks to solve your purpose using Chef audit but it has got it's own problems. https://joshuakugler.com/using-chef-zero-audit-mode-and-packer.html

A custom solution - If you want to use a Compliance Policy ( for example , CIS, HIPPA ), create a job which will run audit parser on running package images and notify a update task upon non-complaint status. There are OVAL parser and OVAL definition available for free for above policies. For custom aduit policies you can write your own parser and audit the image.