Baking Immutable Images

Question

I am working on a project and we are trying to bake immutable images on containers as IaC. These images can have J2EE, .NET or Python apps on it. All OS patches should be applied to this image frequently and app updates should be also updated on this ready to run images. Having said that, an option we discuss includes Terraform for provisioning, Packer for building images. Many internet resources deploy additionally Ansible, Chef or Puppet into this mix. My question why Packer is not enough to cover baking images why would I need to consider Ansible/Chef/Puppet additionally? What can't Packer do and others can for these requirements? Thanks for your time and help...

Answer 1

An idea key to your question is that of immutable infrastructure, which is the idea that an image is built once, deployed many times, and never changed at runtime. If the contents of the image need to change, an entirely new image is built from scratch and new instances replace the ones running the old image.

This is the opposite of what you might now call mutable infrastructure, where often a machine (or container) is booted from a generic OS image and then its contents managed by a configuration management tool on an ongoing basis. In this model, instances generally live longer and are changed in-place to respond to new requirements.

Packer is intended as a building block for immutable infrastructure. It is intended to orchestrate the creation of customized images that boot immediately into the desired state. If immutable infrastructure is the right fit for your use-case (which is not necessarily the case!) then indeed tools such as Ansible, Chef and Puppet may be unnecessary.

However, running these tools in a "one-shot" mode (e.g. Chef Solo) with Packer can often be useful for non-trivial images: these tools generally provide more powerful abstractions than Packer alone provides, making it easier to do things such as configure the init system to run certain services, or create user accounts. In this case, the configuration management tool would run as a provisioner in Packer, creating the necessary configuration within the image, but is not involved in the ongoing upkeep of instances booted from the image.

Most practical systems include a mixture of mutable and immutable infrastructure, since e.g. immutable infrastructure is inappropriate for stateful, centralized systems. In that case, using a traditional configuration management system can be useful to create a common base of functionality between your immutable and mutable systems, applying configuration in a one-shot way for immutable images and in an ongoing, stateful way to long-running, mutable machines.

Answer 2

Image generation and distribution quickly rises to O(n^2) or higher, and as terraform is explicitly focused on instantiation it relies on external provisioners for internal state.

It will work at a smaller scale, but will greatly complicate a hybrid-cloud model and will have problems if the systems are long lived.

It also tends to cause difficulties with refactoring an iterating but these would be dependent on your use case.

For example, lets say you wanted to deploy a new version of a python module. If you had a wheel or a requirements.txt you could deploy this to cloud provider a,b,c on OS version c and d. But with fully baked images you would have to produce and test images a_c, b_c, c_c, a_d, b_d,c_d. If you have to do this per iteration the costs to developer productivity grow very quickly.

Note that you may be assuming that the images will be fully immutable, but without a configuration management tool you have no way to ensure that they are.

As containers in general right now work under a trusted model, and it is fairly trivial for any process or person who has access to launch a machine under frameworks like docker to make arbitrary changes to anything on the system (every API user is root, even on the host for docker) this is a contingency you want to account for.

If you keep the loose coupling between cloud provider, application and base OS it is much easier to iterate over time and vendor mitigation is far easier.

If you maintain a terraform configuration for GCE, AWS and Azure, a packer configuration per OS then an application deployment method separately it may increase the initial complexity but those costs will quickly amortize away over the total life cycle of the project.

I personally find a tool like ansible, which can be run without centralized infrastructure, and which can make orchestrated changes based on state to be more flexible in a pure deployment model but I have always had to retrofit in some form of configuration management at some point.

This may not directly map to your specific use case but it is a very good reason that it is a common practice.

Answer 3

I don't see the relation between packer and ansible/chef/puppet the way you do. Packer might give you a means to declare that you are building an image but if you aren't going to use something like ansible/chef/etc to load up the stuff you want in that image then how do you propose to get the stuff into the image?

Usually the default answer to this would be "well we would script it" but she'll scripts are just the glue that can stick anything together, they aren't designed specifically to handle infrastructure and configuration.

I've been working for a little while on a project where we have made the concious decision to try and write no shell scripts, at all, even where it's a little less convenient we write everything in a clearly defined, preferably declarative, language that someone else is maintaining standards for (ie. We don't need to make sure that ansible can do the things we need it to do, we let the developers do that, as opposed to scripting everything and needing to write all the detailed actions outselves).

If you consider your configuration management tools as just a nice way to declare what is actually in he base image then your off to a pretty good start if you want developers to trust the image their code will be running on.