What are immutable servers?

Question

There are some questions about immutable servers, such as:

• How to implement the immutable server pattern without loosing the ability to do post-mortems?
• What role do Configuration Managment tools play in immutable infrastructure?

It seems obvious that it has to do with servers (that part I get). And just digesting the grammar of immutable, I would think it has something to do with "not possible to mute". If that guess is close, I'd have no clue what exactly cannot be muted (and I doubt it has to do with soundcards or something ...).

My questions:

• What is actually a "immutable servers" (in the context of DevOps)?
• Why are they used?

Answer 1

Immutability is a term often used in computer science circles, which generally boils down to "not possible to change after creation". It is typically used in reference to parallelism, concurrency, and thread safety.

The discussion of that topic is fascinating, but can generally be found elsewhere on Stack Overflow. I'm resisting the urge to dive into it here. The key concept is 'not possible to change after creation'.

Imagine if, on Amazon, you deploy a web service by baking it into a Machine Image (AMI - a pre-built instance you can repeatedly re-provision). It connects to a backend database via credentials it gets out of a registry on startup. It dumps logs into a logging tool like Splunk. For normal day-to-day operation, you have no reason to ssh into this box. If you need to scale up that service, you just create more instances of that AMI and adjust a load balancer. Spinning it down is simply the destruction of instances and load balancers.

For the day-to-day operation, this box has no reason to change. We can just fire up more off of the AMI.

What happens when you need to provide an OS-level security patch? This is when you have a decision to make... do you bake a new AMI with the patch installed and redeploy all running instances, or do you ssh into existing images and update the patch? There are plenty of people who would just ssh in. The adherents of 'immutable architecture' just yelled at me for even suggesting such a thing is possible.

Immutabilists (if there is such a word) advocate baking the new ami. They advocate removing all reason to ssh into a machine ever. They advocate that any specific machine configuration should happen on the startup of that machine by pulling config details out of a repository. This is the ultimate expression of 'cattle, not pets'.

Immutable architecture is specifically about machine configurations that have no reason to change after the creation of the machine image. If something needs to change, bake a new instance image, shut down the old, bring up the new.

Answer 2

Immutable servers are servers on which no changes can be made (other than updates and security patches ideally). Instead of changing the software on the server, you spool up a new server with the desired software and then terminate the older one.

This concept helps to ensure that your test, development, and QA server are all identical, which is important for multiple reasons out of the scope of this question. Another benefit of immutable servers is the ability to rollback the application onto an older server. For example, I need to change K on production server 1, so I spool up server 2 and change K. Now after 10 minutes, I notice that K broke something with my application, rather than having to fix it right away which could take hours and potentially cause downtime for my customers, I redirect the traffic back to server 1, while I figure out what is wrong with 2.

Answer 3

Cloud technologies shifted the frontier between hardware and software so that many technical operations formerly exclusive citizen of the hardware world are also subjects of the software realm. Shared computing environments might be as old as computers themselves¹ but cloud technologies could popularise them by offering convenient and familiar metaphors to interact with them: cloud users reserve an instance, a complete computer or a mimic, whereas older shared computing environments have all possible set of unwieldy limitations and “your program have to be uploaded on that FTP server, will run in environment X (usually with 10y old version of whatever software you want to use), for at most 60 minutes” could sound familiar to former or actual users of computing centres.

The practical consequence of this shift is that deployment procedures can now be represented by software artefacts. (Deployment procedures are the instructions telling how to setup an infrastructure, with databases, web-servers or whatever belongs to this infrastructure, together with the network where they run on.) Geared with these new lenses, manual maintenance of servers pretty much looks like manual patching of production code – which is only in very rare occasions a desirable thing. Manual maintenance is susceptible to introduce discrepancies between the systems actually running in production and the code describing these systems, which in turns mean irreproducible behaviour and impossible bug analysis, dual bug fixing, and other calamities.

The immutable server pattern is just the transposition for cloud operations of the above mantra, according to which we should avoid manual maintenance of running programs. Instead of manually configuring severs, the immutable server pattern recommends to automatise this configuration.

Implementation flavours

While the general idea of the immutable server pattern is quite clear, there is a lot of implementation nuances. For instance, some approaches suggest to not update servers at all but to systematically replace servers instead. This is because updating yields situation where a deployment consists of servers having been started at several distinct times and having gone through several, distinct, updating processes which implies an inhomogeneous set of servers and can lead to subtle differences in how servers handle their jobs. A second popular variation point is the discipline regarding remote access to the servers. Some like to disable completely remote administrative access to servers, in order to guarantee that manual maintenance never happens.

History note

To the best of my knowledge, the term “immutable server” has been popularised by Kief Morris but the idea itself is much older. In 1999 FreeBSD jails already popularised the idea of fully automatising the configuration of disposable computing environments, this is how I started to implement the “immutable server” pattern many years before I heard this name to describe this technique.

Immutability, in the guise of physical immutability based on CD-ROMS, has also been a popular measure to manufacture trusted computing systems. This is not to be mistaken with the immutable server pattern.

---

¹If we don't count automatic loom tables or roller organs as computers.

Answer 4

The best explanation can be found (as always) on Martin Fowler's bliki article on Immutable Servers.

A server, be it hardware or a virtual server in the cloud, usually has an operating system and application running on it.

Often application, and components of the operating system, require configuration and require changes to be applied. For example security patches, deployment of new versions of the application and configuration changes.

When you consider that any change is a mutation on the state of the server, the term immutable starts to make more sense. It means that no mutations are allowed on such a server.

It is often the case, when people are involved in changing the state of the server - be it a deployment of a version, or configuration change, or a security path. The result is a server that is no longer working as expected. For example, the application might not run now because of misconfiguration, etc.

This is why a practice for creating immutable servers is established. With immutable servers, an image of a server is created with all the configuration, patches, application versions bundled in. Then that server image can be used to create servers in various environments.

The first environment where such an image is used, would be an environment where the image can be tested to work. Any abnormalities are detected, and only then such an image can be promoted to a production environment to replace the servers there with the new version (which is known to work well).

Once the process of creating the images and promoting the images is automated, you get a very failure-proof process that involves very little human effort and very low chance to introduce failure into your service.

Often immutable servers don't even include any way to "enter" them, such as for example the ssh server is missing. In this case it is also often the case that all the metrology of a server (metrics, logs) is shipped to systems outside such as a metrics database or log aggregation service.

With containers (see: Docker) there is also a process to create images, and then spawn these into running containers. These are quite often replaced by new containers based on updated images, and are never mutated. Meaning that no human enters into the container to "fix something" by introducing a change.

Answer 5

Let's start with the inverse, what is a mutable server?

Traditionally, a mutable server infrastructure is one that is continuously modified and updated in place. You can secure shell into it, upgrade packages, configure it, install services and deploy new code to it. This is what makes it mutable, you can mutate or modify it.

An immutable infrastructure is another infrastructure paradigm in which servers are never modified after they're deployed. If something needs to be updated, fixed, or modified in any way, new servers built from a common image with the appropriate changes are provisioned to replace the old ones. After they're validated, they're put into use and the old ones are decommissioned.

Why are they used? The benefits of an immutable infrastructure is more consistency and reliability in your infrastructure and a simpler, more predictable deployment process and it mitigates common server issues in mutable infrastructure, such as downtime from the server crashing or whatever.

But you have to know how to provision it efficiently via comprehensive deployment automations and fast server provisioning.

Imagine you are mining bitcoin, you would not want any downtime if your server crashed, you would need it back up as fast as possible, so an immutable infrastructure is supposed to be the solution.