The immutable server pattern is a deployment discipline favouring the reproducibility of deployments. It is characterised by the fact that “a server that once deployed, is never modified, merely replaced with a new updated instance” and implementing this discipline requires an automation of server deployment. This automation has numerous operational advantages, one of the most important is allowing the quick and reliable replacement of failing instances in an infrastructure. This automation also implies that server deployment is described by versioned software artefacts and is subject to iterative improvements.
A popular aspect of implementations of this discipline is the removal of remote access methods to the server once it has been launched (esp. removing SSH access). Removing remote access is an easy way to ensure that the configuration of the server matches the configuration prepared by the deployment automation.
However, when investigating the causes of a software failure in a post-mortem, relying on structured monitoring is not always enough and remote access to the machine could be necessary. It is a common practical situation that server monitoring does not cover all of the failure sources, or that monitoring can be impaired by the server failure itself, which would likely be the case if the server runs out of memory or reaches its process limit.
How to implement the immutable server pattern without loosing the ability to do post-mortems?