Does full automation still provide a better ROI at small scales?

Question

Obviously, Automated configuration management and deployment makes sense at large scales when you need to manage hundreds of servers and you need those servers to be identically configured to a consistent standard.

However, in situations where systems are small and need only 2, 3 or 4 servers, it can often be faster in build time to simply stand up a first instance and clone subsequent instances - changing IP addresses and hostnames by hand.

The theoretical advantage to automated deployment and configuration management is that code can be deployed faster, tested faster and services are provided more reliably by automating out the problem of human error. Future upgrades of the underlying OS (eg, CentOS 6 -> CentOS 7) are better/faster and in the long run the above benefits outweigh the initial larger investment of programmatically automating deployments using a configuration management system?

Are there any real-world non-anecdotal studies and data-points to back each of the above claimed benefits to small shops with very low numbers of servers to manage?

Answer 1

There is definite value in automating manual tasks and placing systems under configuration management that is done using code, not using paper and human intervention.

One huge benefit is the reduction in the amount of rework. You can consider any problem caused by a human mistake in repeating the same sequence a case of rework. As well as any bug that needs to be fixed more than once.

Rework is costly, and there has been some research done to quantify its cost. You can read about it in this whitepaper - https://devops-research.com/roi/

The same whitepaper also quantifies the cost of failure for services/systems, and the slowness to resolve failure. All of which are greatly reduced by having a robust automated process that prevents human mistakes from ending up in production systems.

The topic of automating manual tasks has been also investigated in the State of DevOps Report that has been issued every year since 2013. Most of these are available at https://devops-research.com/research.html as well (with the exception of 2013). In every report it was mentioned as the biggest driver for excellence in high-performing organizations of all sizes. There are plenty of reasons and explanations that can be used as proof for your case.

Answer 2

The main advantage to automation isn't just automation itself, but also that it provides you with an identical configuration across all of your resources.

Say, for example, you stand up a webserver hosting your application and all relevant configs. Then you register it as an Amazon AMI so you can deploy multiple copies (say, one for dev, one for test, and one for production). You roll it out, and with how AWS works, you don't even need to change any IPs by hand.

A few weeks later you realize you need to change SSL session cache size in nginx, or users of your application can't upload large files.

Since you only have a few boxes, you do the update by hand. You do these config changes a few more times and well-document them.

Then a week later you deploy a new box (a customer is asking for a UAT environment to test their integration). You're sick, so someone else is standing up the server instead of you. Or maybe your instance in AWS dies, so you have to rebuild production in the middle of the night after a good binge-drinking session. Everything works perfectly... except for one specific config-related feature. Cue wasting two hours of figuring out why it's happening as an account manager is riding your ass, only to realize the other admin forgot a single stupid config setting when deploying a new server.

Have this happen to you a few times, and you'll want to find a way to minimize these issues.

So why not take a few days of your time and write out a simple deployment pipeline with Ansible or Puppet and put it in git? From that point on, any config changes are only a push away, and you'll rarely have to worry about a similar incident occurring.

Now, it's not peer-reviewed, APA-cited research papers you're asking for, but point me a sysadmin that's never had this happen to them, and my company will probably throw them a job offer with a pretty nice signing bonus.

Answer 3

One of the things I have the hardest time getting across to people is that the man-hours saved doing repetitive tasks is often only a small part of the value of automation. The bigger part is often hard to measure, and nearly impossible to estimate when automating something for the first time: how it changes the way you work. Talking with developers, this is a lot easier, because they know test automation (unit tests) and it's an easy comparison to make. It's things like:

• How does your workflow change when you can do X as often as you want with negligible costs and in very little time?
  • For unit tests, it means bugs get caught much, much earlier - you can test everything multiple times per day, instead of shipping something to QA once a week. Turnaround on iterations gets orders of magnitude shorter.
  • For infrastructure, it means that doing a short-lived test environment or proof of concept is a non-event, as is scaling out, as is replacing an unhealthy instance. The LoE nears zero.
• How does your workflow change when you think about automation as part of your process?
  • For unit tests, this means you design your code differently, to facilitate automation, which (often) leads to better, cleaner code.
  • For infrastructure, this encourages you to develop a standard plan for how you're going to automate things - when a new service is being developed, you know what questions you need to ask about it.
  • Developers start to change the way they develop as well, thinking about keeping stateless, graceful shutdown, fast restarts, how configuration is handled, keeping log or data file paths configurable, thinking about what paths the application user needs access to and what access it needs, etc.

The "pets vs. cattle" shift was driven in large part by automation; when creating a working server is a non-event, then servers become fully fungible and you no longer need to care about individual instances.