12

Recently the Amazon S3 had a major outage in the us-east-1 region. It looks like it was likely caused by a spelling error when running a maintenance playbook in Ansible or a similar tool. You can put a shell script wrapper around ansible-playbook to look like:

#!/bin/bash
/usr/bin/ansible-playbook "$@" --list-hosts --list-tasks
read -p "Are you sure? (y/n) " answer
test "$answer" = "y" || exit 0
exec /usr/bin/ansible-playbook "$@"

But what are some other ways you use to improve the safety and reduce a chance of error causing a major outage for your company.

6
  • 1
    I'm voting to close this question as off-topic because it will be more suitable for unix.stackexchange.com or superuser.com Mar 4, 2017 at 6:55
  • 4
    Infrastructure as code, is one of the key components to get to hundreds of deployments a day. Being able to secure the tools that provide this speed from creating major outage in the operations seems to me like a relevant topic. I might be wrong, of course. I do appreciate your view though. Would you like to join this discussion about on and off topic questions in Meta? Mar 4, 2017 at 7:02
  • For example this question seems to be accepted as on topic: devops.stackexchange.com/questions/98/… Mar 4, 2017 at 7:09
  • Jiri, do you make difference between your and other question you mention? Mar 4, 2017 at 7:16
  • 5
    If these kinds of questions would be suitable for superuser, there would be no need for devops.se. This is definitely on topic here. Operations and mitigating human error are in the core of DevOps. Mar 4, 2017 at 8:57

3 Answers 3

6

We're using jobs in jenkins to trigger deployments. It ensures that no matter who does the deployment, the ansible command that is run will be the same. A nice bonus is the build logs record when deployments were triggered, who triggered them and what exactly happened during the deployment.

It's certainly not foolproof, but it's been a nice improvement over running ansible playbooks by hand.

For larger/riskier changes this should ideally be combined with some form of change management so changes are made only after another person/team reviews the change and the approach to the change to help identify and resolve potential issues early.

Additionally it never hurts to have a teammate who understands the change you're making be present and watching while you make big changes so they can watch for and help prevent mistakes in the execution of the change.

4

Categories of errors

There are two ways of looking at human factors that lead to problems and accidents:

  1. You can see the human error as the cause of a mishap. In this case "human error," under whatever label—loss of situation awareness, procedural violation, regulatory shortfalls, managerial deficiencies is the conclusion of your investigation.
  2. You can see the human error as the symptom of deeper trouble. In this case, human error is the starting point for your investigation. You will probe how human error is systematically connected to features of people's tools, tasks, and operational / organizational environment.

The first is called the human approach, and the second as the system approach.

To explain failure using the human approach, you would seek failure and find people's inaccurate assessments, wrong decisions or bad judgments.

To explain failure using the system approach, you are not trying to find where people went wrong. Instead, find how people's assessments and actions made sense at the time, given the circumstances that surrounded them.

For example, Donald Berwick of the Institute for Healthcare Improvement (IHI) argues that improving patient safety requires changes in the design of systems:

...We are human, and humans err. Despite outrage, despite grief, despite experience, despite our best efforts, despite our deepest wishes, we are born fallible and will remain so. Being careful helps, but it brings us nowhere near perfection... The remedy is in changing systems of work. The remedy is in design. The goal should be extreme safety. I believe we should be as safe in our hospitals as we are in our homes. But we cannot reach that goal through exhortation, censure, outrage, and shame. We can reach it only by a commitment to change, so that normal, human errors can be made irrelevant to the outcome, continually found, and skillfully mitigated.

Donald M Berwick. Not again! BMJ 2001


Removing mistakes from the system

A great way to find (and correct) the various ways failure happens after the fact, is searching for the root cause without blaming the people. This is often called "blameless post-mortems", and Etsy Code as Craft blog post expands on the concept. People at Etsy presented and wrote more about it on other forums and blogs.

To prevent mistakes in the first place, some culture traits are a must. Procedures and various artifacts created in the system must test that using them by humans is very clear and self-explanatory. Often those who create are not those who consume, leading to a disconnect and lack of clarity. The system is then not safe to operate since the only person who knows all the assumptions is the one who created it (and no one else).

Effective control measures

Put in place effective control measures to stop the process when an error occurs. This is mistake-proofing. Effective control measures are design changes that prevent or stop processes from continuing when an error has occurred by introducing a process failure

Example:

In 1896, Sakichi Toyoda invented Japan’s first power loom called “the Toyoda Steam power loom.” This development increased productivity by twenty times, and the quality of the textiles improved and caused a revolution in the textile industry in Japan. But here’s the subtle but very important discovery and principle:

when the needle broke, the machine stopped

Sakichi Toyoda created an innovation to the Loom that would later become one of the pillars in the Toyota Production System (Lean). That pillar we now call Jidoka, sometimes called “smart automation with a human touch” or “autonomation.”

In large part, Andon (stop at first defect) and Poka-Yoke (mistake proofing) are later developments that find their influence from the Loom.

Removing Single-Point Weaknesses

The term single-point weakness refers to the creation of redundancies in the system as an approach to improving system reliability. Redundancy is created by increasing the number of systems or individuals involved in the process. Having more backup systems or more checks (double, triple, or more) increases the probability that the process will proceed correctly.

One great example for this is the "four-eyes principle," which means that "all business decisions and transactions need approval from the CEO and CFO. Since the CFO is not reporting to the CEO, there is an independent controlling mechanism in place".

source: https://en.wikipedia.org/wiki/Two-man_rule

Make Hazards Obvious

If hazards are made obvious, or impossible to reach, humans cannot create mistakes. For example, color-coding is a common approach to making mistakes more obvious. Or if you think of various computer sockets that can only be inserted one way and not the other, etc.


Some great books are talking about the subject, and it would not be a good answer without mentioning them:

6
  • 1
    A very important method you do not mention is the “four-eyes principle” which is used in the finance – either as a regulatory obligation or as a safe-guard. In the software industry it is implemented in various ways, as e.g. code reviews but also can be used to validate commands affecting live systems. Mar 4, 2017 at 11:27
  • I'll add that to the SPW principle. Mar 4, 2017 at 11:29
  • 1
    Great discussion about errors, but it doesn't say how to safeguard against accidental deployments.
    – Alexandre
    Mar 4, 2017 at 13:03
  • 1
    The question asks about Ansible specifically. This answer is very thorough and well-researched but it is one step removed from the real-world problem.
    – Woodland
    Mar 5, 2017 at 0:13
  • 1
    @Evgeny When I replied to your AWS Lambda performance question, at first I didn't say how to run your tests and you pointed that out. You were right, and I adjusted my answer. I understand people that are down voting your answer here. Your answer would be good for a question about "How to approach and reduce errors in our workplace?". Here, OP has a question about Ansible and you don't even mention it. Worse, OP gives an indication on the kind of solution he's looking for, and you are going the other way. Your answer is great (really), but not for this question.
    – Alexandre
    Mar 5, 2017 at 6:39
1

As @bradim said using your CI/CD tool to initiate the deployment instead of hand based commands is usually good step forward, as is adding tests in your pipeline that actually test your deployment scripts on your staging (or a freshly created) environment, where you can pick up bugs earlier.

I would also add that instead of calling your ansible scripts directly, you can also add tools like Ansible Tower into your flow, which will let you track the changes that have been run more easily, and can give you an additional step of security into your flow.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.