3

We want our post-mortems to be blameless.

Recently someone included this statement in a post-mortem report:

On DATE, PERSONMAME did code a fix (link), but forgot to merge it to master

It was suggest that to be "blameless" we shouldn't include PERSONNAME. The suggested replacement was:

On DATE, the code fix (link) was created but the engineer failed to merge it to master.

In a blameless culture, nobody should be worried about having their name listed in a report. In the first example, it feels blame-full to point out their name.

On the other hand, in the second example, someone could follow the link and see who the engineer was. This seems to be hiding someone's name without really hiding it. It feels like something we'd do in an organization that is having a problem with blame/shame.

What should we do?

  • 1
    IMHO it's not a matter of hiding the name - anyone with a bit of skill and knowledge of your process and tools can find that name (maybe just by following that link, for example). What matters is what happens (if anything) even if the name is shown. – Dan Cornilescu Jul 26 '18 at 2:20
2

Usually for a blameless postmortem, the best idea is to go further than the human error (which for a proper 5 whys should not arise, that's rule 11, but we're just humans :)) and complete the 5 Whys with the 5 Hows.

To follow on this particular case for a blameless postmortem I'd go few step further whith those iterations (For the sake of the example) of How possible follow up interpretation (possible answers):

  • How this fix has been left unmerged after creation ? (it wasn't reviewed and didn't follow the usual path)
  • How did it miss review ? (That was urgent and straightforward, review wasn't necessary and decision was taken to deploy)
  • How this process has been defined ? (It wasn't)
  • How was the bug report closed without merge (No validation/no defined process for this case)
  • How to prevent the problem arise again ? (Define the process and set validation/review even for quick patches)

Now you don't have someone who have forget to merge something, you have an analysis of how it was possible and more important, you have something actionnable to improve which is the main, if not the sole, reason to do a postmortem.

A more detailled criticism of the 5 whys (which as you show end often with a Who instead of a Why) can be found on O'reilly's blog post The infinite hows

To address the specific point of naming in the analysis, I'd argue neither the name nor the fix itself are of value, wording the step in the analysis as "An engineer wrote a fix and didn't merge it" without any link would be enough for the report.

  • I love the enthusiasm you have for the topic of post-mortems and I completely agree about using "how" instead of "why" (I say something similar in my own writing). That said, I'm trying to focus on the question on whether or not names should be used in the report. Could you focus on that? – TomOnTime Jul 25 '18 at 21:27
  • Well, your question from an external reader point of view seems to present the line as the root cause and not just a step, for the sake of completeness I think that's a good idea to have the whole picture in an answer, the last paragraph address the point I think, with a clear No. – Tensibai Jul 25 '18 at 21:37
  • Good feedback! I’ll revisit the question. – TomOnTime Jul 25 '18 at 21:38
2

Yes, you should include names.

"Blameless" postmortems do need to have detailed information and not be bowdlerized. Let me quote from one of the most definitive sources on blameless postmortems, the Etsy blog post that started it all.

A Blameless Post-Mortem

What does it mean to have a ‘blameless’ Post-Mortem?

Does it mean everyone gets off the hook for making mistakes? No.

Well, maybe. It depends on what “gets off the hook” means. Let me explain.

Having a Just Culture means that you’re making effort to balance safety and accountability. It means that by investigating mistakes in a way that focuses on the situational aspects of a failure’s mechanism and the decision-making process of individuals proximate to the failure, an organization can come out safer than it would normally be if it had simply punished the actors involved as a remediation.

Having a “blameless” Post-Mortem process means that engineers whose actions have contributed to an accident can give a detailed account of:

  • what actions they took at what time,
  • what effects they observed,
  • expectations they had,
  • assumptions they had made,
  • and their understanding of timeline of events as they occurred.

…and that they can give this detailed account without fear of punishment or retribution.

The ENTIRE POINT of blameless postmortems is not to hide names, but so that people can boldly state what they did in the open without fear of punishment or retribution.

1

So, the second example is clearly better - but surely they're missing the whole point of blameless culture?

Firstly, it's not relevant who did it or if anyone really wants to follow the link and discover the person's name. People make mistakes and forget things.

More importantly, nothing you've written is the root cause! Dig deeper into the "why"s.

I'd speculate that the root cause is that you have systems and processes where it can happen in the first place, and when it does, it's not caught in time. Could the merge/PR process be improved? Would more reviews help? Should each merge be matched against your bug list and vica-versa?

Edit: If that's just one line of a six page document then, I assume you're not naming said person as the root cause. So, fairer. But I'd disagree that "hiding" his name is something a company having blame/shame problems would do. They shouldn't include the name because, one also assumes, they don't want to start having a blame/shame culture?

  • 1
    Root cause is immaterial to this question: that was one line out of a 6-page document. – TomOnTime Jul 25 '18 at 21:19
0

We also struggled with this balance. Having a name attached to certain records does tend to reveal some process faults. Things like:

  • If a given person did not do something that they arguably should have
  • What role a person played through the entire post mortem
  • What a given persons contributions to each process in the post mortem were

All of which is useful when attempting to process change. For example, it might be that a given person did not know that they were responsible for ensuring the safety of a system, or a person did not feel that they were adequately qualified to solve a problem.

The thing is, to understand the problem it does not need to be allocated to a specific person - just to the same person throughout the duration of the report. Accordingly, we do not name people, but rather their role in their role in the management process:

  • at 10:24:22 "John" reported that the system was inaccessible → At 10:24:22 the project owner reported the system ...
  • at 10:38:00 "Bill" restarted MySQL to attempt to free the socket pool → at 10:38:00 Operator A restarted MySQL ...

This allows this sort of internally consistent reference, and we can evaluate the role each team member played. It additionally means that "gut reaction" to each team members failure is lost.

We do keep a key as to who is what, at the bottom of the report. But it's so we can ask for more information, and has no bearing on the reports content.

0

In the post-mortems I've taken part in, one of the components of the report is:

What happened, and when did it happen?

Which includes a timeline. Timelines are incredibly useful for debugging process faults and providing context to people reading it long after it happened. Since humans are responding to whatever happened (if not kicked it off entirely), there will be references to humans in it. However, indirections that mention a role like an operations engineer are preferred.

Depending on the severity of the incident undergoing review, there may be chat-logs, tickets, and code-commits attached to the post-mortem as artifacts. Those will by definition name-names. The point of the blameless post-mortem is to keep the tone of it from assigning blame directly, and to avoid where possible assigning it indirectly. Indirect blaming is entirely cultural, though.

You can't stop people from assigning blame in their heads, but you can avoid assigning blame in the report. Also, by not including extensive membership in the post-mortem archive during performance reviews. If someone wants to assign blame, make them work for it.

The other area where names sometimes come up in the report itself is in the short and long descriptions of what happened.

NAME merged to master before authorized, triggering a CD pipeline. While attempting to prevent the unauthorized change from reaching production, NAME caused the CD system to be mistakenly recycled stopping all deploy pipelines for six hours.

Which is clearly blamey. Compare this to:

A Release Manager merged to master before authorized, triggering a CD pipeline. The mistake was realized immediately. While attempting to prevent the change from deploying, the CD system was destroyed by accident. Restoration by the CD team took six hours.

A certain kind of nerd will notice that this is very passive-voice. You can tell those nerds that this is very much intentional. Passive-voice has been the voice of deflecting blame for a very long time now, so go ahead and passive it up.

Mistakes were made.

Yes, indeed. And which nebulous, not well defined people were involved in this?

A Release Manager.

Doing what, vaguely?

Merging to the master branch of a project.

And what, unspecifically, did they do in response to this?

Broke the deploy system.

What, generally, was the impact of this?

The deploy pipleine system was unavailable for a period of about six hours.

And how was this rectified?

The CD team rebuilt the pipelines.

Very good. Now give me some action items.

  • A process should be put in place to prevent merges to master before authorization.
  • A circuit-breaker should be introduced in the deploy pipelines to allow a build to be safely blocked.

Thank you.

0

Short answer: Keep things open, transparent and fun.

Long Answer:

You have to use names in postmortems. Postmortems are essential and meant to discover the root cause of problems. I have been involved in many postmortems where someone was named and it turned out that that named person was not the culprit and talking to the named person was critical to discover the real cause of the problem. Without the name of the person, this would not have been possible. So you have to use names.

As long as humans are involved in any process, mistakes will be made. So its better to admit that up front, share a laugh about it often, and keep your team in a open and transparent environment where someone can find out names if they needed to and not be afraid or timid to approach anyone.

If names were stricken from the report, you are kidding yourself if you think names will not be mentioned during post meetings discussions. This type of non discussion of names can cause "whisper campaign" effect when the person is not around and can cause unneeded gossip and will poison the environment.

If management requires no names on the postmortem, names will still be used in discussion to verify facts and follow ups.

An example of keeping it fun for everyone: At a past company, in a .com era start up, we had a lively group of coders(~15), devops(~10) and 20 others in a large open office environment and everyone shared a similar sense of humor. We had an orange traffic cone that the developers referred to as the "cone of shame".

If someone did not merge correctly, or their changes broke the build or regression testing wasn't all green, the build master would make a judgement on the spot and leave the cone of shame on the monitor of the offender. A cone on top of the monitor was visible to everyone as you walked into the office.

We would all come in the next day and see the cone of shame and share a laugh. Often times the build master had chosen the wrong person and the break was caused by another person and the cone was passed around. It was proudly wore on your head and seen in a casual office water cooler discussion that day. It got the team talking, interacting and laughing which is all good for team chemistry.

You have to keep it open and part of that is not scared of using names and really knowing your team and keeping a fun culture.

0

Professionals are responsible. Name names.. It sounds a bit hipster.. but we issue a special award for mistakes. Only in this way can mistakes be corrected promptly. The problem is not blame.. the problem is potential retribution. If your company is in the habit of punishing people for mistakes then you should leave. Companies that behave like that eventually find it very difficult to hire professionals. What you need to foster is an environment where people own up to.. and fix their mistakes. Blaming is not personal.. it's transparency and done correctly promotes camaraderie.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.