We changed our privacy policy. Read more.
16

As with any outage communications, a non-technical reader will be primarily looking to understand: How long was it? How bad was it? Amazon CloudWatch metrics provides the following metrics for SQS queues that can help answer these questions: NumberOfMessagesSent: The number of messages added to a queue. NumberOfMessagesReceived: The number of messages ...


9

General ways to track why a process in Linux failed are good. One such way is to run a process using strace which will tell you the system calls process did and usually point to the reason for a failure. You can create a Dockerfile that looks something like this: FROM original_image RUN apt-get -y update && apt-get install -y strace # build with `...


9

First of all, removing ssh on an immutable server doesn't guarantee there'll be no change, it's more that as there should be no need to change something you reduce the attack surface by removing a remote access channel. One way to keep a sort of post-mortem is log centralisation. There's a myriad of methods to achieve it, ELK stack, Splunk, syslog... ...


7

The fact that you don't have SSH access doesn't mean there is no way to access the machine. Most likely you'll be running it on some cloud operator, where you can also do the following: take a snapshot of the machine. You could simply take a snapshot of the box before destroying it, for later analysis. access the machine through the console. You'll probably ...


3

As far as I am aware, commit and run are the best options here to give you full access to the container as it was when it died. Ideally, your container would spit out some more useful information when it fails, but that is another topic altogether. Edit: to expand my answer, if the container is dying right on start, you can also use docker run to specify ...


3

Yes, you should include names. "Blameless" postmortems do need to have detailed information and not be bowdlerized. Let me quote from one of the most definitive sources on blameless postmortems, the Etsy blog post that started it all. A Blameless Post-Mortem What does it mean to have a ‘blameless’ Post-Mortem? Does it mean everyone gets off ...


3

Usually for a blameless postmortem, the best idea is to go further than the human error (which for a proper 5 whys should not arise, that's rule 11, but we're just humans :)) and complete the 5 Whys with the 5 Hows. To follow on this particular case for a blameless postmortem I'd go few step further whith those iterations (For the sake of the example) of ...


1

So, the second example is clearly better - but surely they're missing the whole point of blameless culture? Firstly, it's not relevant who did it or if anyone really wants to follow the link and discover the person's name. People make mistakes and forget things. More importantly, nothing you've written is the root cause! Dig deeper into the "why"s. I'd ...


Only top voted, non community-wiki answers of a minimum length are eligible