As with any outage communications, a non-technical reader will be primarily looking to understand:
- How long was it?
- How bad was it?
Amazon CloudWatch metrics provides the following metrics for SQS queues that can help answer these questions:
- NumberOfMessagesSent: The number of messages added to a queue.
- NumberOfMessagesReceived: The number of messages returned by calls to the ReceiveMessage API action.
- ApproximateNumberOfMessagesVisible: The number of messages available for retrieval from the queue.
When graphed correctly, these metrics can be powerful visual aides in describing queue processing delays. Here are a couple of examples from an outage I experienced where the job's capacity to process queue messages was severely degraded:
NumberOfMessagesSent & NumberOfMessagesReceived
- Graph Type: Line Graph
- Statistic: Sum
- Period: 5-minutes
This graphs the contrast between messages sent and received, which helps isolate which processing component is responsible for the delay. In this graph, the received metric drops dramatically while the sent metric continues on its normal trend, so we can infer that the issue lies in the queue reading component instead of the queue writing component.
Does this answer how long/how bad event was? Yes; describes processes impacted over time.
NumberOfMessagesReceived & ApproximateNumberOfMessagesVisible
- Graph Type: Stacked Area Graph
- Statistic: Sum
- Period: 5-minutes
This graphs the queue depth on top of messages received, which helps show how far the queue backed up and how it recovered. In this graph, we can see that the queue depth backed up dramatically while the queue reading component was having issues, and began recovering when the queue reading component began reading messages again.
Does this answer how long/how bad event was? Yes; describes messages impacted over time.
Graphing Discussion
In both graphs, queue processing can generally be considered healthy when the lines overlap, and unhealthy when the lines diverge. This is an easy pattern to teach to a non-technical team member, and can help them to quickly dissemenate where and how there are problems when presented with these graphs.
To further communicate specific points in graphs, you can simply annotate them:
Graphing Tips:
- Label units and axes.
- Use consistent colors for matching metrics across graphs. Note that NumberOfMessagesReceived is orange in both graphs; this will help visualize the same metric across different graphs.
- Vertically align graphs that describe similar metrics so that they are easier to compare across time.
Note: I've formatted these graphs for presentation on StackExchange, so these aren't necessarily how I would present them in an outage post-mortem. I have explicitly removed values from the left axis here to obscure them from StackExchange; you'll want to keep them in your post-mortems.
Additional tips
- Empower Your Team: After training your team members to read these graphs, there is no reason keep them hidden away. Consider setting up a CloudWatch Dashboard and giving your non-technical team members read-only IAM access to CloudWatch, so that they can view these graphs any time.
- Set Up Notifications: Consider setting up Cloudwatch Alarms based on the ApproximateNumberOfMessagesVisible metric if it exceeds some agreed-upon high value, and subscribe team members to notify them of potential issues. Cloudwatch Alarms have description fields that are sent along with the notification emails -- make sure to include a human-readable description to help your non-technical members disseminate the alarm.
- Explore Other Data: Per Evgeny's comment, explore other data beyond what CloudWatch provides and think about how you can convey that data to your team. His example of using message lifetime in the queue to create a histogram is a great example of this creative thinking, and can be accomplished by logging both the message send and message receive times in your application. You can get the message Sent Timestamp via the SentTimeStamp Attribute on each queue message of the ReceiveMessage API response. More details here.
