11

I am responsible for a set of SQS queue-processing jobs with a scaling policy on the ApproximateNumberOfMessagesVisible CloudWatch metric. These jobs can fail to keep up with the amount of messages sent for any number of reasons:

  • Service degradation reduces capacity of messages able to be processed.
  • AutoScaling max limit reached while queue depth continues to rise.
  • S3 Outage impacts other dependent AWS services (AutoScaling service) that the queue processing job uses to keep up with demand.

When discussing outages with non-technical team members, I would like to communicate specific delays of queue processing that can translate into customer-visible degradations. How can I do this with SQS queues?

14

As with any outage communications, a non-technical reader will be primarily looking to understand:

  • How long was it?
  • How bad was it?

Amazon CloudWatch metrics provides the following metrics for SQS queues that can help answer these questions:

  • NumberOfMessagesSent: The number of messages added to a queue.
  • NumberOfMessagesReceived: The number of messages returned by calls to the ReceiveMessage API action.
  • ApproximateNumberOfMessagesVisible: The number of messages available for retrieval from the queue.

When graphed correctly, these metrics can be powerful visual aides in describing queue processing delays. Here are a couple of examples from an outage I experienced where the job's capacity to process queue messages was severely degraded:

NumberOfMessagesSent & NumberOfMessagesReceived

  • Graph Type: Line Graph
  • Statistic: Sum
  • Period: 5-minutes

NumberOfMessagesSent & NumberOfMessagesReceived

This graphs the contrast between messages sent and received, which helps isolate which processing component is responsible for the delay. In this graph, the received metric drops dramatically while the sent metric continues on its normal trend, so we can infer that the issue lies in the queue reading component instead of the queue writing component.

Does this answer how long/how bad event was? Yes; describes processes impacted over time.

NumberOfMessagesReceived & ApproximateNumberOfMessagesVisible

  • Graph Type: Stacked Area Graph
  • Statistic: Sum
  • Period: 5-minutes

NumberOfMessagesReceived & ApproximateNumberOfMessagesVisible

This graphs the queue depth on top of messages received, which helps show how far the queue backed up and how it recovered. In this graph, we can see that the queue depth backed up dramatically while the queue reading component was having issues, and began recovering when the queue reading component began reading messages again.

Does this answer how long/how bad event was? Yes; describes messages impacted over time.


Graphing Discussion

In both graphs, queue processing can generally be considered healthy when the lines overlap, and unhealthy when the lines diverge. This is an easy pattern to teach to a non-technical team member, and can help them to quickly dissemenate where and how there are problems when presented with these graphs.

To further communicate specific points in graphs, you can simply annotate them:

Both previous graphs with annotations.

Graphing Tips:

  • Label units and axes.
  • Use consistent colors for matching metrics across graphs. Note that NumberOfMessagesReceived is orange in both graphs; this will help visualize the same metric across different graphs.
  • Vertically align graphs that describe similar metrics so that they are easier to compare across time.

Note: I've formatted these graphs for presentation on StackExchange, so these aren't necessarily how I would present them in an outage post-mortem. I have explicitly removed values from the left axis here to obscure them from StackExchange; you'll want to keep them in your post-mortems.


Additional tips

  • Empower Your Team: After training your team members to read these graphs, there is no reason keep them hidden away. Consider setting up a CloudWatch Dashboard and giving your non-technical team members read-only IAM access to CloudWatch, so that they can view these graphs any time.
  • Set Up Notifications: Consider setting up Cloudwatch Alarms based on the ApproximateNumberOfMessagesVisible metric if it exceeds some agreed-upon high value, and subscribe team members to notify them of potential issues. Cloudwatch Alarms have description fields that are sent along with the notification emails -- make sure to include a human-readable description to help your non-technical members disseminate the alarm.
  • Explore Other Data: Per Evgeny's comment, explore other data beyond what CloudWatch provides and think about how you can convey that data to your team. His example of using message lifetime in the queue to create a histogram is a great example of this creative thinking, and can be accomplished by logging both the message send and message receive times in your application. You can get the message Sent Timestamp via the SentTimeStamp Attribute on each queue message of the ReceiveMessage API response. More details here.
  • 1
    It is also very useful to look at data from different view points, not just those provided by CloudWatch. For example if you can show a histogram of how long each message stays in the queue, showing that some messages stay for X time while others stay for X*2 time. And during outages the histogram moves its high points towards X*4 or something ... extremely powerful to see. – Evgeny Mar 6 '17 at 5:37
  • 4
    Also, just want to say: this is one absolutely amazing answer. – Evgeny Mar 6 '17 at 5:38
  • Thanks @Evgeny! That is a great idea and I've added another tip to the answer based on it, with credit to your comment. – Anthony Neace Mar 6 '17 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.