19 votes
Accepted

How do I work around the fact that AWS SQS is not HIPAA compliant?

According to Amazon AWS Customers may use any AWS service in an account designated as a HIPAA account, but they should only process, store and transmit PHI in the HIPAA-eligible services ...
user avatar
16 votes
Accepted

How to communicate queue-based processing delays to non-technical team members?

As with any outage communications, a non-technical reader will be primarily looking to understand: How long was it? How bad was it? Amazon CloudWatch metrics provides the following metrics for SQS ...
user avatar
9 votes
Accepted

Is it possible to set CloudWatch alarm for a combination of metrics?

For the last overall goal, I would set the alarm on ApproximateNumberOfMessagesVisible from Cloudwatch documentation on SQS this is: The number of messages available for retrieval from the queue. ...
user avatar
  • 11.1k
9 votes
Accepted

Is it possible to trigger an AWS Lambda function directly from CloudWatch alarm?

As far as I know, it is not possible. The the offcial AWS Lambda documentation says it support CloudWatch Events, but no mention of CloudWatch Alarms (and CW Events != CW Alarms) What you are ...
user avatar
  • 1,945
8 votes

Is it possible to set CloudWatch alarm for a combination of metrics?

Recently AWS just announced CloudWatch Metric Math which should be suitable for your use case. References:- Amazon CloudWatch Adds Metric Math to Enable Custom Operations on Metrics Create a ...
user avatar
  • 181
6 votes

How to ensure equitable distribution of SQS messages in a distributed systems setting?

If there are 100 messages in the queue and 5 consumers, the initial distribution will be no more than 10-10-10-10-10. A single response can never return more than 10 messages. This seems like a ...
user avatar
4 votes
Accepted

Terminate single processing server based on SQS queue condition, but keep/ reinstate autoscaling size at 1

Disclaimer: this is certainly not the best solution for backend workers, I'm just trying to be creative within the given constraints. First, you'll need to be familiar with SQS's visibility timeout (...
user avatar
3 votes

How to ensure equitable distribution of SQS messages in a distributed systems setting?

Good architecture in how you use SQS queues will solve your problems. If we assume there are, say, 3 minutes of processing per message, then you can almost guarantee equal distribution of the messages ...
user avatar
  • 506
3 votes

How to ensure equitable distribution of SQS messages in a distributed systems setting?

Long polling is always beneficial since it results in higher performance at reduced cost for the majority of use cases. Unfortunately, you cannot control the number of messages each worker receives ...
user avatar
  • 2,766
3 votes
Accepted

AWS SQS - use case for "short polling"

While you can benchmark this yourself, benchmark testing has shown that there isn't much of a difference in time between long and short polling when the queue is full. Additionally, with long polling ...
user avatar
3 votes
Accepted

Processing message queues quickly without wasting money on AWS Lambda

An SQS queue can now be wired directly to a Lambda function as an event source, so you don't have to start the Lambda function on a schedule. https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-...
user avatar
2 votes
Accepted

How to programmatically enable and disable SQS triggering AWS Lambda?

Call Lambda's UpdateEventSourceMapping API with the {"Enabled":true/false} request body. If, for some reason, you want your own API for this, you can create one in API Gateway as described ...
user avatar
  • 311
2 votes
Accepted

Is it better to perform delayed jobs using queues or at app level?

I would recommend using a scheduler to schedule job execution, rather than building logic to delay execution. Schedulers give you the added advantage of status updates. Consider using Simple Workflow ...
user avatar
2 votes

Is it better to perform delayed jobs using queues or at app level?

This sounds on the fence with opinion based but I think there's an important thing that narrow enough the problem to be answered. It is still an opinion but, as far as you're horizontally scaling, ...
user avatar
  • 11.1k
2 votes

SQS: What could high number of visible messages mean if messages received & deleted are normal?

there was no sudden surge in the number of messages received or depression in number of messaged being deleted from the queue that corresponds with this surge Yes, it looks like there was. But first,...
user avatar
1 vote
Accepted

Filtering AWS SQS Tags using JQ

Take heart, I've been struggling with JMESPath query expressions for the past couple of years! It's been a long time getting used to them. In your example output, Tags is a JSON object rather than a ...
user avatar
1 vote

AWS SNS-SQS: Checking a subscription already exists

In the end I decided to go another way. I don't think it was actually a good idea to modify things in that way when the worker boots. It can cause problems because if you have to do some emergency ...
user avatar
  • 31
1 vote

Terminate single processing server based on SQS queue condition, but keep/ reinstate autoscaling size at 1

Use AWS Automation document on state change to modify autoscaling group to 0 and return to 1 after. It should terminate and recreate.
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible