19
votes
Accepted
How do I work around the fact that AWS SQS is not HIPAA compliant?
According to Amazon AWS
Customers may use any AWS service in an account designated as a HIPAA
account, but they should only process, store and transmit PHI in the
HIPAA-eligible services ...
- 8,963
16
votes
Accepted
How to communicate queue-based processing delays to non-technical team members?
As with any outage communications, a non-technical reader will be primarily looking to understand:
How long was it?
How bad was it?
Amazon CloudWatch metrics provides the following metrics for SQS ...
- 1,484
10
votes
Accepted
Is it possible to trigger an AWS Lambda function directly from CloudWatch alarm?
As far as I know, it is not possible. The the offcial AWS Lambda documentation says it support CloudWatch Events, but no mention of CloudWatch Alarms (and CW Events != CW Alarms)
What you are ...
- 1,955
9
votes
Is it possible to set CloudWatch alarm for a combination of metrics?
Recently AWS just announced CloudWatch Metric Math which should be suitable for your use case.
References:-
Amazon CloudWatch Adds Metric Math to Enable Custom Operations on Metrics
Create a ...
- 191
9
votes
Accepted
Is it possible to set CloudWatch alarm for a combination of metrics?
For the last overall goal, I would set the alarm on ApproximateNumberOfMessagesVisible from Cloudwatch documentation on SQS this is:
The number of messages available for retrieval from the queue.
...
- 11.3k
6
votes
How to ensure equitable distribution of SQS messages in a distributed systems setting?
If there are 100 messages in the queue and 5 consumers, the initial distribution will be no more than 10-10-10-10-10.
A single response can never return more than 10 messages.
This seems like a ...
- 805
4
votes
Accepted
Terminate single processing server based on SQS queue condition, but keep/ reinstate autoscaling size at 1
Disclaimer: this is certainly not the best solution for backend workers, I'm just trying to be creative within the given constraints.
First, you'll need to be familiar with SQS's visibility timeout (...
- 1,338
3
votes
How to ensure equitable distribution of SQS messages in a distributed systems setting?
Long polling is always beneficial since it results in higher performance at reduced cost for the majority of use cases. Unfortunately, you cannot control the number of messages each worker receives ...
- 2,826
3
votes
How to ensure equitable distribution of SQS messages in a distributed systems setting?
Good architecture in how you use SQS queues will solve your problems. If we assume there are, say, 3 minutes of processing per message, then you can almost guarantee equal distribution of the messages ...
- 516
3
votes
Accepted
AWS SQS - use case for "short polling"
While you can benchmark this yourself, benchmark testing has shown that there isn't much of a difference in time between long and short polling when the queue is full. Additionally, with long polling ...
- 2,707
3
votes
Accepted
Processing message queues quickly without wasting money on AWS Lambda
An SQS queue can now be wired directly to a Lambda function as an event source, so you don't have to start the Lambda function on a schedule.
https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-...
- 805
2
votes
Accepted
Is it better to perform delayed jobs using queues or at app level?
I would recommend using a scheduler to schedule job execution, rather than building logic to delay execution. Schedulers give you the added advantage of status updates. Consider using Simple Workflow ...
- 46
2
votes
Is it better to perform delayed jobs using queues or at app level?
This sounds on the fence with opinion based but I think there's an important thing that narrow enough the problem to be answered.
It is still an opinion but, as far as you're horizontally scaling, ...
- 11.3k
2
votes
Accepted
How to programmatically enable and disable SQS triggering AWS Lambda?
Call Lambda's UpdateEventSourceMapping API with the {"Enabled":true/false} request body.
If, for some reason, you want your own API for this, you can create one in API Gateway as described ...
- 311
2
votes
SQS: What could high number of visible messages mean if messages received & deleted are normal?
there was no sudden surge in the number of messages received or depression in number of messaged being deleted from the queue that corresponds with this surge
Yes, it looks like there was.
But first,...
- 805
1
vote
Accepted
Filtering AWS SQS Tags using JQ
Take heart, I've been struggling with JMESPath query expressions for the past couple of years! It's been a long time getting used to them.
In your example output, Tags is a JSON object rather than a ...
- 171
1
vote
AWS SNS-SQS: Checking a subscription already exists
In the end I decided to go another way.
I don't think it was actually a good idea to modify things in that way when the worker boots. It can cause problems because if you have to do some emergency ...
- 31
1
vote
Terminate single processing server based on SQS queue condition, but keep/ reinstate autoscaling size at 1
Use AWS Automation document on state change to modify autoscaling group to 0 and return to 1 after. It should terminate and recreate.
1
vote
Is it possible to trigger an AWS Lambda function directly from CloudWatch alarm?
It is possible to trigger Lambda function from CloudWatch alarm, by using EventBridge https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch-and-eventbridge.html
- 11
Only top scored, non community-wiki answers of a minimum length are eligible