We have a range of AWS/NatGateway alerts set up to monitor the health of our networking. When the cloudwatch alerts fire, this goes to an SNS/Lambda, which then fires a message to Slack. One of the alerts is an ActiveConnectionCount, which fires if the nat gateway has zero connections. However, outside of regular office hours, it is really common for zero connections. What this leads to is dozens of messages sitting in Slack on Monday morning, and there is no problem to resolve.
While this specific alert was an issue, it has led to a wider question: Has anyone found a good solution to manage cloudwatch alerting out of hours? I haven't found a way to set a time in cloudwatch to run the alerts, such as Mon-Fri 8am-6pm, i don't think it's possible. I could set a cron job to disable and enable alert actions which could just disable alerts outside office hours, but then again if something breaks at the weekend, I would want to know about it on Monday morning, so that isn't a suitable solution.
The ideal I guess would be to get one message to highlight an issue, but block all subsequent errors to cut down on noise in Slack. Would be great to hear what others are doing in this space
