AWS CloudWatch Alarms includes most of the features you would expect in a monitoring solution. However, the ability to set up blackout windows to pause alarms is sorely lacking.
Popular solutions includes relying on lambdas - to enable / disable the alarm, or to act as a gateway for the alarm. Here’s a solution which does not use lambdas.
Step 1 — Specify a Blackout Window
Set up a blackout window using CloudWatch Event Rules schedule cron expressions. For instance, this cron expression triggers every minute from 8 am UTC to 8:59 am UTC.
0/1 8 * * ? *
Step 2 — Publishing the Blackout Window
You now need the black window to generate a metric. The easiest way to do this is to push to a SNS topic blackhole - you don’t need to do anything with the message. Set up a
breakfast-blackout-window SNS topic as a target for the rule.
And voila! You now have the Invocations metric for the rule during the blackout window.
Step 3 — Pausing the Alarm During the Blackout Window
Now edit your alarm metric. Add the blackout rule Invocations metric to the graphed metrics. Then add a new Math expression utilizing your original metric (say m1) and your blackout metric (say m2)
IF(FILL(m2, 0) > 0, 0, m1)
The expression is pretty straightforward — during the blackout window (that is - when the blackout rule has invocations), we force a value within the alarm limits. In all other cases, use the actual metric.
And that is all!
If 0 triggers the alarm, you will have to replace the 2nd parameter to IF with a value that doesn’t trigger the alarm.
AWS CloudWatch Alarms are an easy way to get started on operational monitoring. You might want to get a full on monitoring solution as your application matures though.
The ability to set up blackout windows is a good feature that should be built into AWS. Till then, this hack should get you through.