Custom Alert Not Firing When Using Code-based Metric Query

swin12 — Thu, 30 Jun 2022 17:25:22 GMT

Hi,

I am trying to configure custom alerts for a service's processing time. These alerts are based on a calculated service metric I created that allows me to filter request data by certain dimension; request attributes.

I have configured this alarm end-to-end and gotten an alert to successfully fire under the condition I configure the alert using the 'build' option (filtering by only a single dimension value):

When I try to create a very similar alert using the 'code' option (because I need to filter on multiple values for the dimension, which doesn't seem possible using the 'build' option), I get a preview that indicates alerts should fire, but they never actually fire:

The settings for the two alarms are identical, in that:

They have identical static threshold settings
They have identical severity settings (Slowdown)

Where they differ is:

I used the 'build' option for one and the 'code' option for the other.
For the alert that uses the 'build' option, I am able to set entity scopes for the alert (such as management zone).
For the alert that uses the 'code' option, I am unable to set these entity scopes (i.e. I can't select management zone from the dropdown).

My leading theory is that the issue lies with the fact that the code-based alert does not have a management zone entity set (it doesn't seem to let me set one). As a result, the alert doesn't match the management zone for my alerting profile, and therefore I don't get an alert.

How would I associate a code-based alert with a specific management zone, such that it is picked up by the alerting profile when the alarm fires? Is there something I can add to the metric query?

Re: Custom Alert Not Firing When Using Code-based Metric Query

swin12 — Thu, 30 Jun 2022 20:41:06 GMT

The problem was that the custom alert wasn't related to the management zone condition that I had configured in my alerting profile.

I ended up removing the management zone condition from the alerting profile, instead relying on a different condition on the alerting profile; matching on the name of the custom alert. Once I did this, the alert starting sending notifications as expected.

I still don't fully understand which conditions cause a custom alert to be associated with a management zone (such that it is 'respected' by an alerting profile that has a management zone condition set). My best guess is that you must have a management zone entity filter applied in the custom alert configuration. I still don't understand how this is possible when the custom alert uses a code-based metric expression (as you seem to lose the ability to set a management zone entity filter this way).

topic Re: Custom Alert Not Firing When Using Code-based Metric Query in Alerting

Custom Alert Not Firing When Using Code-based Metric Query

Re: Custom Alert Not Firing When Using Code-based Metric Query