topic Send alert notifications only on specific custom metric value's changes in Alerting

Send alert notifications only on specific custom metric value's changes

danielbusquets — Thu, 18 Dec 2025 13:59:50 GMT

Hi!

We've set up a custom metric that contains the name of an external entity (API Gateway) and its health status, like this:

0: unhealthy

1: healthy

Example:

We are receiveing and ingesting multiple same values over time for a given entity, for example:

Entity 1 - healthy (value: 1) -> everything OK, don't send notification
Entity 2 - healthy (value: 1) -> everything OK, don't send notification
Entity 1 - healthy (value: 1) -> everything OK, don't send notification
Entity 1 - unhealthy (value: 0) -> send notification as it is unhealthy
Entity 3 - healthy (value: 1) -> everything OK, don't send notification
Entity 1 - unhealthy (value: 0) -> don't send any notification because we've already did it
Entity 1 - healthy (value: 1) -> send a solved problem notification, as it became healthy again

So, we need to alert only on health status changes:

from 1 to 0: send new problem notification, and don't notify any other 0's received for that particular entity.
from 0 to 1: send resolved problem notification, and don't notify any other 1's received for that particular entity.

The trick here is that the email notification should be sent to the team responsible for that entity, which will be different on each case. Alerting Profile + Problem notifications doesn't seem to be able to specify variable destinations to send emails. Workflows have, but then I don't know how to send only emails when health changes and not for every redundant event.

Anyone have some advice in how can we set that up?

***UPDATE: Metric events doesn't seem to provide enough customization for that. Could be Open Pipelines the answer?***

Thanks in advance!

Re: Send alert notifications only on specific custom metric value's changes (new problem: from 1 to 0, closed problem from 0 to 1)

AntonPineiro — Wed, 02 Jul 2025 11:39:46 GMT

Hi,

This can be done using metric events.

Best regards

Re: Send alert notifications only on specific custom metric value's changes (new problem: from 1 to 0, closed problem from 0 to 1)

lubrman — Wed, 02 Jul 2025 13:23:33 GMT

Hi @danielbusquets

After setting up the metric event, you can link our event to the desired alerting profile either by tagging the metric event accordingly, or by using a custom rule within the alerting profile to directly target the problem name you’ve configured.

Re: Send alert notifications only on specific custom metric value's changes (new problem: from 1 to 0, closed problem from 0 to 1)

danielbusquets — Wed, 02 Jul 2025 13:28:06 GMT

Hi @AntonPineiro,

I've tried Metric Events, but I see no way to have the logic to fit these requirements:

API GW 1 - healthy event (value: 1) -> IGNORE (new event but it is healthy)
API GW 2 - healthy event (value: 1) -> IGNORE (new event but it is healthy)
API GW 1 - healthy event (value: 1) -> IGNORE (repeated event!)
API GW 1 - unhealthy event (value: 0) -> send notification as it is unhealthy
API GW 3 - healthy event (value: 1) -> IGNORE (new event but it is healthy)
API GW 1 - unhealthy event (value: 0) -> IGNORE (repeated event!)
API GW 1 - healthy event (value: 1) -> send a solved problem notification, as it became healthy again

Re: Send alert notifications only on specific custom metric value's changes (new problem: from 1 to 0, closed problem from 0 to 1)

danielbusquets — Wed, 02 Jul 2025 13:30:40 GMT

Hi @lubrman thanks, but the problem is exactly on how to set up the metric event or the alerting mechanism (whether is an Anomaly Detector or a Metric Event, or a Workflow, or whatever way).
Next steps to link the Alerting Profile are clear.

Re: Send alert notifications only on specific custom metric value's changes (new problem: from 1 to 0, closed problem from 0 to 1)

lubrman — Thu, 03 Jul 2025 05:34:23 GMT

Hi @danielbusquets

And a similar format doesn’t work for you when searching in the Data Explorer?
You can then specifically define the services you want to monitor — you can use the same selector in the metric event as well.

custome.api.gateway.healt.status:filter(and(or(in("api_name",entitySelector("type(api_name),entityName.equals(~"Entity 1~")"))))):splitBy("dt.entity.process_group_instance"):sort(value(auto,descending))

Re: Send alert notifications only on specific custom metric value's changes (new problem: from 1 to 0, closed problem from 0 to 1)

AntonPineiro — Thu, 03 Jul 2025 07:00:39 GMT

Hi,

Check sliding window https://docs.dynatrace.com/docs/shortlink/anomaly-detection-configuration#sliding-window

Best regards

Re: Send alert notifications only on specific custom metric value's changes (new problem: from 1 to 0, closed problem from 0 to 1)

danielbusquets — Thu, 03 Jul 2025 15:37:40 GMT

Is not doable. Shortest dealerting sliding window is 3 minute. So the unhealthy API Gateway will take 3 minutes at minimum to close the open problem.

Re: Send alert notifications only on specific custom metric value's changes (new problem: from 1 to 0, closed problem from 0 to 1)

AntonPineiro — Fri, 04 Jul 2025 08:29:07 GMT

Hi,

It means, that is how Dynatrace works. If that does not fit your use case, that cannot be done today.

Best regards