Send alert notifications only on specific custom metric value's changes (new problem: from 1 to 0, closed problem from 0 to 1)

[danielbusquets]

Hi!

We've set up a custom metric that contains the name of an external entity (API Gateway) and its health status, like this:

0: unhealthy

1: healthy

Example:

We are receiveing and ingesting multiple same values over time for a given entity, for example:

Entity 1 - healthy (value: 1) -> everything OK, don't send notification
Entity 2 - healthy (value: 1) -> everything OK, don't send notification
Entity 1 - healthy (value: 1) -> everything OK, don't send notification
Entity 1 - unhealthy (value: 0) -> send notification as it is unhealthy
Entity 3 - healthy (value: 1) -> everything OK, don't send notification
Entity 1 - unhealthy (value: 0) -> don't send any notification because we've already did it
Entity 1 - healthy (value: 1) -> send a solved problem notification, as it became healthy again

So, we need to alert only on health status changes:

• from 1 to 0: send new problem notification, and don't notify any other 0's received for that particular entity.
• from 0 to 1: send resolved problem notification, and don't notify any other 1's received for that particular entity.

The trick here is that the email notification should be sent to the team responsible for that entity, which will be different on each case. Alerting Profile + Problem notifications doesn't seem to be able to specify variable destinations to send emails. Workflows have, but then I don't know how to send only emails when health changes and not for every redundant event.

Anyone have some advice in how can we set that up? 

***UPDATE: Metric events doesn't seem to provide enough customization for that. Could be Open Pipelines the answer?***

Thanks in advance!

[AntonPineiro]

Hi,

This can be done using metric events.

Best regards

[danielbusquets]

Hi @AntonPineiro,

I've tried Metric Events, but I see no way to have the logic to fit these requirements:

API GW 1 - healthy event (value: 1) -> IGNORE (new event but it is healthy)
API GW 2 - healthy event (value: 1) -> IGNORE (new event but it is healthy)
API GW 1 - healthy event (value: 1) -> IGNORE (repeated event!)
API GW 1 - unhealthy event (value: 0) -> send notification as it is unhealthy
API GW 3 - healthy event (value: 1) -> IGNORE (new event but it is healthy)
API GW 1 - unhealthy event (value: 0) -> IGNORE (repeated event!)
API GW 1 - healthy event (value: 1) -> send a solved problem notification, as it became healthy again

[AntonPineiro]

Hi,

Check sliding window https://docs.dynatrace.com/docs/shortlink/anomaly-detection-configuration#sliding-window 

Best regards

[danielbusquets]

Is not doable. Shortest dealerting sliding window is 3 minute. So the unhealthy API Gateway will take 3 minutes at minimum to close the open problem.

[AntonPineiro]

Hi,

It means, that is how Dynatrace works. If that does not fit your use case, that cannot be done today.

Best regards

[lubrman]

Hi @danielbusquets 

After setting up the metric event, you can link our event to the desired alerting profile either by tagging the metric event accordingly, or by using a custom rule within the alerting profile to directly target the problem name you’ve configured.

[danielbusquets]

Hi @lubrman thanks, but the problem is exactly on how to set up the metric event or the alerting mechanism (whether is an Anomaly Detector or a Metric Event, or a Workflow, or whatever way).
Next steps to link the Alerting Profile are clear.

[lubrman]

Hi @danielbusquets 

And a similar format doesn’t work for you when searching in the Data Explorer?
You can then specifically define the services you want to monitor — you can use the same selector in the metric event as well.

custome.api.gateway.healt.status:filter(and(or(in("api_name",entitySelector("type(api_name),entityName.equals(~"Entity 1~")"))))):splitBy("dt.entity.process_group_instance"):sort(value(auto,descending))