topic Re: Help needed with Custom Alerting in Alerting

Help needed with Custom Alerting

dannemca — Tue, 03 Sep 2024 06:20:04 GMT

Hi community, good day.

I have a specific need with a custom metric we are ingesting in our Dynatrace env.

To simplify, we are sending traffic data from 4 different hosts, where the host id () and app id are present as metric dimensions. By default, a single app traffic should happen at one host per time.

We now want to alert when an app is generating traffic in more than one host at the same time, for at least 10 min.

I am not finding a way to calculate this, either in DataExplorer (metric expression) nor using Davis Anomaly detection (DQL).

Let's see if anyone can help me to clear this out.

Example:

That's a normal behavior, we changed the traffic from one host to another. If the line (traffic) continues for both (or more) hosts at the same time, for 10 min, we have an issue.

Thanks.

Re: Help needed with Custom Alerting

GosiaMurawska — Tue, 17 Dec 2024 10:17:27 GMT

Hi @dannemca, did you solve this one? 🤔

Re: Help needed with Custom Alerting

Peter_Youssef — Wed, 18 Dec 2024 11:24:51 GMT

Hello @dannemca

As an applicable suggestion, will divide the solution into "2" tasks as follows:

Metric Expression in Data Explorer: Creating a custom metric expression that combines traffic data and groups by the app id and host id. Using time window function to check if the app is sending traffic to more than one host at the same time for at least 10 minutes.
Dynatrace Query Language: Defining a custom DQL query to detect anomalies in the metric where traffic occurs across multiple hosts.

Hoping it helps.

BR,

Peter

Re: Help needed with Custom Alerting

dannemca — Mon, 13 Jan 2025 20:19:00 GMT

Our team managed to "resolve" that creating one metric selector by app and by host grouping... it is not the ideal, since the number of metric selector depends on the number of apps (the more apps, the more metric selectors).
But at least, we could use the new Davis Anomaly Detector App to create the alerting rules with no limit issues (as we have in metric events settings).

Re: Help needed with Custom Alerting

Peter_Youssef — Mon, 13 Jan 2025 22:39:36 GMT

Thanks @dannemca for keeping us posted.