Create Alerts for Multiple Location Failures Using DQL

[sivart_89]

I'm trying to essentially duplicate this type of alerting but with a custom alert in the Davis anomaly detection app. What I'm struggling most with is the '3 times consecutively' part. The synthetic I am working with runs every 5 minutes, any ideas of a DQL that would work for this? I have some simple code below, it just needs to be improved upon.

timeseries synthetics_v = avg(dt.synthetic.browser.availability), by: {dt.entity.synthetic_test, dt.entity.synthetic_location}, interval: 5m
| fieldsAdd syntheticName = entityName(dt.entity.synthetic_test)
| fieldsAdd locationName = entityName(dt.entity.synthetic_location)
| filter contains(syntheticName, "<synthetic name>")

[p_devulapalli]

@sivart_89 - I think you can achieve it by using violating samples and sliding window in advanced properties of anomaly detector app

https://docs.dynatrace.com/docs/shortlink/anomaly-detection-configuration#sliding-window

[sivart_89]

If I set it up this way then an alert will get created when a single location fails 3 times within a 5 minute window. There's 2 things I see wrong with this. First, I need it to alert when there are 3 consecutive failures and second, I only need an alert if 2 locations fail 3 times consecutive, I don't want an alert if just 1 single location is failing.

[IzabelaRokita]

Hey @sivart_89 ,
I just wanted to check in and see if you still need help with this. If so, I’d be happy to look into it for you! 😊
Please let me know what works best for you.

[t_pawlak]

Hi,
What you’re missing is that “3 consecutive failures” can’t be reliably expressed with violating samples or sliding windows alone, especially when you also need multi-location logic.

So, I think i find other solution, something like workaround. A practical way to model consecutive failures in DQL is to treat them as continuous failure over the evaluation window.
For a browser synthetic running every 5 minutes,
3 consecutive runs = 15 minutes of continuous failure.

You can express this in DQL by checking that all samples in the window failed, and then counting how many locations meet that condition.

Example DQL:

timeseries avail = avg(dt.synthetic.browser.availability),
by:{dt.entity.synthetic_test, dt.entity.synthetic_location},
interval:5m
| fieldsAdd syntheticName = entityName(dt.entity.synthetic_test)
| fieldsAdd locationName  = entityName(dt.entity.synthetic_location)
| filter syntheticName == "<synthetic name>"
| fieldsAdd
    samples  = arraySize(avail),
    maxAvail = arrayMax(avail)
| filter samples >= 3
| filter maxAvail < 1
| summarize
    failing_locations = count(),
    by:{dt.entity.synthetic_test}
| filter failing_locations >= 2

arrayMax(avail) < 1 guarantees there was no successful execution in the window → truly consecutive failures

samples >= 3 ensures at least 3 runs were evaluated

failing_locations >= 2 enforces the multi-location condition

In Davis Anomaly Detection, configure:

Evaluation window: 15 minutes

Evaluation frequency: 5 minutes

Trigger condition: Result exists

No sliding window / violating samples needed

Try this and give me feedback. I test it, but I don't have test with 3 failures.