Automations
All questions related to Workflow Automation, AutomationEngine, and EdgeConnect, as well as integrations with various tools.
cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Site Reliability Guardian Validation Failed

Ganeshmete77
Participant

Hi Team,
I need help to understand how Dynatrace Site Reliability Guardian works, because i have created some guardian in Dynatrace but they are failing when there is slight change in metric values.
And the threshold definition is Auto-adaptive Threshold
first five run's results are [711.73ms, 711.73ms, 1321.07ms, 1321.07ms, 1370.06ms, 1370.06ms] and it's failed for 1391.21ms
SRE Guardian.png
below is the query i am using to get the Latency
timeseries p95 = percentile(dt.service.request.response_time, 95), by:{dt.entity.service}
| fieldsAdd entityName(dt.entity.service)
| filter startsWith(dt.entity.service.name, "prod-***-*-sessionutil")
| filterOut contains(dt.entity.service.name, "bff-live")
| filterOut contains(dt.entity.service.name, "sidecar")
| summarize response_time_95 = avg(arrayAvg(p95))

Also what best practices we can follow here.

2 REPLIES 2

You can view the thresholds by scrolling down on the analysis page.dylan_taelemans_0-1783418052931.png
It will use the values of last 5 runs to create those auto-adaptive thresholds.

Personally, I recently started migrating (still in progress) all auto-adaptive thresholds to static thresholds in my SRG. I’m not a big fan of relying solely on auto-adaptive thresholds, as they often don’t align with what I actually want and don't allow any specific configuration or fine-tuning. (For example, as shown in the screenshot, the warning and error thresholds are either identical or have such a small difference that it doesn’t make a meaningful impact.)
)

Hi,
I agree with this. It looks like in this case Auto-adaptive Threshold isn't the best choice.

Based on the results shown:

711 ms, 711 ms, 1321 ms, 1321 ms, 1370 ms, 1370 ms

Guardian built its baseline from a very small and rather unstable sample. As a result, 1391 ms (only about 20 ms higher than the previous value) can already be considered a threshold violation, even though it doesn't look like a meaningful performance degradation.

For latency checks, I would probably prefer static thresholds based on the application's SLO/SLA rather than relying solely on auto-adaptive thresholds.

Featured Posts