Percentiles in Anomaly Detection Alerts

Platform DQL
1

Hi everyone,

I’m trying to create an anomaly detector based on a DQL query using span duration.

fetch spans
//| filter transaction.is_root_span AND transaction.is_endpoint_request
| filter matchesValue(endpoint.name, “<REDACTED_ENDPOINT>”)
| makeTimeseries {
avg = avg(duration)
// p95 = percentile(duration, 95)
}, interval: 1m

The query works when using avg(duration), but if I try to use:

percentile(duration, 95)

Dynatrace reports that percentile aggregations are not supported for alerting/anomaly detection.

My goal is to create an alert based on the 95th percentile of response time rather than the average.

I know that a similar use case can be achieved today using the classic approach with Key Requests and response-time thresholds based on P95. However, my understanding is that the Classic UI and related classic configuration options will eventually be deprecated, so I’m looking for the recommended approach in the new platform.

What is the recommended way to implement P95-based alerting going forward?

  • Is there an alternative DQL function that is supported by anomaly detectors?
  • Should I first create a metric and then alert on that metric?
  • Is there a best-practice approach for alerting on the 95th percentile of span or endpoint response times in the new Dynatrace platform?

Any guidance, recommendations, or roadmap information would be greatly appreciated.

Thanks!

2

Hey arobayo,

I would first start by looking at a way to get the value as a metric as span queries in an anomaly detector can get expensive. For that, you’ve generally got 2 main options:

  1. You can enable performance metric capture for ALL endpoints in your environment using the following feature Enable endpoint metrics in Service Detection v1 — Dynatrace Docs. This means you no longer have to define key requests. I would recommend enabling it for a small portion of the environment first to get an understanding of how it works.
  2. If the above does not suit the environment, you can also set up an Openpipeline rule to do metric extraction. Based on the example query you’ve provided, you aren’t doing a complex calculation across multiple spans within 1 trace, so Openpipeline should be able to extract the metric you’re looking for.

Once you have the metric, I’d then rewrite the anomaly detection using the timeseries command.

3

Hi @arobayo17

percentile() isn’t supported in makeTimeseries on raw spans for anomaly detectors. But you don’t need raw spans for this. The built-in metric dt.service.request.response_time is stored as a histogram, so timeseries supports a proper percentile() on it, and that works in Anomaly Detection:

timeseries p95 = percentile(dt.service.request.response_time, 95),
by: { dt.entity.service, endpoint.name },
filter: dt.entity.service == “SERVICE-07F2B72F5C1AA481” and endpoint.name == “POST /get-quote”,
interval: 1m

image
image1511×555 65.8 KB

Also the metric is in microseconds (800 ms = 800000). Also, for classic services only key requests get their own endpoint.name; others appear as NON_KEY_REQUESTS, so run it without the filter first to check your endpoint shows up.

Thanks,

Sujit