topic Davis AI - Key Request vs Service baseline in Alerting

Davis AI - Key Request vs Service baseline

alexandre_marl2 — Mon, 31 Aug 2020 17:06:40 GMT

Hello,

We have one key request on a service which is slow by design (it requests a lot of external provider and the response time is between 5s and 10s) as you can see here :

Sometimes during the day we have more traffic for 10/15 minutes and it is normal.

But the services detect an increase of response time and report an incident (both nginx and phpfpm).

But this is normal as it is only one of the endpoint (and one of the key request where the threadhold as already been adjusted).

It is very annoying and i didn't find any way to tell to dynatrace. It is normal as it affect only one request which has a specific configuration.

Any idea ? I am running out of idea. I think i am not the only one to get that kind of issue (one endpoint for one specific services slower than the others one).

I could configure the service to allow increase of response time by +5000% but for me it is not the best way to handle that.

Bigs thanks in advance,

Alexandre

Re: Davis AI - Key Request vs Service baseline

ChadTurner — Mon, 31 Aug 2020 18:23:22 GMT

Davis will take this and add it into the base line and if this problem keeps coming up, it will detect it as a frequent issue.

To check this setting go to Settings>Anomaly Detection>Frequent Issue Detection.

For custom alerts you can also look at the new adaptive control features:

https://www.dynatrace.com/support/help/how-to-use-dynatrace/problem-detection-and-analysis/problem-detection/metric-events-for-alerting/auto-adaptive-baseline/?_ga=2.244787461.1428338629.1598875712-641312477.1598875712

Re: Davis AI - Key Request vs Service baseline

JamesKitson — Mon, 31 Aug 2020 19:31:37 GMT

For each service there is the overall baseline that is learned in addition to the top 100 requests also get a baseline. Marking a request as key lets you ensure there is tracking on that request regardless of its frequency and you can tune the anomaly detection behavior (i.e. thresholds) but it still will have an impact on the overall service baseline (they're note exclusive).

If you don't want to have that one request impacting the overall service baseline you could look at defining a service detection rule (through the API) that will break that request into a separate service then you can manage them separately.

https://www.dynatrace.com/news/blog/new-dynatrace-api-enhances-automatic-service-detection/

https://www.dynatrace.com/support/help/dynatrace-api/configuration-api/service-api/detection-rules/full-web-request/post-a-rule/

This is the only solution that comes to mind that would let you see and monitor the request without impacting the rest of the requests in what is currently one service.

Re: Davis AI - Key Request vs Service baseline

wolfgang_beer — Tue, 01 Sep 2020 06:27:08 GMT

The only way of solving this is to split the service and to put the slow request in a separate service. In that case the newly created services gets its own baseline and the calls are removed from the original service.

Or, as it was already suggested below, if you don't want to split the service, just disable default baseline alerting on the service and set up a custom event for individual key requests.

Re: Davis AI - Key Request vs Service baseline

alexandre_marl2 — Wed, 02 Sep 2020 15:07:08 GMT

Hello,

Thanks for all your interesting answer. I was not aware we could split a specific request outside a service as i didn't see it in the UI.

I am surprised it is only available in the API but not in the UI as it would be usefull.

As my webapp is nginx/phpfpm and one other component which is slow. I will have split this specific key request for those 3 services to be able to avoid my alert.

Not so easy 😞

Alexandre

Re: Davis AI - Key Request vs Service baseline

larry_roberts — Tue, 08 Sep 2020 23:30:12 GMT

We ran into the same thing and had to do the split via the API with a rule. No issues with it, but it will be great once this is added to the actual UI rather than having to go through the API.

Re: Davis AI - Key Request vs Service baseline

alexandre_marl2 — Wed, 09 Sep 2020 10:20:19 GMT

Confirmed. For a small ops team it is easier to do it directly in the UI and I think a lot of client need to do that.