topic Business impact analysis is incorrectly evaluating impacted users in Alerting

Business impact analysis is incorrectly evaluating impacted users

kalle_lahtinen — Fri, 14 May 2021 12:28:22 GMT

Hi,

I ran into a case where the "impacted users" section in the problem summary was listing apps and services that weren't actually affected in any way. To avoid showing customer data here, I'll just describe this in a simplified way:

1. Application BananaApp uses service BananaSvc which then calls external service ExtSvc with request GetBananaInfo.

2. Service OrangeSvc calls external service ExtSvc with request GetOrangeInfo. OrangeSvc has no web UI usage and thus no RUM / "Application" data.

In my case, ExtSvc requests for "GetOrangeInfo" were failing, which generated a new problem. During this time, the failure rate for GetBananaInfo was a stable 0 %.

In the problem analysis, Dynatrace said that 100 users from BananaApp were impacted by this. In fact, zero users were, because the issue was limited to OrangeSvc.

So Davis deduced that because ExtSvc was having issues, and BananaApp generally uses that service, BananaApp is impacted. If Davis was instead correlating the data on a request level, it would've seen that zero GetOrangeInfo requests are coming from BananaApp, and thus it's not impacted.

I'm not sure whether this should be regarded as a bugfix or an RFE? And also I'm not certain what is the role of this external service here, i.e. would the same thing happen if it was a local service monitored by OneAgent? This ExtSvc was originally under "Requests to public networks", which I then isolated via "Monitor as separate service".

Re: Business impact analysis is incorrectly evaluating impacted users

Anonymous — Tue, 08 Dec 2020 12:00:18 GMT

When you open the user sessions from the affected users and you check the session service flow from the app POV, how does it show?

Re: Business impact analysis is incorrectly evaluating impacted users

kalle_lahtinen — Tue, 08 Dec 2020 12:11:02 GMT

Not sure how to exactly view the service flow from a single user session, at least in the same sense you can view it from "Transactions and services" point of view as a diagram ("View service flow" button).

But looking at the individual Load actions from impacted user sessions, I can see that this external service was indeed requested (by drilling down from Load Action to PurePath view). But the requests were successful; like I said the requests affecting these apps were not the same ones that were failing at the time. Davis just lumped all of this together because of the common nominator i.e. the external service.

Re: Business impact analysis is incorrectly evaluating impacted users

Anonymous — Tue, 08 Dec 2020 12:30:06 GMT

I see.

The external service shows up like "ExtSvc", is possible to see the GetOrangeInfo and GetBananaInfo from DT as individual requests from the root ExtSvc? or is all the same requests inside ExtSvc and when checking the pure path you see the difference?

Might be needed

A) install OA to add that context and allow the causation engine to correctly see the difference or B) open up ExtSvc in individual services then so it treats each one as an ExtSvc (Via API Service Detection for external services)

Re: Business impact analysis is incorrectly evaluating impacted users

kalle_lahtinen — Tue, 08 Dec 2020 15:45:19 GMT

Hi,

Thanks for the link! I was actually about to do that anyway, not just due to this issue. So you actually took me right to the config I was looking for 🙂

Regarding your first question, yes those requests are separately visible under ExtSvc, not just under the PurePaths. So e.g. by activating the failure rate tab, I can see right away which requests are failing and which are not. That's why I was surprised that the business impact wasn't correctly detected by Davis, since it should be possible to do based on the data at hand. But again, I'm not certain which part of the misconception is due to this being an external service.

There will never a possibility to install OneAgents on the remote hosts, so they will remain as external services. As for what's the impact of separating ExtSvc by subdomain, I guess I'll see in time if it helps, during a next similar problem event.

For now I'm not sure whether this question is answered 🙂 Maybe I'll keep it open for a while yet.

Re: Business impact analysis is incorrectly evaluating impacted users

kalle_lahtinen — Tue, 08 Dec 2020 19:18:27 GMT

Just to reply to myself: after splitting the service based on the subdomain, of course I won't see this exact issue anymore, because now the application usage won't be linked anymore to this shared "ExtSvc" I referred to earlier. To use my earlier example names, now "GetBananaInfo" and "GetOrangeInfo" are under separate services and thus won't be mixed up anymore in the root cause analysis.

I'm still not sure whether Davis evaluates things on a request level or only the service level when it comes to analyzing the business impact. But I'll nonetheless mark this as answered, because my problem basically got solved.

Re: Business impact analysis is incorrectly evaluating impacted users

Enrico_F — Wed, 09 Jun 2021 08:36:39 GMT

I'm investigating a very similar case in our environment where any anomalies detected on the built-in service "Requests to unmonitored hosts" seem to cause spurious problem associations and I wondered what you meant by "split the service based on the subdomain"? Did you create key requests for the detected requests to BananaSvc and OrangeSvc on your "ExtSvc" service that you separated from "Requests to public networks" earlier? Or did you create custom services on the PG's that provide the BananaSvc and OrangeSvc representing the client-side code of the requests to GetBananaInfo/GetOrangeInfo? Or something else entirely?

Your feedback would be highly appreciated.

Re: Business impact analysis is incorrectly evaluating impacted users

kalle_lahtinen — Wed, 15 Jan 2025 12:07:18 GMT

Hi Enrico,

I've originally missed your question, so my response is too late probably to be of any help... But I'll reply anyway, in case it's useful for someone searching on this topic. Back then, I created service detection rules for external web requests (or web services, can't remember anymore). PGs weren't involved, because those were external services with no OneAgent and thus no processes visible.

Overall I find the Business impact analysis still quite unreliable. I just now bumped into a case where it's reporting the impact in a similar way. Database calls coming from 2 services were failing, which is clearly visible in the backtrace: the scope is limited to only those 2. But the Business impact analysis reported an uninvolved service just because it has a high request rate. If I filter the database calls for requests which originated from that service, zero of them have failed. So the impact analysis should not state that it's impacted. Anyway, for me the solution is simply to not read the contents of the Business impact analysis box 🙂