@ted d. Can you confirm that the alerting profile is set to alert immediately of the issue? and not "X" amount of mins. after? Also the system might have a 1-5 min delay of the email arriving in the users inbox. To combat this, we also incorporate the Dynatrace UFO and Mobile APP Alert.
Hi @ted d. in first place is important to understand that Dynatrace is not 'just another silly alerting tool'. When an event took place the IA or Davies is doing a huge correlation between all your components. And alerts you when all the analysis took place, this is not in realtime but of course it can not take 5' as you described.
In my experience with several clients this could be due to an incorrect sizing of the underlying hardware where Dynatrace runs. (Of course this applies to Dynatrace Managed only) Please check the requirements at https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-managed/installation/dynatr...
Dynatrace Managed operates on all common file systems. The recommendation is that you select fast local storage appropriate for database workloads. High latency remote volumes like NFS or CIFS aren't recommended. While NFS file systems are sufficient for backup purposes, it's not recommended for primary storage.
If SaaS is your case, is important to check the bandwith between ActiveGate and SaaS is not suffering from starvation due to 'all the other traffic'. Depending of your firewall, you can prioritize this, also check hardware requirements for your ActiveGate, please check it at https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-activegate/installation/act...
At last when we take care of this, the delays you described were kept to a maximum of 2' for complex cases on environments > 500 Hosts Units and near real time in smaller installations.
In addition to @AntonioSousa reply you can check dsfm:server.notifications.problem_notifications & dsfm:server.notifications.problem_notifications_duration metrics which will give you the cluster perspective on when cluster fire the notification with status and timings
Even if configured to alert immediately, it takes some time:
I confirmed at Perform that Dynatrace is trying to bring these times down. I was explained something I don't recall exactly 😞 but I remember has to do with not bringing a lot of false positives... I also need faster alerting...