Delay in email notifications

[tushar_dhule]

HI,

We observed 15 min delay in problem email notification. But the teams expect the alert notification at real time. Kindly let me know the reason behind this delay and if there is any workaround.

[ChadTurner]

@ted d. Can you confirm that the alerting profile is set to alert immediately of the issue? and not "X" amount of mins. after? Also the system might have a 1-5 min delay of the email arriving in the users inbox. To combat this, we also incorporate the Dynatrace UFO and Mobile APP Alert.

[AntonioSousa]

Chad,

Do you see differences between emails and the app? They fall in my case almost exactly immediately.

[ChadTurner]

we found the App to be 1 second behind the UI raising a problem card. same with the UFO - for a managed stand point with 3 clustered nodes

[AntonioSousa]

Thought it was a lot more. In my case I have not analyzed it with that precision 😉

[ChadTurner]

Do you guys use the ufo as well?

[AntonioSousa]

No, not yet...

[ChadTurner]

I would recommend it

[tushar_dhule]

How UFO will help in this case

[ChadTurner]

The ufo will visually alert you to an issue in your infrastructure via the color codes. We have 5 deployed, and people love it.

[DanielS · Accepted solution]

Hi @ted d. in first place is important to understand that Dynatrace is not 'just another silly alerting tool'. When an event took place the IA or Davies is doing a huge correlation between all your components. And alerts you when all the analysis took place, this is not in realtime but of course it can not take 5' as you described.

In my experience with several clients this could be due to an incorrect sizing of the underlying hardware where Dynatrace runs. (Of course this applies to Dynatrace Managed only) Please check the requirements at https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-managed/installation/dynatr...

Dynatrace Managed operates on all common file systems. The recommendation is that you select fast local storage appropriate for database workloads. High latency remote volumes like NFS or CIFS aren't recommended. While NFS file systems are sufficient for backup purposes, it's not recommended for primary storage.

If SaaS is your case, is important to check the bandwith between ActiveGate and SaaS is not suffering from starvation due to 'all the other traffic'. Depending of your firewall, you can prioritize this, also check hardware requirements for your ActiveGate, please check it at https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-activegate/installation/act...

At last when we take care of this, the delays you described were kept to a maximum of 2' for complex cases on environments > 500 Hosts Units and near real time in smaller installations.

[ThaplooT]

Hi Daniel

we saw a delay of 56 min in receiving the email the problem started at 13:01 and the customer got an email notification at 13:57 can that be network issue or any thing else?

Thanks

Taneshaa

[AntonioSousa]

@ThaplooT,

Please check the email headers to see where it might have got delayed. Also check it's not a frequent issue, because in these cases, the alert might be delayed.

[Yosi_Neuman]

Hi @ThaplooT 

In addition to @AntonioSousa reply you can check dsfm:server.notifications.problem_notifications & dsfm:server.notifications.problem_notifications_duration metrics which will give you the cluster perspective on when cluster fire the notification with status and  timings 

HTH

Yos 

[AntonioSousa]

Even if configured to alert immediately, it takes some time:

I confirmed at Perform that Dynatrace is trying to bring these times down. I was explained something I don't recall exactly 😞 but I remember has to do with not bringing a lot of false positives... I also need faster alerting...

[AntonioSousa]

Just to add that the person who I talked to said that they were looking into this carefully...