cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Delay in email notifications

tushar_dhule
Participant

HI,

We observed 15 min delay in problem email notification. But the teams expect the alert notification at real time. Kindly let me know the reason behind this delay and if there is any workaround.

15 REPLIES 15

ChadTurner
DynaMight Legend
DynaMight Legend

@ted d. Can you confirm that the alerting profile is set to alert immediately of the issue? and not "X" amount of mins. after? Also the system might have a 1-5 min delay of the email arriving in the users inbox. To combat this, we also incorporate the Dynatrace UFO and Mobile APP Alert.

-Chad

Chad,

Do you see differences between emails and the app? They fall in my case almost exactly immediately.

Antonio Sousa

we found the App to be 1 second behind the UI raising a problem card. same with the UFO - for a managed stand point with 3 clustered nodes

-Chad

Thought it was a lot more. In my case I have not analyzed it with that precision 😉

Antonio Sousa

Do you guys use the ufo as well?

-Chad

No, not yet...

Antonio Sousa

I would recommend it

-Chad

How UFO will help in this case

The ufo will visually alert you to an issue in your infrastructure via the color codes. We have 5 deployed, and people love it.

-Chad

DanielS
DynaMight Leader
DynaMight Leader

Hi @ted d. in first place is important to understand that Dynatrace is not 'just another silly alerting tool'. When an event took place the IA or Davies is doing a huge correlation between all your components. And alerts you when all the analysis took place, this is not in realtime but of course it can not take 5' as you described.

In my experience with several clients this could be due to an incorrect sizing of the underlying hardware where Dynatrace runs. (Of course this applies to Dynatrace Managed only) Please check the requirements at https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-managed/installation/dynatr...

Dynatrace Managed operates on all common file systems. The recommendation is that you select fast local storage appropriate for database workloads. High latency remote volumes like NFS or CIFS aren't recommended. While NFS file systems are sufficient for backup purposes, it's not recommended for primary storage.

If SaaS is your case, is important to check the bandwith between ActiveGate and SaaS is not suffering from starvation due to 'all the other traffic'. Depending of your firewall, you can prioritize this, also check hardware requirements for your ActiveGate, please check it at https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-activegate/installation/act...

At last when we take care of this, the delays you described were kept to a maximum of 2' for complex cases on environments > 500 Hosts Units and near real time in smaller installations.

The true delight is in the finding out rather than in the knowing.

ThaplooT
Dynatrace Advocate
Dynatrace Advocate

Hi Daniel

we saw a delay of 56 min in receiving the email the problem started at 13:01 and the customer got an email notification at 13:57 can that be network issue or any thing else?

 

Thanks

Taneshaa

@ThaplooT,

Please check the email headers to see where it might have got delayed. Also check it's not a frequent issue, because in these cases, the alert might be delayed.

Antonio Sousa

Hi @ThaplooT 

In addition to @AntonioSousa reply you can check dsfm:server.notifications.problem_notifications & dsfm:server.notifications.problem_notifications_duration metrics which will give you the cluster perspective on when cluster fire the notification with status and  timings 

HTH

Yos 

dynatrace certificated professional - dynatrace master partner - Matrix Soft Ware Division - Israel

AntonioSousa
DynaMight Guru
DynaMight Guru

Even if configured to alert immediately, it takes some time:

I confirmed at Perform that Dynatrace is trying to bring these times down. I was explained something I don't recall exactly 😞 but I remember has to do with not bringing a lot of false positives... I also need faster alerting...

Antonio Sousa

Just to add that the person who I talked to said that they were looking into this carefully...

Antonio Sousa

Featured Posts