cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

This product reached the end of support date on March 31, 2021.

Continuously generate alerts rather than one at start and one at end?

collin_lesko
Dynatrace Advisor
Dynatrace Advisor

Currently, at my location, we have had an alert fire during downtime (maintenance window). Therefore, as expected, we did not receive an e-mail.

However, after the downtime (say 7 a.m. is the end for example) the alert would still not fire even though it is still going on.

I understand that alerts fire on beginning as well as on end of incident, but would it not make sense that if an incident is still being triggered after a downtime window that I would get an e-mail about it?

Maybe I am missing a certain setting or something has slipped my mind, but is there any way to have the above happen?

Also, is there a feature that will continuously generate alert e-mails (lets say every 5 minutes) until the incident is confirmed?

Thanks,

Collin

7 REPLIES 7

richard_boyd22
Inactive

andreas_grabner
Dynatrace Guru
Dynatrace Guru

Hi Collin

As you correctly said - emails get sent when the Incident Starts. If that happens during a downtime then there is simply no email sent. Also not after the downtime is over as the Incident itself doesnt trigger any additional events while the Incident is still open.

Dynatrace in the moment only sends a single email for one incident. If you have an Incident such as "Slow Response Time" and the within an hour this incident is raised 10 times you can receive 10 emails unless you choose "Smart Alerting". Smart Alerting is an option in the Advanced Action Settings where dynatrace will only trigger an Action for the first incident of a Rule. Unless that Incident is not confirmed no other emails are sent. This is however not what you are looking for

We have a lot of users that use our integration with Service Now or PagerDuty for "smarter alerting". these are tools that are specialist in alerting/notifications and also have some advanced workflow features. So - you might want to look into this and integrate dynatrace with it

Andi

fyi - also have a look at the documentation: https://community.dynatrace.com/community/display/DOCDT62/Incidents%20and%20Alerting

Hi, my company is using Dynatrace Synthetic (AKA Gomez Synthetic, that was acquired by Dynatrace). I am trying to integrate this with PagerDuty and only documentation that I could found was related to Dynatrace (real/browser) monitoring that has the client installed in the server. Do you have an idea how I should approach to integrate between Dynatrace Sunthetic and Pagerduty? If you do, please advise. Thank you for your time in advance.

Hi. As this is the Dynatrace AppMon forum I suggest you repost this question on the synthetic forum. Here is the link Dynatrace Synthetic Open Q&A

Sounds good,

Thanks Andi

P.S. Is there any chance that a feature, as mentioned above, will exist in the client sometime? Possibly 6.3?

kasey_castanier
Contributor

In addition to ServiceNow and PagerDuty, it seems as though it is also possible to create a mechanism that confirms an Incident as soon as it triggers, as mentioned by @Graeme Williams in the following forum post:

https://answers.dynatrace.com/content/idea/134102/...

We were able to create and automatically execute a script with the Generic Execution Plugin that uses the REST interface to automatically grab a list of all "InProgress" or "Created" incidents under a certain Incident rule and confirm them, "restarting" the Incident alert timer. This script is run via the execution plugin in addition to sending an email. The problem that we have found, though, is that Dynatrace immediately generates another Incident if the issue is still occurring, immediately generating another email and triggering the execution plugin to run the Incident-confirming script again.

Our thought/hope was to get around this by setting up Incident suppression in the Incident rule. We thought that setting the suppression to 5-minutes would keep another Incident from being created for 5 minutes after our script confirms the first one. This does not appear to be the case though. Would anyone be able to help explain why this is? Is there anything we can do to stymie the creation of further Incidents for a defined period of time aside from putting some sort of sleep function in our script? Any suggestions would be much appreciated!

Many thanks!

- Kasey C.