cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

PRO TIP - Excluding Monitoring Construct for Alerts

ChadTurner
DynaMight Legend
DynaMight Legend

Background: As we move closer and closer to the Holiday season a common issue is suppressing certain alerts that don't reflect a problem. For example, Low User Traffic on a holiday. There have been multiple RFEs to address these type of problems. While we wait for a direct solution from Dynatrace I'd like to offer my Pro Tip on how to handle these unwanted alerts on certain days etc... 

First step: Define what it is that you want to stop alerts on, it could be one thing, or a collection of things. But we need to know what you want to exclude. Lets use this Scenario - Your organization decided to leverage a federal holiday to roll out a bunch of patches. As a result, we fully expect 2 problems. First being 'Unexpected Low user traffic' and Second being 'High CPU Usage'.  

Second Step: Understand how your organization alerts. Some organizations use privatized alerting, meaning that there is an alert profile and an alert integration that shoots out an email to the defined recipients. Other organizations use an event handler, such as Moogsoft, Servicenow etc.. Never the less, you'll need to understand where your alerts go and how they alert to your customers. For this case we will use an event handler as the method can be used for privatized alerting as well. 

Third Step: Now that you know what you want to suppress, and you know what type of notification delivery method your organization uses, we need to validate what alert profiles these 'alerts' you want to suppress apply to. This is easier when you have an event handler because the average organization sends everything to it. So lets go into Dynatrace and look for the low user traffic: 

ChadTurner_0-1694627250966.png

Now we know what alert profiles qualify for this event. And since we are using an event handler in this scenario, I know that the alert profile starting with "N" goes to our event handler. That alert profile is now my target. 

Fourth Step: Lets go into that Alert Profile so we can make the changes needed to suppress the qualification of the alerts. Looking at the Alert Profile we can see al the rules are set to immediately alert, but to remove an alert aspect we want to go into "Add Event Filter"

ChadTurner_1-1694627558878.png

Fifth Step: The key to this is the "Negate" function. We can select a predefined event, basically anything out of the box with Dynatrace. Then we select that out of the box event we want to target. And the final aspect is to set the 'Negate' to ignore issues that fall under Low Traffic. Now our scenario stated that their is going to be high CPU as well, So ill just make two negate rules: 

ChadTurner_2-1694627768932.png

ChadTurner_4-1694628040982.png

Now the system will qualify alerts that pertain to the defined Management Zone, AND any defined severity rule, AND validate that the alert does not contain Unexpected Low Traffic. We can add in another rule to negate the CPU as well.

 

Granted yes this is a manual effort, but if you know the dates and the scopes, you can automate it via the API as well.  

 

I hope this helps everyone this holiday season 🙂 

 

 

 

-Chad
5 REPLIES 5

natanael_mendes
Champion

Thanks for the tip Chad, appreciate 

Dynatrace Professional Certified

danielD
Dynatrace Helper
Dynatrace Helper

This is great information Chad, thank you!

Kenny_Gillette
DynaMight Leader
DynaMight Leader

The Pro!

Dynatrace Certified Professional

tijust1
Helper

@ChadTurner This is really great Tip.....

Dynatrace Professional Certified

Viachaslau
Advisor

Good idea. Need to automate it.
Basically we will need 2 version of settings and script to push different one into dynatrace based on calendar dates/week days

Featured Posts