Why my alerting profile is not on the problem? Possible reasons

[a_barbera]

Getting started

⚠️   If you haven't created an alerting profile yet, please read the steps provided in the documentation. Link here.

In some situations, we expect an alerting profile to detect a problem, but it doesn't. In this troubleshooting post, we're going to review the top main causes. 

Alerting profile created or enabled when the problem is already open

If the Alerting profile was created or enabled when the problem was already open, you aren't going to see the desired Alerting profile on that problem. 

We will have to consider two things:

1. When was the problem opened on the problems page? We will have to review the time the problem was reported on the problems page.
2. When was the Alerting profile created or enabled? We can check that using Revision History on the settings of the Alerting profile. In order to review this, you'll need to follow the next step:
   1. First, we will need to go to our alerting profile.
   2. In the configuration of our alerting profile, we need to select the button located in the top right corner of the Alerting profile configuration, as you can see in the following image:

3. Once we click, we can see Revision History. In there, we have an overview with different information regarding the changes that have been applied:

At the top left corner, the ID of the alerting profile is shown. Then, we will see a new row every time we make a modification regarding our profile. In each of the changes, we can see when it was made, what type of change it is, and what the original and new values of the recent changes were. If we switch to advanced mode, we can compare everything we had before with the new values in a detailed way. 

The impacted entity didn't have the management zone defined in the Alerting profile when the problem occurred

Suppose we have an Entity "A" with management zone "X":

Sometimes, when a problem is created, the impacted entity doesn't have the X but we expect to see it on A. 

Usually, we see that A doesn't have X due to a problem related to the management zone rule. We recommend checking the following in management zone settings:

1. Check recent revision history changes ( steps explained above ) to see if any user has modified any condition/rule/filter.
2. Check if the rule applies to the entity where the problem is occurring.
   • In order we just have to have two options:
     • Use the button Preview for Conditions. This option will check if there is any entity that applies for that particular condition.
     • Use the button Display entities. This option will check if there is any entity for all the conditions that we have created.
3. If you're using Monitored entity, check the conditions you have added to the rule. For example:
   • If you're applying the rule to "Host tag" with the name "production" and in the condition, we can see you're applying the rule to tags with the name "production. " As you can see, there is a blank in the end, so this condition will never be applied. We can see this happening many times with host names, tags, process group names, etc. 

That is why it is important to check step number two, as it is crucial to verify if the rule applies to any entity we have in our environment.

Event filters

If you're using event filters, you need to verify how they are configured. Here you can see some typical problems we see with event filters:

1. It is not necessary to use predefined filters for predefined events. If you select custom filters, you can also filter predefined events. For example, you can use a custom filter where you select a title you would like to filter for any of the predefined events.

     2.  For metric events, it would be necessary to use the custom filter. Below, you can see how it looks like the custom filter works. You can use, for example, a title filter, but there are other options like description or property filters.

What if my problem is not solved after checking all previous steps?

Once you have checked all the steps that are shown in the article but the issue is still not solved, please proceed to open a support ticket with the following information:

1. Provide a link to the problem that you have questions about.
2. Link to the alerting profile.
3. Short description of what is not working and what you're expecting. 

[noel_david]

Good article. 👍

[Tristan]

Thanks for sharing!

[CMembrive]

Thank you! Everything is very clear

[kalle_lahtinen]

Regarding "Custom filters don't match any predefined events.", something to note that's maybe not obvious to everyone (at least wasn't for me):

If you have a profile e.g. with "Error after x mins" and "Custom after x mins", and then you add a filter for "Custom: Title does not contain <stuff>", then it will work for your Error events and any Custom metric events which do not have <stuff> in the title.

However, if you define "Custom: Title contains <stuff>", you have just now filtered out all your predefined events from the alerting profile scope. Because none of the predefined events will have that custom title, of course.

This behavior is not really logical to me, because I'd expect the predefined filters to only apply to predefined, and the same for custom. But this means that if you have any "include" definitions for metric events, you need to create 2 separate alerting profiles: one for predefined, and one for custom.

[a_barbera]

Hey @kalle_lahtinen , thanks for your pointing this out. 

We have reviewed your comment internally, and now it is possible to filter predefined events using custom filters. I have modified the article so is clear also for everyone. Appreciate your collaboration in this!