topic Re: Custom alerts for log monitoring in Alerting

Custom alerts for log monitoring

jc__ — Fri, 25 Nov 2022 07:58:10 GMT

I would like to create a custom alert on log monitoring whereby an alert is raised when more than 1 log events occurred in a day. I tried to configure the custom event to raise an error when the metric threshold is above 1 in the given day, which is 1440 minutes. However, the maximum value of minutes period is 60. Can anyone advise me on this issue?

Thank you.

Re: Custom alerts for log monitoring

dannemca — Fri, 25 Nov 2022 13:19:01 GMT

Correct me if I am wrong, but if you get an alert in 1h, it means that you got an alert that day, and you will need to be notified about it.

The only problem with the 1h instead 24h is that you may end up receiving more than one alert per day, which may indicates that the system you are monitoring is not so healthy and the attention is required.

You can also work with metrics transformation, for example, limiting the data points to the last day with :timeshift(-1d) and then combine the data to a single point with :fold.

Example:

your.custom.metric.for.log:timeshift(-1d):fold

Try and let us know.

Re: Custom alerts for log monitoring

jc__ — Tue, 29 Nov 2022 08:54:09 GMT

Hi dannemca ,

Thank you for responding.

I might not have make myself clear in the question asked.

The requirement:

Raise an alert if the metric is above the static threshold of 1 in 2 one minute slot during a day (24 hrs)

The scenario:

Our client has a server that will restart once everyday, hence, one "initialized" keyword will be observed in the log. If there is more than 1 "initialized" keyword raised in a day, an alert should be raised so that our client can look into the issue.

May I know if there is any way to achieve the above requirement?

Thank you.

Re: Custom alerts for log monitoring

Michal_Gebacki — Thu, 22 Dec 2022 16:13:45 GMT

Hi @dannemca! Do you know if that alerting requirement that jc_ mentioned above is actually possible to achieve? Thank you for your help in advance!

Re: Custom alerts for log monitoring

Tom_Eaton — Fri, 31 Mar 2023 12:50:04 GMT

You could look into a possible similar solution as I have seen once before:

Create a log metric to find the log messages with 'initialized' - use matchesPhrase
Then create an SLO using a metric selector with the defualt(0,always) for your log metric - this will give every 1 minute sample a 0 default value when data is missing.

I have seen this used for a similar scenario for a batch job with a log being written once a day, looking for a day when it was not written - so someone used a SLO to look back -25 hours

This might be a possible avenue to explore.

Re: Custom alerts for log monitoring

Tom_Eaton — Fri, 31 Mar 2023 12:52:06 GMT

@dannemca - the timeshift only moves the time, it is used as a timeframe window. So -1d will not give the last 24 hours, but just make the current date and time, 1 day before.. I.E 31st March becomes 30th March