topic Re: K8 Cronjob failed jobs should not trigger success alert after triggering failed job alert in Alerting

K8 Cronjob failed jobs should not trigger success alert after triggering failed job alert

akashkatoch — Thu, 08 Feb 2024 07:50:03 GMT

I have created custom log metric event which filter out failed cronjob and set this custom metric in anomalies detection metric event to monitor failed cronjobs . While setting an alert I have notice that dynatrace is triggering the alert when any job got failed and immediately triggers resolved alert for the same failed jobs . Ideally Cronjob that got failed is already failed and there is no point for dynatrace to trigger resolved status . As since Dynatrace does not offer any default cronjob metric event but still they can add filter in metric event which restrict dynatrace to send resolved trigger for failed cronjob

Re: K8 Cronjob failed jobs should not trigger success alert after triggering failed job alert

akashkatoch — Fri, 09 Feb 2024 14:20:02 GMT

I saw you have edited the content , did someone acknowledge it @MaciejNeumann ?

Re: K8 Cronjob failed jobs should not trigger success alert after triggering failed job alert

travis_walker — Thu, 27 Jun 2024 18:31:46 GMT

@wolfgang_beer Is this something you can advise on?

Re: K8 Cronjob failed jobs should not trigger success alert after triggering failed job alert

wolfgang_beer — Fri, 28 Jun 2024 05:47:45 GMT

Dear @akashkatoch ,

With log events you can trigger a problem that is kept open for a default period of 15min. In case the same log event appears again in this period of time, the problem is automatically refreshed and kept in active state.

You can add a custom event property with the value 'dt.event.timeout' = 60 to set the keepalive to 60 minutes to keep it active and open for a longer period of time.

In case you want the same problem to be open for days or weeks, there is no built-in mechanism in Dynatrace. This needs to be handled in an external ticketing system instead.

Best greetings,

Wolfgang

Re: K8 Cronjob failed jobs should not trigger success alert after triggering failed job alert

akashkatoch — Tue, 13 Aug 2024 11:36:15 GMT

I think this is an alternative solution to cronjob to keep the alert alive for defined period of time . I need sort of notification based offering , because seems like metric event doesn't suits well with cronjob alerting.