K8 Cronjob failed jobs should not trigger success alert after triggering failed job alert

[akashkatoch]

I have created custom log metric event which filter out failed cronjob and set this custom metric in anomalies detection metric event to monitor failed cronjobs . While setting an alert I have notice that dynatrace is triggering the alert when any job got failed and immediately triggers resolved alert  for the same failed jobs . Ideally Cronjob that got failed is already failed and there is no point for dynatrace to trigger resolved status . As since Dynatrace does not offer any default cronjob metric event but still they can add filter in metric event which restrict dynatrace to send resolved trigger for failed cronjob

[akashkatoch]

I saw you have edited the content , did someone acknowledge it @MaciejNeumann  ?

[travis_walker]

@wolfgang_beer Is this something you can advise on? 

[wolfgang_beer · Accepted solution]

Dear @akashkatoch ,

With log events you can trigger a problem that is kept open for a default period of 15min. In case the same log event appears again in this period of time, the problem is automatically refreshed and kept in active state.

You can add a custom event property with the value 'dt.event.timeout' = 60 to set the keepalive to 60 minutes to keep it active and open for a longer period of time.

In case you want the same problem to be open for days or weeks, there is no built-in mechanism in Dynatrace. This needs to be handled in an external ticketing system instead.

Best greetings,

Wolfgang

[akashkatoch]

I think this is an alternative solution to cronjob to keep the alert alive for defined period of time . I need sort of notification based offering , because seems like metric event doesn't suits well with cronjob alerting.