Log entry for open and close problem

[tarjei]

Hi!

We have a case where a program logs to a file when it is having issues. It also tries to self remediate which it also enters into the same log.

Hence what I would like to happen is the following:

1. Detecting "Issue starting" log entry, Open Problem

2. Detecting "Issue ending" log entry, Close Problem.

Is this possible in any way?

Possibly even with Workflows?

[AntonPineiro]

Hi,

Dynatrace does not work in that way about closing log problems. Problems are opened base on log match and they are closed base on delay. You can play with delay, it means:

• Log pattern found => Problem is created.
• Log pattern has not been after X minutes (you can configure delay, 15 minutes, 4, hours...) => Problem closed.

It means, problems are not closed when another log pattern is found. They are closed when before log pattern IS NOT found again.

Maybe you can explore Anomaly detection or workflow. You have here a backup example . But keed in mind reading logs consume license.

And of course, you can create your custom script that read log lines and ingest data in Dynatrace.

Best regards

[tarjei]

That is my belief as well. 

A bit sad since we are migrating from nimsoft which is an old tool and it has this functionality. 

[andreas_grabner]

What about using the Close Problems API? https://docs.dynatrace.com/docs/dynatrace-api/environment-api/problems-v2/problems/post-close 

[tarjei]

Hi Andy! 

Yes, that is the solution I am going for. Workflow with this API. 

Would be nice if there was some other more native way, but not sure how feasible it is?

[dylan_taelemans]

Could you provide an example of both an open and a close log?

That way I can check whether it's possible to build a workflow that fits your use case.

I believe this should be possible to implement entirely within a Dynatrace workflow.
In the workflow we could call Dynatrace api to close problems. 
The main challenge would be identifying a unique field that we can add to the problem when it's created and then use later to locate the same problem when it needs to be closed.

[Julius_Loman]

@tarjei I'd not recommend using using the problems v2 api @andreas_grabner suggested - first of all, it's classic api and likely will be deprecated sooner or later. There is a simple solution:
For log events which should open a problem, create a davis event with dt.davis.timeout set to maximum value (360 minutes). For log events which close the problem, create the same davis event event with dt.davis.timeout set to 1 (1 minute), so problem will be closed after one minute.

The challenge is to keep the problem open if the timeout is exceeded - which can be accomplished by a workflow (I only did this on Managed where you need to do it externally) - query such events (preferably add some event property) and just "resend" them.

Hope this helps.

[tarjei]

This sounds like a great suggestion with less required maintenance. Do you know how I ensure that the closing event and the opening event "hit" are considered the same one?

[AntonPineiro]

Hi,

Is it going to consume license? It means, for scanning log patterns which close the problem.

Best regards

[tarjei · Accepted solution]

I have created an RFE here: RFE: Create the possibility for a log event / pattern for OPEN and a different one for CLOSE - Commu...

@andreas_grabner any thoughts on the difficulty of implementing such functionality?

[andreas_grabner]

I am not in Product Engineering - so - I cant comment on how long this would take. Lets see what the reaction is on the RFE Ticket. In the meantime I hope you got some hints here in the thread on how to move forward