topic Dynatrace logs on problem notifications in Log Analytics

Dynatrace logs on problem notifications

clc — Tue, 25 Apr 2023 06:36:32 GMT

We had a issue with some problem notification so I tried to find any related log entries for this to see if they where actually made, but was unable to find any.

We are running on Saas and sounds like these entries could be found on a managed solution in the server.log, if possible where are information regarding problem notification logged?

Can verify a event is triggered, problem made, alerting profile is notet in problem, alerting profile is linked correctly to problem notification - but anyway to verify that in this case webhook to 3rd party is made?

-Claus

Re: Dynatrace logs on problem notifications

AntonPineiro — Mon, 24 Apr 2023 10:11:19 GMT

Hi,

Maybe you can create another problem integration to send an email to you, using same alerting profile and payload.

Or contacting Dynatrace ONE in case they can provide some log to you.

Best regards

Re: Dynatrace logs on problem notifications

MichalOlszewski — Tue, 25 Apr 2023 07:35:09 GMT

Hi @clc ,

Have You tried to confirm logs/errors on this 3rd party webhook?
Even if You will get logs from DT they will only contain info returned from this 3rd party. So for sure less than 3rd party logs itself.

So I would go with this option as it leads to faster solving issue.

Bests

Michal

Re: Dynatrace logs on problem notifications

clc — Tue, 25 Apr 2023 08:34:28 GMT

I am interested in confirming that Dynatrace in fact did send a problem notifications.

In this case it never arrived at 3rd party so for me the best way to diagnose this would be to start a Dynatrace end.

Event captured correctly - alerting profile affected on event - alerting profile correctly linked to problem notification - Custom webhook verified, test succesfull - 3rd party never received anything.

So would be nice to log that it actually did fire of this specific problem notification - I am aware the webhook is fire and forget so would not expecting a error message just verify that Dynatrace did its part.

-Claus

Re: Dynatrace logs on problem notifications

MichalOlszewski — Tue, 25 Apr 2023 08:42:01 GMT

Then You need to open support ticket to get some data form Support (log records)

Only scenarios when webhook failed for me was with SSL errors - DT is very strict about it and checks both: signing chain(beware of self-signed stuff) as well as SAN field (often missing - as most browsers aren't forcing it event thou they should).