Conditional alerting in DQL

[CTull]

I have a use case for our Command Center.

For example we have ingested two metrics, Metric1 and Metric2.

The case for this is to generate a workflow that is conditional.  Logic for the flow of the use case.

If event.name == "Title Of Custom Event" is OPEN 

Enable Problem Integration "Conditional Alarm Event to Service Now" using an API call.  

This would be something like this:

fetch events, from:-30m
| filter event.name == "Title Of Custome Event"  IS open

Then Enable Problem Integration to send events to trigger incidents for call out to support teams.

IF false then Disable Problem Integration.  

We are doing this as we only want visibility to an issue ONLY when a specific problem is met.

The reason why is this is only impactful when Metri1 and Metric2 reach a web traffic count of 1 million users.  This triggers alerts to be more sensitive for custom metric events.  We normally do nothing with these events when customer usage is low.  However, when customer usage is high we want the other alarms to start sending notifications when 5 custom events start triggering a failure rate over 5% in their api calls.  This problem will stay open as long as Title Of Custom event is open.  As soon as this problem goes away (When user activity drops below 1 million disable the integration by api call.  Is this possible?

[sinisa_zubic · Accepted solution]

Hi @CTull 

To best of my knowledge this should be possible.

Within workflows you have access to davis events. So you can set up a workflow with a time interval trigger to check if the conditions are met.

The question is if your two ingested metrics are available on Grail, you can check it out in our help: https://www.dynatrace.com/support/help/observe-and-explore/metrics/built-in-metrics-on-grail 

Also in the metrics explorer all metrics are flagged, if they are available on grail or not. If your 2 metrics are not yet available on grail, you could access them via the API or even via the query function of the metrics client 

Disabling and enabling problem notifications settings is also possible via api and the putSettingsObjectByObjectId function of the settingsObjectClient .

In case you have a services engagement or a dedicated product specialists, you can reach out to them to help you with the configuration.

Best,
Sini