Where I can check my custom events created from anomaly detection using dql over logs data

[heramb_sawant]

Hi,
I  have created anomaly detection to capture specific error from the logs.In that I have defined dql that get timeseries data for the error occurrence for particular k8s application. Statis threshold is defined.

In the anomaly list , status shows "success". I verified target error has occurred , seen in the logs as well.But somehow  I am not able to see my custom event in in grail-> events.

Can somebody please help me to find all the custom events in grail which are related to all the Anomaly detection.


Note - My final goal is to trigger workflow if custom events occurs.

Any help will be appreciated.


Regards,
Heramb Sawant

[heramb_sawant]

Any help would be appreciated.

[t_pawlak]

Hi,
dt.system.events will usually not show the actual custom event raised by anomaly detection. That bucket is mainly for detector execution/status information, such as ANOMALY_DETECTOR_STATUS_EVENT and ANALYZER_EXECUTION_EVENT—for example, whether the detector ran successfully, timed out, or produced warnings
If the anomaly detection configuration really raised an alert, I’d look for it in Davis events, for example:

fetch events, from: now()-24h
| filter event.kind == "DAVIS_EVENT"
| sort timestamp desc

and then narrow it down using your event template fields / event name if available. Dynatrace treats events from custom alerts as Davis events, which can then be correlated into Problems

So I’d separate it like this:

• dt.system.events → anomaly detector execution health/status
• events / Davis events → actual alert events raised by the detector
• Problems → correlated incidents created from those events

System event models 
For example, if you want to see the related active problems created from custom alerts:

fetch dt.davis.problems, from: now()-24h
| filter event.status == "ACTIVE"
| filter event.category == "CUSTOM_ALERT"
| fields timestamp, display_id, event.name, event.description, event.category, event.status, affected_entity_ids
| sort timestamp desc

So if your final goal is to trigger a workflow, I’d first verify whether the anomaly detector is producing a `CUSTOM_ALERT` Davis event, and then decide whether to trigger on the event itself or on the correlated problem.

[AntonPineiro]

Hi,

Maybe you can check System event models.

Best regards

[heramb_sawant]

Thanks t_pawlak, you shared very useful information and thanks to AntonPineiro

I referred System event models. ,  checked  below DQL, and found my task is failed with reason "The task was disabled by the service operator to maintain system stability. Contact support if the task keeps failing."

fetch dt.system.events, from:now()-24h
| filter event.kind == "ANALYZER_EXECUTION_EVENT"

what you will suggest  on this?? How it can be enabled.

Note - DQL defined in anomaly detection is  pulling timeseries data capturing specific 4xx error.

[heramb_sawant]

Any inputs ??

[heramb_sawant]

any further help will be appreciated 

[t_pawlak]

Hi,

I tested the same DQL on my side:

fetch dt.system.events, from:now()-24h
| filter event.kind == "ANALYZER_EXECUTION_EVENT"

and the query itself works, so this does not look like a DQL syntax issue.

Based on the error message:

"The task was disabled by the service operator to maintain system stability. Contact support if the task keeps failing."

it looks like the specific anomaly detection task was automatically disabled by Dynatrace safeguards, most likely because the detector execution was considered too expensive or unstable.

So I would suggest review the anomaly detector query and try to reduce its cost, like narrow the timeframe, or use some filters. Make it less expensive. 
And if the issue continues, opening a Dynatrace Support case, because the message suggests a service-side safeguard.

[sonja]

hi! in addition to all the useful comments, I wanted to share this new page in the documentation that helps better understand how everything fit together: https://docs.dynatrace.com/docs/analyze-explore-automate/alerting-and-notifications including this graphic.

I hope this helps,
Sonja