Davis Anomaly Detection

[anuj08_jain]

Hello Team,

We are learning Alert mechanism in Dynatrace and we have couple of questions regarding that. So, while migrating Splunk Alerts into Dynatrace.

What should I use?

• Davis Anomaly Detection?
• Or Workflow?

So, in Splunk i have alert like below:

source="win_security.json" host="dynatrace" index="win_sec" sourcetype="_json_custom"
| search "result.EventCode"=5379
| stats count("result.EventCode") count_EventCode as by "Account_Name"

Alert type - Scheduled - Run on Cron Scheduler at every minute (* * * * *) - Time Range - Last 15 minutes.

Custom Trigger condition -

where count_EventCode > 85

Trigger Action - Send Email.

Now i would like to migrate this Splunk alert into Dynatrace. I know this can be achieve via Workflow and workflow also cost as per every trigger. 

So, my question is should i have to create this alert via Davis Anomaly Detection? If so, then who i will send those alert on Email and also after the configuring the alert on Davis Anomaly Detection, what is the purpose of it? Does it only create metric events which we can use in notebook? or dashboard? 

Like what should i do after creating alerts via Davis Anomaly Detection?

[gschramm]

Hi,

I just dealt with a similar situation. In my case the event is a specific error code in a logline. When that error occurs X times within a specific timeframe a custom alert is created.

How I built it:
1. I use openpipeline to extract an event and I also create a metric that counts the occurances.
2. In the Davis Anomaly Detections I've created an anomaly that utilizes the metric. I'm using the advanced version:
- In my case the threshhold is 0 since the metric is always 0 when the issue does not happen. When a log line contains the errorcode the metric deviates from 0 > counts as violation.

- violation samples is 5 for me, so if it happens 5 times within the slideing window (60 = 60min) the alert is triggered

In your case that config would be something like violating samples = 85 and sliding window = 15 --> 85 events within 15min, similar to your Splunk config. For the DQL source you can use events and metrics, both will work.

[anujjain504]

Okay, I understand how the alert is generated. But my question is: what am I supposed to do with this alert afterward?
This alert is not sending me an email, nor is it creating an incident ticket. I can only see it listed under Problems in Dynatrace.

So my question remains the same: What should I do after an alert is created or triggered by Davis Anomaly Detection?

[gschramm]

You need to include the Problem in an Alerting Profile (Settings Classic > Alerting) and also configure how to be notified, under Settings Classic > Integrations > Problem Notifications

[dannemca]

If the alert is not triggered to any specific entity, you can use the custom properties to act as filter for the Alerting Profile.

[anujjain504]

I tried it, but I’m still unable to trigger any alerts. I think I might be using the wrong filter.
Is there any straightforward documentation available? The current Dynatrace docs are a bit confusing.

[O11y_Noob]

if your impacted entity of the problem created is the environment itself and not your specific entity  , try to change the dt.source_entity property in your alert to your entity which is impacting , that will tag it to MZ and relevant Problem notification, 

[sonja]

Hi @anuj08_jain!

I understand where your confusion might come from. Let me clarify how alerting and notifications work in Dynatrace.

ALERTING

1. Dynatrace automatically detects many issues and creates Davis events for them. These events are based on anomalies such as unusual service failures or performance degradations.

2. If you want more control, you can define your own custom alerts using Anomaly Detection. These allow you to create alerts based on specific metrics or thresholds.

3. To see which alerts have been reported, you can use Dynatrace Query Language (DQL) to fetch events with the query: fetch dt.davis.events.

   These events are also visible in various parts of the product to highlight issues like increased failure rates or performance anomalies.

4. Since many events are related, Dynatrace groups them into Problems. This is what you see in the Problems App. You can also query these problems using DQL with the same fetch dt.davis.events query.

NOTIFICATIONS

Classic Notifications:
In the past, we recommended using problem notifications and alerting profiles for setting up notifications. While this is still supported, the recommended approach has shifted.

Workflows for Notifications:
We now recommends using Workflows to set up notifications. Workflows provide greater flexibility and allow you to define custom actions for different scenarios. Think about a workflow like a notification channel (who needs to be notified, what information do they need, on what kind of issues should they be notified). 

You can decide if you want to send notification for each new problem (problem trigger) or for each Davis event (Davis event trigger).
 

Example:
Start by creating a simple workflow that sends an email notification for every problem.
Once you’ve confirmed this is working, you can extend the workflow to include other notification channels (e.g., Slack, Microsoft Teams, etc.) or refine it to send notifications only for specific, critical problems.
Use filters in the workflow trigger to define which problems should generate notifications.

Here is a tutorial in the documentation: https://docs.dynatrace.com/docs/analyze-explore-automate/workflows/use-cases/workflows-tutorial-prob....

I hope this helps!
Sonja

[gschramm]

As Sonja explained Workflows are a powerful tool. However I recommend caution as they may introduce additional costs. Only single-step workflows (eg the send-email example) are free of charge. Once you add more steps worksflows will be billed by the hour according to your rate card (Calculate your consumption of Automation Workflow (DPS) — Dynatrace Docs)

[anuj08_jain]

Thanks for the detailed Information. It helps me a lot to create Anomaly Detection. However, I still have one question. 
So, basically i have created the alert and its getting triggered. But while checking under the "Problems" I can see that the alert is taking Default alert Profile. However, I have already created a new custom alert profile but its not taking that one. 

Is there any way to change the alert profile or specifying custom alert profile in Davis Anomaly Detection?