topic Re: Custom Log Metric Never Shows Value of 0 - Did Not Alert When Expected To in Alerting

Custom Log Metric Never Shows Value of 0 - Did Not Alert When Expected To

ASE — Tue, 18 Jun 2024 07:05:35 GMT

Good afternoon.

I created a custom Log Metric that tells us each time a new account is created.

In the logs, any time we see "/member/onboarding/create_account" (for example), this means that a new account has been created.

For every account created, there is one corresponding log record created in our logs. It's a one to one relationship. So using Occurrence of logs records for my metric measurement makes sense. In the metadata, I'm using Count and the Value type of Score, as high values indicate a good situation.

Because these values go up and down throughout the day, using the Seasonal Baseline Model monitoring strategy made sense to me.

However, today, we had about 40 minutes worth of ZERO accounts being created, as shown in the attached graph.

Dynatrace failed to alert, and I can't quite understand why. As the failure conditions were perfect.

I also noticed that I never see a ZERO value in Data Explorer. The lowest number displayed is 1 or it's just blank. It never drops to zero. Why is this the case?

Any thoughts as to why Dynatrace did not alert us, even when there was a clear sign of zero occurrence of logs records?

Thank you.

Re: Custom Log Metric Never Shows Value of 0 - Did Not Alert When Expected To

PacoPorro — Tue, 18 Jun 2024 07:31:28 GMT

Dynatrace don't treat absence of values as a 0

If you want to set all missing values to 0 you should use :default() transformation.
https://www.dynatrace.com/support/help/shortlink/api-metrics-v2-selector#default
Try with a metric selector and default transformation.

https://docs.dynatrace.com/docs/shortlink/metric-events-metric-selector-events

Re: Custom Log Metric Never Shows Value of 0 - Did Not Alert When Expected To

jaume_reverte — Tue, 18 Jun 2024 07:36:55 GMT

Hello,

This is normal behavior how Dynatrace captures it is either we have data or we don't have, so is either more than 1 or there is no data at all.

To solve this you have two different approaches:

In the Metric Event select the toggle that says "Alert on missing data" this will make Dynatrace create a problem everytime there is no data, that in your case is when there are no new accounts. I don't feel this is the right approach because it'll overalert.
Use the default in the metric selector definitio, will look something like the next:

queue.length:splitBy("host","queue","type"):filter(ne("queue","csr_process"),ne("queue","onprem")):avg:default(0, always)

The second approach let you define that when the value doesn't exist by default it will be 0, then Davis will be able to analyze it and alert if it deviates from the normal behaviour.

Hope this helps. Wish good monitoring!

Re: Custom Log Metric Never Shows Value of 0 - Did Not Alert When Expected To

ASE — Tue, 18 Jun 2024 22:09:39 GMT

Thank you, @PacoPorro and @jaume_reverte.

This is exactly what we needed to fix our problem. I kept thinking it was a problem with how I built the Log Metric. 😅

With this solution, we were able to switch to a Static Threshold monitoring strategy and trigger alerts successfully.