Custom Log Metric Never Shows Value of 0 - Did Not Alert When Expected To

Platform Alerting
, , ,
1

Good afternoon.

I created a custom Log Metric that tells us each time a new account is created.

In the logs, any time we see “/member/onboarding/create_account” (for example), this means that a new account has been created.

For every account created, there is one corresponding log record created in our logs. It’s a one to one relationship. So using Occurrence of logs records for my metric measurement makes sense. In the metadata, I’m using Count and the Value type of Score, as high values indicate a good situation.

Because these values go up and down throughout the day, using the Seasonal Baseline Model monitoring strategy made sense to me.

However, today, we had about 40 minutes worth of ZERO accounts being created, as shown in the attached graph.

Dynatrace failed to alert, and I can’t quite understand why. As the failure conditions were perfect.

I also noticed that I never see a ZERO value in Data Explorer. The lowest number displayed is 1 or it’s just blank. It never drops to zero. Why is this the case?

Any thoughts as to why Dynatrace did not alert us, even when there was a clear sign of zero occurrence of logs records?

Thank you.

Log_Metric_Graph.png
Log_Metric_Graph.png3015×1124 27.2 KB

2

Dynatrace don’t treat absence of values as a 0

If you want to set all missing values to 0 you should use :default() transformation. 
Metrics API - Metric selector — Dynatrace Docs
Try with a metric selector and default transformation.

3

Hello,

This is normal behavior how Dynatrace captures it is either we have data or we don’t have, so is either more than 1 or there is no data at all.

To solve this you have two different approaches:

  1. In the Metric Event select the toggle that says “Alert on missing data” this will make Dynatrace create a problem everytime there is no data, that in your case is when there are no new accounts. I don’t feel this is the right approach because it’ll overalert.
  2. Use the default in the metric selector definitio, will look something like the next:
queue.length:splitBy("host","queue","type"):filter(ne("queue","csr_process"),ne("queue","onprem")):avg:default(0, always)

The second approach let you define that when the value doesn’t exist by default it will be 0, then Davis will be able to analyze it and alert if it deviates from the normal behaviour.

Hope this helps. Wish good monitoring!

4

Thank you, @PacoPorro and @jaume_reverte.

This is exactly what we needed to fix our problem. I kept thinking it was a problem with how I built the Log Metric. :sweat_smile:

With this solution, we were able to switch to a Static Threshold monitoring strategy and trigger alerts successfully.