Why does metrics API try to apply filter to a dimensionless metric?

[ext-Antti_Elonh]

I have a problem trying to configure alerting (custom anomaly detection) where the calculation includes one dimensionless metric and another with the wished dimensions. The same metric expression works great in Data explorer and dashboards but not in alerting because the metrics API tries to apply the filter also to the dimensionless metric.

The metric expression uses Prometheus metrics exposed by cert-manager like this:

(certmanager_certificate_renewal_timestamp_seconds:splitBy("name","namespace","k8s.deployment.name","dt.entity.kubernetes_cluster")/86400)-(certmanager_clock_time_seconds_gauge:splitBy()/86400)

 
I have checked the Dynatrace server logs and the actual alerting component queries metrics API for each combination with selector as this:

((certmanager_certificate_renewal_timestamp_seconds:splitBy(name,namespace,\"k8s.deployment.name\",\"dt.entity.kubernetes_cluster\")/86400)-(certmanager_clock_time_seconds_gauge:splitBy()/86400)):filter(eq(\"dt.entity.kubernetes_cluster\",KUBERNETES_CLUSTER-4F32D9E28C22C7F1),eq(\"k8s.deployment.name\",cert-manager),eq(name,test-cert),eq(namespace,test-namespace))

If I try that query in Data explorer it fails because it tries to apply the filter also to that dimensionless metric "certmanager_clock_time_seconds_gauge:splitBy()" - it's made dimensionless with that empty splitBy.

I'm just wondering why does Dynatrace even try to apply a filter to a dimensionless metric because it would always fail?

[wolfgang_beer · Accepted solution]

Yes that is a bug we recently encountered in the alerting module. We are working on a fix.

[ext-Antti_Elonh]

I understand that this can be fixed in the alerting module also but I think there should be a fix in the metrics API as well.

The documentation basically says that a fixed number and a dimensionless metric behave similarly (creating tuples etc.) except that a dimensionless metric is a series. They still don't behave the same regarding that filter transformation. I think they should both be excluded from filter transformation and it might have solved the issue also in this other topic: https://community.dynatrace.com/t5/Dashboarding/I-can-t-create-latency-SLO-using-calculated-metric/m...

[wolfgang_beer]

Yes sorry I was not precise enough. We are working on a fix at the moment and the fix is done in the metric query subsystem that includes a fix on API as well as on alerting.

Best greetings,

Wolfgang

[ext-Antti_Elonh]

Hi, @wolfgang_beer . Could you tell in which version of a Managed Dynatrace can we expect the fix?

[wolfgang_beer]

Hi @ext-Antti_Elonh there are 2 topics still upcoming for DT Managed support which are:

- View the list of problems (precondition is Managed Release 240) 

- Push notifications for Managed Environments (expect that in Mai)

Best greetings,

Wolfgang

[ext-Antti_Elonh]

Hi,
I was wondering about the fix to metric query problem in anomaly detection.

[wolfgang_beer]

According to what I see from dev side, this reported issue was fixed with Dynatrace release 239. 

Best greetings,

Wolfgang

[ext-Antti_Elonh · Accepted solution]

Our managed cluster was updated to v1.240 last night and these custom alerts started to work as expected.

[wolfgang_beer]

@ext-Antti_Elonh Excellent, many thanks for sharing this confirmation with us!

Best greetings,

Wolfgang