Impact of Default Transformation on Metric Events

[arnel97]

Dear Dynatrace Community,

I have been searching for quite a while on the web without any success for the following scenario:

One of our customer's requirements is to display the latencies of specific requests between two service components in a dashboard and also configuring alerting based upon agreed latency thresholds. For the calculation of the latencies, it was agreed that data gaps should also be interpreted as "0" and not "no data". This was managed with the default transformation "default(0)".

Question: The question now arises as to how this affects alerting if we activate "Alert on missing data"?

At the moment we assume "default(0)" replaces our data gaps "no data" with "0" and therefore this would mean that data is not missing, because"0" requests have been received . Is this correct?

Question 2: But what does now "no data" in this context mean? If the host goes offline, the service status gets inactive, OneAgent loses connection to Dynatrace, etc. I am asking this question out of curiosity, because if the host goes offline or the service is inactive, but "alert on missing data" is turned on and "default(0)" also in place, will an alert be triggered or will this behave differently?

I have read a few topics related to the default transformation and alerting, just to mention some:

• Replace "no data" with "0" in a dashboard
• Display "0" for no data in the custom chart
• Metric event alerting

However, I did not find anything on the official documentation on the exact behaviour when "default(0)" is in place and "Alert on missing data" turned on.

I would appreciate any help and also references if possible on this subject.
Thank you,
Arnel

[Mizső]

Hi @arnel97,

Default transformation can be use on a visualization. Your metric timeseries will not extened with the default value just for the visualization purposes.

I hope it helps.

Best regards,

Mizső  

[arnel97]

Hi @Mizső 

thank you very much for your answer, but is the default transformation really only used for visualization purposes? Because when we use default(0) for "no data" and also turn "alert on missing data" on, will we receive alerts for default(0) or "no data"? I cannot find anything related to this on the official Dynatrace documentation page.

Br,
Arnel

[Mizső]

Hi @arnel97,

Sorry for my late response. It is a good question and maybe I have ot change my previous answer becasue you can create an metric event from a metric expression where you can use the default(0). Mybe in this case there will not be "alert on missing data" only if the metric provider is down. I have to try / check it.

Best regards,

Mizső

[GosiaMurawska]

Hi @arnel97 
Have you managed to solve the problem? It would be great if you shared the solution with the Community users. 🙂 

[arnel97]

Hello @GosiaMurawska 

I am not sure myself but it is documented here that e.g. if the sliding window is set to 3 minutes during any 5 minutes, Dynatrace triggers an alert if there's no data within a 3-minute period. But what does no data in the context of metrics mean? Does that correspond to "null" or "0" after the default transformation, or does it mean something else that has nothing to do with metrics?

To give some example scenarios that I have not tried yet:

Scenario A:
My service is running and we see datapoints in the metrics, 0 also counts as data after applying the defaut(0) transformation, but all of a sudden I start receiving alerts because "data is missing". Of course data is missing as expected, because my service is not sending anything, but still active. This is a false positive alert.

Scenario B:
2 weeks later, my service goes down and then I receive alerts because "data is missing". In this case it does make sense, because the service went offline for whatever reason and we still see datapoints in the metrics because of the default transformation that turns "no data" into "0".

Scenario A throws false positivies. Scenario B is the expected behavior, however note that if the service goes down the default transformation should realize that "no data" in this case means something entirely else and not "0". This brings me to Scenario C.

Scenario C:

My service goes offline, but I receive "data is missing" alerts. Additionaly, no datapoints are visible in the metrics.

Scenario C makes more sense than B, because in this case we see no datapoints in the metrics at all and we know for sure that default(0) cannot turn "missing data" into a "0", because it only turns "no data" into "0".

This causes our customer a lot of confusion and we decided to turn the feature "alert on missing data" off. However, it is possible to find more information about missing data under Settings>Anomaly detection>Metric events:

The ability to set an alert on missing data in a metric. When enabled, missing data samples will contribute as violating samples defined in advanced model properties. We recommend to not alert on missing data for sparse timeseries as this leads to alert spam.

But even here it is still unclear if "missing data" corresponds to "null" or "0" after applying the default transformation in metrics.

Br,
Arnel

[Julius_Loman]

It depends if you use the always argument or not. The :default transformation without any arguments, just replaces datapoints with the chosen value, but at least one datapoint in the queried interval must be present.
If you use also the always argument, the missing datapoints are replaces also if there isn't any datapoint.

Btw - the service itself doesn't go down. The underpinning process may be shut down/crashed/etc. Service is just an artificial representation of a set of requests.