topic Re: [Hack of the week] expand Metric Event duration simply with rollup in Dynatrace tips

[Hack of the week] expand Metric Event duration simply with rollup

uros_djukic1 — Tue, 30 Jul 2024 12:34:53 GMT

This trick is applied using the metric expression (Metrics API - Metrics Selector).
[Batch/ Job case] :
To meet a particular need, in particular that of a batch/job which fails randomly and thus generates only one piece of data at a time, we will very quickly be inundated with alerts. Compared to the IA limitation (Sliding window + Dealerting samples), no more than 60min.

rollup transformation is the solution :

Like I said, if you're only using a single rollup, we're dealing with limitations, so missing data will end the problem...

*** WORKAROUND
To answer this we can use the rollup transformation. As illustrated below, we see how to extend the analysis window when we have detected the failed event at least once in the last 24 hours.

The advantage is to therefore avoid the repetition of the known event which will require correction.
Therefore, reduces the number of problems in the Dynatrace sense and avoids spamming notifications.

Given the 60 minutes limitation on rollup, 24 was set to ensure we cover one time slot per day and is very useful if at least 1 single datapoint is generated (e.g. metric C above = red area).

Metric B = represents the number of times the batch fails (purple points).

Result of detection:

Thanks

Re: [Hack of the week] expand Metric Event duration simply with rollup

AntonPineiro — Thu, 25 Jul 2024 10:46:28 GMT

Thank you!

Re: [Hack of the week] expand Metric Event duration simply with rollup

AntonioSousa — Thu, 25 Jul 2024 18:15:36 GMT

@uros_djukic1,

Your multiple rollup is really an interesting hack. Also gave me some ideas, but it seems something like an abuse 😂

Rollup rapidly gets out of control when used with most metrics. You might get affected by throttling quite fast, as I documented in https://community.dynatrace.com/t5/Alerting/Metric-events-that-sum-values-in-a-certain-time-period/m-p/217633 but really would have to test how it goes with your idea ...

Re: [Hack of the week] expand Metric Event duration simply with rollup

DanielS — Wed, 21 Aug 2024 13:26:49 GMT

Good Trick!!!! Thanks for share.

Re: [Hack of the week] expand Metric Event duration simply with rollup

dannemca — Wed, 21 Aug 2024 13:57:58 GMT

Wait, we can hack the rollup to higher than 1h? Wow, lol!!

That's awesome!