[Hack of the week] expand Metric Event duration simply with rollup

This trick is applied using the metric expression (Metrics API - Metrics Selector).
[Batch/ Job case] :
To meet a particular need, in particular that of a batch/job which fails randomly and thus generates only one piece of data at a time, we will very quickly be inundated with alerts. Compared to the IA limitation (Sliding window + Dealerting samples), no more than 60min.

rollup transformation is the solution :

Like I said, if you’re only using a single rollup, we’re dealing with limitations, so missing data will end the problem…

*** WORKAROUND
To answer this we can use the rollup transformation. As illustrated below, we see how to extend the analysis window when we have detected the failed event at least once in the last 24 hours.

The advantage is to therefore avoid the repetition of the known event which will require correction.
Therefore, reduces the number of problems in the Dynatrace sense and avoids spamming notifications.

Given the 60 minutes limitation on rollup, 24 was set to ensure we cover one time slot per day and is very useful if at least 1 single datapoint is generated (e.g. metric C above = red area).

Metric B = represents the number of times the batch fails (purple points).

Result of detection:

Thanks

11 Likes

Thank you! :money_mouth_face:

1 Like

@uros_djukic1,

Your multiple rollup is really an interesting hack. Also gave me some ideas, but it seems something like an abuse :joy:

Rollup rapidly gets out of control when used with most metrics. You might get affected by throttling quite fast, as I documented in https://community.dynatrace.com/t5/Alerting/Metric-events-that-sum-values-in-a-certain-time-period/m-p/217633  but really would have to test how it goes with your idea …

1 Like

Good Trick!!! Thanks for share.

1 Like

Wait, we can hack the rollup to higher than 1h? Wow, lol!!

That’s awesome!

1 Like