topic Re: Metric Events and number of metric dimensions in Alerting

Metric Events and number of metric dimensions

AntonioSousa — Tue, 13 Aug 2024 10:44:54 GMT

I have a situation where I am trying to understand why one particular metric has so many dimensions in the "Alert preview" in Metric Events, when used with the rollup() transformation:

But starting with the beginning, the metric in question, with the specific filter used in the metric expression, has only 81 entries/combinations in the last 24 hours, when displayed in Data Explorer, in Table mode.

If I start with no rollup, I get the following in "Alert preview" in Metric Events:

Now, the first calculation that doesn't make sense to me is that 81 x 614 = 49734. So I have only 81 entries, why is this multiplied by the 614 value?

I had to try to figure out where the 614 was coming from. 614 / 81 doesn't give an integer value.

So, I started out with several metric expressions. For each the corresponding number of "metric dimension values" in Metric Events:

metric.test
The number of metric dimension values in the last 24 hours (27269)
metric.test:splitBy(name,tag)
The number of metric dimension values in the last 24 hours (27269)
metric.test::filter(and(or(eq(event_type,OK)))):splitBy(name,tag)
The number of metric dimension values in the last 24 hours is 614
metric.test::filter(and(or(eq(event_type,OK)))):splitBy(name)
The number of metric dimension values in the last 24 hours is 614
metric.test::filter(and(or(eq(event_type,OK)))):splitBy(tag)
The number of metric dimension values in the last 24 hours is 614
metric.test::filter(and(or(eq(event_type,OK)))):splitBy(name,tag):avg:rollup(avg,60m)
The number of metric dimension values in the last 24 hours (49734)

Now, if I do a count of how many datapoints there are in the last 24 hours, in Data Explorer, I have 78686 (BTW, last 24 hours are evolving as I write this post). The data series is sparse, with values every 5 minutes, and only when the metric has values. And yes, this metric has 38 dimensions...

Now, back to 614. While writing this, it changed to 613. 613 is prime, so it's not related to any multiplication, so now I'm not sure where it comes from.

So, I restarted calculations with only the metric, with the value above, 27269. And I found out several values, with the following factors:

27269
11, 37, 67, 407, 737, 2479
27265
5, 7, 19, 35, 41, 95, 133, 205, 287, 665, 779, 1435, 3895, 5453
27268
2, 4, 17, 34, 68, 401, 802, 1604, 6817, 13634

So, with no common factors, these values seem to be related to the amount of datapoints that are being recorded, but once again they do not seem to have a relation.

I also tried to use limit(), but it only got more confusing to me...

So, is there any info out there on how this calculation is made, so we can setup Metric events that make sense, and that are eventually not throttled?

Re: Metric Events and number of metric dimensions?

kalle_lahtinen — Mon, 27 May 2024 10:24:28 GMT

I was struggling with a bit similar issue. My selector produced 59 unique dimensions when counting them in table view. Below is the selector, it shows OS service availability in percentage format:

(builtin:osservice.availability:splitBy("dt.entity.os:service","dt.entity.host"):sort(value(auto,descending)):limit(100)/6)*100

...But the metric event definition said there are 3481 dimensions, i.e. too many. That number is 59 times 59 actually.

Then I removed the "limit(100)" definition:

(builtin:osservice.availability:splitBy("dt.entity.os:service","dt.entity.host"):sort(value(auto,descending))/6)*100

And now it works! It says 59 dimensions are there. I don't really understand how it makes sense, but removing the limit fixed it in my case. These are mysterious things sometimes...

Re: Metric Events and number of metric dimensions?

dannemca — Mon, 27 May 2024 15:03:24 GMT

In your case, removing the limit will make it the default, of 20 entries.

Not sure if limit is used in metric event, but in Data explorer, that's how it works.

Re: Metric Events and number of metric dimensions?

AntonioSousa — Mon, 27 May 2024 16:18:53 GMT

@dannemca,

Great point! But I believe in metric events it doesn't default to 20, or the expressions that I mentioned above would also not have issues...

Re: Metric Events and number of metric dimensions

GosiaMurawska — Tue, 13 Aug 2024 10:54:39 GMT

Hi @AntonioSousa
Have you managed to solve the problem? It would be great if you shared the solution with the Community users 💡