Metric to track high cardinality

[lucrisca]

We use this metric "Metric Server - Metrics - Metric Dimensions Usage (
dsfm:server.metrics.metric_dimensions_usage)
" to track the growth of metric dimensions and prevent them from stopping due to high cardinality.

The point is that the metric is limited to dimensions >500k or 50% usage.

We would like to have access to visibility of data below 50% / <500k to track growth and direct teams to adjust before it becomes a problem.
If possible by metric, but if someone has an example for us to create this metric by logs, it would help a lot.

[nickvogt]

Hi @lucrisca,

Could you tell me more about how you're using this dsfm metric to track growth and direct teams?

Take care,
Nick

[lucrisca]

hello nick,

We have a “Metrics ingest limits” dashboard to track this growth.
When it's a team metric, We contact the microservice owner. When it's builtin dimensions, we validate whether any configuration changes in the clusters have caused this increase.

A dashboard was made available for the teams to monitor the ingestion of their metrics and correct the cases that are ingesting more data. The feedback received from the teams was to present the data < 50%.

[nickvogt]

Hi @lucrisca,

I'll offer an alternative: you can look for spikes by counting dimensions on a metric. The example metric selector below counts the number of dimension tuples on a kubernetes metric. With it you can create a line chart and quickly find spikes in dimension usage, and your teams can adapt it for their metrics.

builtin:kubernetes.container.restarts:auto:splitBy():count

This dashboard is designed to help diagnose high cardinality dimensions and metrics. We are pretty clear on our product roadmap for the next six months and have a good idea of where we are headed for the year or two after that. What you’re suggesting sounds interesting but doesn’t fit well here. You can, however, approach other users in Community and see if this really is something lots of people want.

Take care,
Nick

[GosiaMurawska]

Hi @lucrisca 
Have you managed to solve the problem? It would be great if you shared the solution with the Community users 🙂