Time-based Custom Metrics Population

[skirkham]

We would like to be able to report monthly figures for metrics that are captured from RUM on Grail. However, given the volume of available traces we hit DQL Query limits from just a few days of history - so a 12 monthly view is going to be impossible.

We are currently looking at custom metrics built from our own application but as I read the documentation on customer metrics I can see that there is a limitation for the timestamp (no later than 1 Hour and no earlier than 10 Minutes). As we would be waiting for the start of the new month, which would then trigger the specific app function to gather the previous months data and summarize it and then report it against the first day of the month - isn't going to work.

Is anybody doing something similar to this, if so how are you storing the data (doc or metric) and how are you passing the information (api or pipeline)?

Thanks

[sujit_k_singh]

Hi @skirkham 

I am bit late here 😀 however let me add some cents if it help others

This is a well-known challenge with high-volume RUM data in Grail.

You're correct the Dynatrace metric ingest API enforces a timestamp window of no earlier than 1 hour in the past and no later than 10 minutes in the future. This means you cannot backfill a monthly aggregate metric with a timestamp of "first day of last month" after the fact. That approach won't work as-is.

Recommended Architecture: Continuous Aggregation via OpenPipeline + Metrics
The cleanest long-term solution is to not aggregate retroactively — instead, aggregate continuously at ingest time so the data is already summarized before it hits Grail storage.

Use OpenPipeline → User Sessions pipeline to extract custom metrics from RUM events as they arrive
Define counter or gauge metrics with the dimensions you care about (e.g., app name, region, user segment)

These metrics land in Grail's metric store with correct timestamps and are queryable over any time range

Result: Instead of querying millions of raw user.sessions records for a month, you query a pre-aggregated metric timeseries — which is orders of magnitude cheaper and has no DQL scan limits in practice.

Setup path: Settings → OpenPipeline → User Sessions → Pipelines → Add metric extraction rule

Thanks,

Sujit

[skirkham]

Hi @sujit_k_singh,
Thanks for the info, the continuous aggregation approach is something we have used elsewhere and it works great in the majority of the situations that we have needed a longer term view... but good to have it document.

Simon