DQL costs when querying spans – looking for advice

[michiel_otten]

Hi everyone,

I’m looking for advice on managing and controlling query costs in DQL, especially when working with spans.

In my recent experiments, I’ve been querying spans for about 1–2 hours, even with a 500GB limit in a notebook, and I’ve already incurred more than €70 in costs. At that rate, this could scale to roughly €25K per year, which feels quite high for platform usage. Therefore I'm guessing there is a better way to achieve certain informational needs.

I’d love to understand how others are handling this more efficiently. Specifically:

• What are best practices to control or reduce query costs when working with spans?
• Do you rely on approaches like pre-aggregation (e.g., creating metrics or business events) instead of querying raw spans?
• How do you structure or use buckets to reduce the amount of data scanned?
• Are there recommended strategies for limiting data (time ranges, filters, sampling, etc.)?
• How do you monitor and optimize expensive queries effectively? Do we need to create anomaly detectors?
• Any insights, patterns, or real-world examples would be greatly appreciated.

Thanks in advance!

Michiel

[FShimaya]

Hi Michiel,

Did you ever get a response?  I am also very interested if there is a solution.

[Fin_Ubels]

Hey Michiel,

Going through your questions:

What are best practices to control or reduce query costs when working with spans?
Do you rely on approaches like pre-aggregation (e.g., creating metrics or business events) instead of querying raw spans?
How do you structure or use buckets to reduce the amount of data scanned?
Are there recommended strategies for limiting data (time ranges, filters, sampling, etc.)?

For querying spans there are a few things I often do to help manage costs:

• Reduce timeframes - I always start building a span query by focusing on 1 trace in a very limited timeframe. Once I have my query working for the single focus I then expand my timeframe and remove limiting filters to allow myself to see how the query works with more than my single focus trace. This often works because I start by identifying a trace in the distributed tracing app that is of a type that I want to analyze so I can focus on that.
• Aggressively filter - Filtering on buckets is always the most efficient way of ensuring you query less data, however that requires data to already be put into different buckets which can be hard to achieve if not designed outright. If you don't have buckets but want to set them up, consider the following docs Use Grail buckets to partition data — Dynatrace Docs. For cases where a tenant is new to Grail or bucket separation wasn't designed at implementation time, other filters can reduce the amount of data queried. I have found the most luck filtering on things like smartscape IDs or common identifiers like k8s.* fields or host group fields.
• Pre-aggregation - Exactly! If you can identify a common pattern you will query all of the time, find a way to capture it as a metric rather than query spans all of the time across dashboards. If what you are querying can be converted to a metric using OpenPipeline then this is a great way to extract metrics. Unfortunately some use cases can't be extracted using OpenPipeline due to complexity of requirements. I have also found use cases where setting up a workflow to query spans repeatedly and the re-ingest the output as a metric is viable. If you can turn traces into a metric, it makes it easier to dashboard, alert on, and just generally use. Querying a timeseries is more approachable for novice users and its free (except for the ingest cost of course)! Transforming into metrics is truly the best approach and will be easier with time and a better understanding of what your span use cases are.

How do you monitor and optimize expensive queries effectively? Do we need to create anomaly detectors?

For monitoring/managing expensive queries from others I use 3 approaches:

• Alerting - In the account management portal you can set alerts for cost increases. This allows you to monitor unexpected spikes in usage. In your tenants as well, you can also transform the audit logs Dynatrace produces that track query consumption into a metric, then you can use that metric in an anomaly detector for more fine grained alerting on cost. While the anomaly detector option is an approach, I rely on the account management portal alerts.
• Dashboarding - You can use the ready made "Usage - Traces" dashboard to get a better understanding on where traces are being queried, by who, and what those expensive queries are. This dashboard will also provide the foundations of what queries can be run for your own more detailed analysis if required. I use a modified version of that ready made dashboard with additional use cases.
• Education - I set best practices guidelines for people that get onboarded into the environment I work in. We have a DQL guidelines doc that outlines things like "dont do fetch spans on a dashboard". This gives new users the dos and donts so that if they then excessively use, we can then talk to them easier going "so, we said don't do that. Whats the use case? Can we help you improve your practices?".