Myself and a colleague were looking around at the ready-made dashboards and seeing what we might want to use in the future. We stumbled upon the "Azure AI Foundry - AI Observability" dashboard and took a look at what panels it offered. Of course, none of the panels populated with any data, since we have yet to onboard any OpenLLMetry spans in our tenant. So it was just a completely blank dashboard that showed the capabilities of AI Foundry monitoring. I should note that the sampling ratio dropdown at the top of the page was set to "1" by default and the timeframe selector was set to 2 hours.
Later on that day, I checked out recent costs in Account Settings and saw a large spike in Trace Query costs, despite there not having been any reason for such a spike to occur. I dug around in the dt.system.events table and discovered that every time we opened the blank ready-made dashboard, Dynatrace queried over 2,700 GiB of data in our Grail, meaning we were incurring several dollars just for opening the dashboard because it was querying our entire set of spans for the last 2 hours.
This can't be expected behavior, right? I know the usual answer to log management boils down to segments, OpenPipeline, and smarter querying, but none of that applies when you're just browsing ready-made dashboards of technologies you don't even have onboarded to Dynatrace. Am I doing something wrong here or is there something very wrong with having a dashboard that is essentially a $5 charge every time you open it?
