We are currently trying to resolve an issue with low disk space problems in our Dynatrace environment.
Dynatrace is opening low disk space problems for a large number of discovered disks, but the client never requested alerting for all of them. We only want problems to be generated for disks that the client has explicitly asked us to monitor.
The environment contains thousands of disks, so creating individual anomaly detection overrides for every disk that should not alert does not seem scalable or maintainable.
There is also another requirement: the client can define different thresholds for different disks.
For example:
Disk A: alert below 5% free
Disk B: alert below 10% free
Disk C: alert below 15% free
Disk 😧 no low disk space alerting
Ideally, we would like an opt-in model where only explicitly configured disks are evaluated, and each disk can have its own threshold.
We would also prefer not to use tags as the configuration mechanism. While tags could probably be used to build a custom solution today, we are trying to avoid designing this around tags because of the direction Dynatrace is moving with Grail, Smartscape, newer entity models, DQL-based configuration, and newer anomaly detection capabilities. We would rather use a mechanism that is more aligned with the current and future Dynatrace platform.
We have been looking at options such as:
Disk Edge anomaly detection
Grail lookup tables
DQL-based anomaly detection
Record-based anomaly detection
Any newer native configuration mechanism
Conceptually, the ideal configuration would be something like:
Disk Threshold
| Disk A | 5% |
| Disk B | 10% |
| Disk C | 15% |
Any disk not present in that configuration would simply not generate a low disk space problem.
The main goal is to avoid maintaining thousands of individual Settings API objects or anomaly detection overrides while still keeping the resulting alerts correctly associated with the DISK entity.
Has anyone implemented something similar with the newer Dynatrace capabilities?
Is there currently a recommended/native approach for managing per-disk alerting scope and individual thresholds at scale?
