How are you reducing alert noise in Dynatrace SaaS?

[doodledaron]

I’m looking for some advice from the Dynatrace community on best practices for reducing alert noise in Dynatrace SaaS.

So far, I’ve explored:

● SLO-based alerts — but my understanding is that SLO alerts are not really designed to reduce alert noise. We would still need to configure Custom Alerts/Problems appropriately.

● ServiceNow Workflows — this seems like a good approach for filtering and managing alert noise, but the customer cannot subscribe to the ServiceNow Workflow module due to budget constraints.

What other approaches or best practices would you recommend for reducing alert noise directly within Dynatrace SaaS, without relying on ServiceNow or additional paid modules?

For example, are there recommended ways to use Problem notifications, alerting profiles, management zones, event filters, Davis AI, or other Dynatrace-native capabilities to achieve effective alert/noise reduction?

Would appreciate any real-world approaches or examples you’ve used. Thanks!

[Mohamed_Hamdy]

I would first analyze the existing Problems and alerts before changing the configuration. In many environments, a small number of alert sources, entities, services, or recurring conditions may be responsible for most of the noise.

Start by identifying:

• The most frequent Problems/events

• The entities generating them

• Whether the source is built-in anomaly detection, Synthetic, Kubernetes, extensions, custom alerts, etc.

• Whether the alerts are actionable incidents, transient spikes, expected behavior, or duplicate/unnecessary detection

Once you identify the main sources, I would focus on:

1. Tune the relevant anomaly detection at the source. This includes both Dynatrace built-in detection and custom alerts. For built-in service, application, infrastructure, and database detection, review sensitivity, automatic baselines versus fixed thresholds, minimum traffic, persistence/duration, and entity-specific overrides where applicable. Dynatrace explicitly supports tuning these settings to avoid over-alerting.

2. Review custom alerts separately where they contribute to the noise. Choose the appropriate detection model, thresholds, aggregation, and sliding window, and avoid unnecessary high-cardinality dimensions that can turn one condition into many alerts. Dynatrace's current overalerting guidance specifically recommends aggregated data, suitable detection models, and sliding windows.

3. Use Maintenance Windows for deployments, patching, DR drills, load tests, and other planned activities where abnormal behavior is expected.

4. Improve alert scoping and context. For new SaaS designs, I would explore Primary Grail fields/tags and Segments rather than building new dependencies only around Classic auto-tags and Management Zones. Good context such as environment, application, team, and criticality allows detectors to be scoped more precisely.

5. Use Latest Dynatrace SLOs and error-budget burn-rate alerting for critical services where appropriate. This can complement technical alerting by focusing on actual service reliability, but it doesn't replace tuning the underlying anomaly detection.

So overall:

Analyze the noise → Identify the noisy alert sources → Tune built-in and custom detection → Scope alerts correctly → Suppress planned activity → Use SLO burn-rate alerting for critical services.

The main principle is to reduce the noise as close to its source as possible, rather than only filtering it later in ServiceNow or at the notification layer.