03 May 2025 02:20 AM
We have a memory close to the limit (memory saturation problem for one of our services).
We want to mute this alert, just for this service. However, use API to look up the json (attached) I am not able to filter it out:
I have tried:
Custom: Title not contains 'Memory usage close to limits' AND event property k8s.workload.name not equals 'my service name'
Custom: Title not contains 'Memory usage close to limits' AND event property dt.event.deployment.name not equals 'my service name'
Custom: Title not contains 'Memory usage close to limits' AND event Entity not equals 'my service name'
None of them works
05 May 2025 06:00 AM
Hi @qizhang1
Did you allready try this, Alert on common Kubernetes/OpenShift issues — Dynatrace Docs
you can disable the Memory alert just for one workload.
07 May 2025 06:34 AM - edited 07 May 2025 06:34 AM
Your screenshot above is to disable Memory Saturation alert in the namespace, not just one workload, unless my understanding is wrong.
10 May 2025 01:34 AM
You're right—this applies at the namespace level. However, you can always leverage automatic tagging to fine-tune control.
A good approach is to create a global automatic tag rule (e.g., monitoring=true) and configure alerting or SLO evaluations to only apply when this tag is present. For workloads or namespaces that should be excluded, simply omit this tag.
Additionally, to increase reliability and avoid dependency on automatic rules (which may occasionally fail or be delayed), you can define the tag directly as an annotation within your Kubernetes workloads. This makes the tag native and persistent, ensuring consistent behavior regardless of the Dynatrace rules engine.
This provides a scalable way to manage what gets monitored or alerted on—without impacting the whole environment.
14 May 2025 06:04 PM
Could you give more info for the kubenetes annotation? do you have example yaml and what set up dynatrace need to config to ingest this info? Dynatrace is managed by other team so I am not familiar with what is missing on that part.
14 May 2025 10:47 AM
Hello qizhang1! Did the above solution work in your case? Let me know if you need our further assistance 🙂
14 May 2025 06:05 PM
I have some follow up questions, we are on the right direction.
14 May 2025 07:49 PM
Hi @qizhang1
For Kubernetes auto and tags you can check this Thread,
Solved: Tagging Kubernetes Clusters, Workloads, and Pods - Dynatrace Community