Problem alerts for memory consumption for kubernetes pods

Platform Alerting
, , , ,
1

Dear Community,

I have a requirement:

Scenario: If a specific k8s workloads is using above 90% of memory against memory limit then, I should have an alerts for it, Either in dynatrace probelm notfication page or in custiom integretion(email or webhook)

Note: We have dynatrace managed environment

To understand it I have added a demo screenshot from our environment. This workloads memory limits is 6GiB as soon if consumption goes beyond 90% which more that 5.4GiB. We should have notification for it. Thanks

skhamitkar_0-1766657536410.png
skhamitkar_0-1766657536410.png1749×430 50.9 KB

2

Hi,

Have you check Alert on common Kubernetes/OpenShift issues?

Best regards

3

@AntonPineiro is it possible to get notification in email? I am asking this coz the team who put this requirement they want an email too. If its possible can you let me know with the detailed steps?

4

Hi there,

Yes it is possible to get these alerts per e-mail.  First you’ll need to create an alerting profile matching the event’s you would like to have covered.
After this you can set up problem notification toward e-mail.

The alerting profiles filters the events related to Kubernetes (if configured correctly of course). Afterwards you’ll enable problem forwarding with this alerting  profile attached to ensure they only get the Kubernetes based problems (and not everything :slightly_smiling_face: )

5

@michiel_otten Thanks for the great insights. But in the alerting profile how can I Put the condition. Which calculates memory consumption for the specific workload(If not here is there any alternate way) e.g. If I have xyz-workload and the memory more than 90% consumed for this workload only. So, I should have memory- alerts for xyz-workload only.

I saw in the alerting I could not it any alternate way?

skhamitkar_0-1767077543507.png
skhamitkar_0-1767077543507.png1306×829 61.3 KB

6

Hi there,

The alerting profile is only configured for distributing alerts.  So you could choose to only distribute alerts containing your event name f.e.

If you want to finetune the alerts themself you can use one of the predefined alerts (mentioned by Anton) or create a custom alerts via de Davis Anomaly detector app.

Within this app you can write your own DQL code that generates events (thus alerts). F.e.:

timeseries avg(k8s.pod.memory.usage), by: { dt.entity.cloud_application_instance }
| fieldsAdd name = entityName(dt.entity.cloud_application_instance)
| filter contains(name, "dynatrace")
7

This query wont work and this app is not also available for self managed setup. @michiel_otten

What could be the best solution for managed setup?

8

My bad, for a managed solution you could use “metric events”. The query itself you can test via the data explorer.

For example:

builtin:containers.memory.usagePercent:splitBy("dt.entity.container_group_instance","Container"):filter(contains("Container","YOURCONTAINER"))

This works on Managed / Classic.

9

Thanks @michiel_otten , for the insights but the question remains same.

Thanks for the great insights. But in the alerting profile how can I Put the condition. Which calculates memory consumption for the specific workload (If not here is there any alternate way) e.g. If I have xyz-workload and the memory more than 90% consumed for this workload only. So, I should have memory- alerts for xyz-workload only.