Problem alerts for memory consumption for kubernetes pods

[skhamitkar]

Dear Community, 

I have a requirement: 

Scenario: If a specific k8s workloads is using above 90% of memory against memory limit then, I should have an alerts for it, Either in dynatrace probelm notfication page or in custiom integretion(email or webhook)

Note: We have dynatrace managed environment

To understand it I have added a demo screenshot from our environment. This workloads memory limits is 6GiB as soon if consumption goes beyond 90% which more that 5.4GiB. We should have notification for it. Thanks

[AntonPineiro]

Hi,

Have you check Alert on common Kubernetes/OpenShift issues?

Best regards

[skhamitkar]

@AntonPineiro is it possible to get notification in email? I am asking this coz the team who put this requirement they want an email too. If its possible can you let me know with the detailed steps?

[michiel_otten]

Hi there,

Yes it is possible to get these alerts per e-mail.  First you'll need to create an alerting profile matching the event's you would like to have covered.
After this you can set up problem notification toward e-mail.

The alerting profiles filters the events related to Kubernetes (if configured correctly of course). Afterwards you'll enable problem forwarding with this alerting  profile attached to ensure they only get the Kubernetes based problems (and not everything 🙂 )

[skhamitkar]

@michiel_otten Thanks for the great insights. But in the alerting profile how can I Put the condition. Which calculates memory consumption for the specific workload(If not here is there any alternate way) e.g. If I have xyz-workload and the memory more than 90% consumed for this workload only. So, I should have memory- alerts for xyz-workload only.

I saw in the alerting I could not it any alternate way?

[michiel_otten]

Hi there,

The alerting profile is only configured for distributing alerts.  So you could choose to only distribute alerts containing your event name f.e.

If you want to finetune the alerts themself you can use one of the predefined alerts (mentioned by Anton) or create a custom alerts via de Davis Anomaly detector app.

Within this app you can write your own DQL code that generates events (thus alerts). F.e.:

timeseries avg(k8s.pod.memory.usage), by: { dt.entity.cloud_application_instance }
| fieldsAdd name = entityName(dt.entity.cloud_application_instance)
| filter contains(name, "dynatrace")