cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Disable alerts of memory exhausted on pod / process

Yosi_Neuman
DynaMight Leader
DynaMight Leader

Hi guys,

Customer asked if there is a way to disable memory anomaly detection issue on pods.

The problem customer get is on a PGI which is a container and look like:

Yosi_Neuman_0-1680081389883.png

There is no K8S alerts set on this cluster, node, namespace or workload so question are: from where this  anomaly detection is coming and how can we switch it off per a pod / process instance ?

Thanks in advance for your inputs

All the best and stay safe 

Yos 

dynatrace certificated professional - dynatrace master partner - Matrix Soft Ware Division - Israel
6 REPLIES 6

Mizső
DynaMight Leader
DynaMight Leader

Hi @Yosi_Neuman,

I think it comes from the infrastucture high gc suspension becasue the default / global threshold is 25%. In this case the gc suspension is 50%. It is not so healthy...

Mizs_0-1680089848631.png

As far as I know there is not possible to fine tune it on PG or PGI level. I guess you have only two options: global setting fine tune or host level fine tune.

I hope it helps.

Best regards,

Mizső

Certified Dynatrace Professional

Hi @Mizső 

Same here did not find a way to set anomaly detection per PG or PGI 🤔

As this is a k8s cluster, host setting will need to be done on all the workers which will affect all the PGs running in this k8s cluster and this is not what customer is looking for :facepalm:

Thanks for your insight!!!

Yos 

dynatrace certificated professional - dynatrace master partner - Matrix Soft Ware Division - Israel

Hi @Yosi_Neuman,

Yes you are absolutly right. I do not have any other idea. Maybe the customer should solve the initial suspension issue with some java start parametrization, if it is possible. 😉

Ps. I have checked in case of app only instumetation without success. ☹️

Mizs_0-1680095634465.png

Have a nice day!

Mizső

Certified Dynatrace Professional

Yep, the fixing the issue was my first answer.

How ever the developers reply "its working fine for us, there is no problem, forget for it"

So NOC guys trying to find the  right a way to "forget it"

Thanks @Mizső 

dynatrace certificated professional - dynatrace master partner - Matrix Soft Ware Division - Israel

MichalOlszewski
Advisor

Hi @Yosi_Neuman.

I believe it came from different source. Most likely from Host Out-of-Memory exception (for PGI).

As those sometimes are off and noisy, especially for IBM javas and Containerization (wrong limits)  I would consider disabling them (notifications only) in favor of kubernetes/openshift events that have their own anomaly detection:

MichalOlszewski_0-1680095666266.png

That You can configure with a little better control on different levels.

I'm not 100%sure if API allows separate configs for individual namespaces or workloads but it's worthy a shot to check it.

Bests Michael.

Hi @MichalOlszewski 

Its a very good idea, will suggest it to customer!

In order to change anomaly detection of a specific NS you need to get into the NS settings --> anomaly detection and set its trasholds.

Thanks! 

Yos 

dynatrace certificated professional - dynatrace master partner - Matrix Soft Ware Division - Israel