Disable alerts of memory exhausted on pod / process

[Yosi_Neuman]

Hi guys,

Customer asked if there is a way to disable memory anomaly detection issue on pods.

The problem customer get is on a PGI which is a container and look like:

There is no K8S alerts set on this cluster, node, namespace or workload so question are: from where this  anomaly detection is coming and how can we switch it off per a pod / process instance ?

Thanks in advance for your inputs

All the best and stay safe 

Yos 

[Mizső]

Hi @Yosi_Neuman,

I think it comes from the infrastucture high gc suspension becasue the default / global threshold is 25%. In this case the gc suspension is 50%. It is not so healthy...

As far as I know there is not possible to fine tune it on PG or PGI level. I guess you have only two options: global setting fine tune or host level fine tune.

I hope it helps.

Best regards,

Mizső

[Yosi_Neuman]

Hi @Mizső 

Same here did not find a way to set anomaly detection per PG or PGI 🤔

As this is a k8s cluster, host setting will need to be done on all the workers which will affect all the PGs running in this k8s cluster and this is not what customer is looking for 

Thanks for your insight!!!

Yos 

[Mizső]

Hi @Yosi_Neuman,

Yes you are absolutly right. I do not have any other idea. Maybe the customer should solve the initial suspension issue with some java start parametrization, if it is possible. 😉

Ps. I have checked in case of app only instumetation without success. ☹️

Have a nice day!

Mizső

[Yosi_Neuman]

Yep, the fixing the issue was my first answer.

How ever the developers reply "its working fine for us, there is no problem, forget for it"

So NOC guys trying to find the  right a way to "forget it"

Thanks @Mizső 

[MichalOlszewski · Accepted solution]

Hi @Yosi_Neuman.

I believe it came from different source. Most likely from Host Out-of-Memory exception (for PGI).

As those sometimes are off and noisy, especially for IBM javas and Containerization (wrong limits)  I would consider disabling them (notifications only) in favor of kubernetes/openshift events that have their own anomaly detection:

That You can configure with a little better control on different levels.

I'm not 100%sure if API allows separate configs for individual namespaces or workloads but it's worthy a shot to check it.

Bests Michael.

[Yosi_Neuman]

Hi @MichalOlszewski 

Its a very good idea, will suggest it to customer!

In order to change anomaly detection of a specific NS you need to get into the NS settings --> anomaly detection and set its trasholds.

Thanks! 

Yos