Customer asked if there is a way to disable memory anomaly detection issue on pods.
The problem customer get is on a PGI which is a container and look like:
There is no K8S alerts set on this cluster, node, namespace or workload so question are: from where this anomaly detection is coming and how can we switch it off per a pod / process instance ?
Thanks in advance for your inputs
All the best and stay safe
Solved! Go to Solution.
I think it comes from the infrastucture high gc suspension becasue the default / global threshold is 25%. In this case the gc suspension is 50%. It is not so healthy...
As far as I know there is not possible to fine tune it on PG or PGI level. I guess you have only two options: global setting fine tune or host level fine tune.
I hope it helps.
Same here did not find a way to set anomaly detection per PG or PGI 🤔
As this is a k8s cluster, host setting will need to be done on all the workers which will affect all the PGs running in this k8s cluster and this is not what customer is looking for
Thanks for your insight!!!
Yes you are absolutly right. I do not have any other idea. Maybe the customer should solve the initial suspension issue with some java start parametrization, if it is possible. 😉
Ps. I have checked in case of app only instumetation without success. ☹️
Have a nice day!
Yep, the fixing the issue was my first answer.
How ever the developers reply "its working fine for us, there is no problem, forget for it"
So NOC guys trying to find the right a way to "forget it"
I believe it came from different source. Most likely from Host Out-of-Memory exception (for PGI).
As those sometimes are off and noisy, especially for IBM javas and Containerization (wrong limits) I would consider disabling them (notifications only) in favor of kubernetes/openshift events that have their own anomaly detection:
That You can configure with a little better control on different levels.
I'm not 100%sure if API allows separate configs for individual namespaces or workloads but it's worthy a shot to check it.
Its a very good idea, will suggest it to customer!
In order to change anomaly detection of a specific NS you need to get into the NS settings --> anomaly detection and set its trasholds.