topic Re: Create custom alert to restart Pods, in my Saas account in Alerting

Create custom alert to restart Pods, in my Saas account

ayamaya — Wed, 08 Dec 2021 04:32:22 GMT

I need help creating a custom alert which tells me when a pod has rebooted, but I can't find any metrics that show me that.

It is possible to carry out this alert, since the only one I found is one that counts the number of times the container has restarted, but what I'm looking for is that it only alerts me when it restarts and disappears when the pod is running again.

Re: Create custom alert to restart Pods, in my Saas account

skrystosik — Wed, 15 Dec 2021 09:28:05 GMT

I think best option would be counting worker processes number. In general you have for example 10 containers this will result 10 worker processes. If threshold will go below static threshold, you should be alerted. Of course there is issue when you have autoscaler, you will not see situation when containers will scale down gracefully but this is only option I know.

Sebastian

Re: Create custom alert to restart Pods, in my Saas account

dannemca — Wed, 15 Dec 2021 12:43:45 GMT

I am not a specialist on K8S/OC, but did you tried to use the Events and/or Prometheus custom metrics for it?

Also, if you find a Process Group that is pertinent to these pods you are interested about , you can try use the Process Group Availability monitoring, it will tell you when a process instance (pod) is down.

Re: Create custom alert to restart Pods, in my Saas account

ayamaya — Thu, 16 Dec 2021 22:44:46 GMT

If we currently have that, what we see is the limitation that when the Pods have automatic scaling, it will change and we can only change the group manually.